I'm building this BPMN in which a user has to fill 6 forms (do 6 tasks). After he's completed all 6 he should get some results, but if any of those tasks are missing, then we do not have the results.
Which gateway should I use? The one I thought suited the most was the inclusive gateway, but all 6 tasks can be completed in any order.
Should I use a complex gateway and just describe the process? Or the parallel gateway works just fine
If exactly 6 tasks have to be performed in any desired order, and the flow shall not continue before all 6 are completed, the simplest would be to use a parallel gateway:
As soon as a token arrives at the first gate, a token is send on every outgoing flow, which activates each of the tasks. The gate at the left will generate a token to continue the flow once every task is complete and passed its token to the gate.
The complex gateway could also address your needs, since it allows complex conditions on the incoming and outgoing flows, including with multiple tokens on multiple branch. This would be necessary for example if 5 out of 6 tasks would have to be completed, or if you only want some combinations of completed tasks to continue the flow. But it seems overkill for your problem.
The inclusive gateway is not a solution for your needs, since it only allows one of the outgoing branch to be activated.
Related
I’m a bpmn newbie with a pretty complicated process to model.
Does it make sense to have the same user (user 1 in my case) in different processes?
And how can I link process 2 and process 3, since process 2 has two different ends?
The meaning of a lane in a pool is left to the modeler, so you. There can perfectly be two lanes in two different pools, that share the same user name. Note that the lane name is free and does not formally define the performer of the tasks in the model (even if visually the meaning is clear).
If you need to link process 2 and process 3 with a message flow the simplest way would be to either issue two message flows or merge the two ends (if the flow ends, it ends, regardless of the user).
Important remark: an end event symbol on a process should in principe not have outgoing flows. If you want to connect the process with a message flow like you did between 1 and 2, the end event should me a message end event (thick circle with envelope in it). And the consequence would be that the receiving process starts with a message start event (thin circle with an envelope in it). If there is no message between the two, don't link them.
Overview
I’m currently building a prototype to track and control a fleet of drones.
The prototype consists of a service and a web app. In the web app, the location of each drone is displayed in real-time on a map and the user can issue basic commands to each of these drones.
The service is automated and can also issue commands to each of the drones at random times when certain conditions occur.
I am using HiveMQ (an MQTT broker) to facilitate communication between drones, the web app and the service. The web app and the service are both subscribed to the 'telemetry' topic to receive real-time data about the network of drones. The broker will store the telemetry data for each drone directly into a database through the use of HiveMQ's extension functionality.
Specific commands can only be executed if certain criteria are met.
For example: To issue an 'execute mission' command to a drone the service or the web app will make a call to an API. The API will:
Check the drone is not currently on a mission (drone status value must be idle)
Check weather conditions are acceptable in the area the mission is to occur
(Note by 'mission' I mean a drone fly's to a series of set locations autonomously).
If conditions aren't met a response indicating this will be returned to the requester (web app or service). If conditions are met the API will issue the command to the appropriate drone via the MQTT broker and send a response to the requester.
Requirements
I need a storage mechanism that meets the following criteria:
I need to ensure that a race condition does not occur between the web app and the service. That is if a request to issue a command to a drone is being made by the web app, a request made by the service in this time should be automatically rejected.
Drone status between the service and the web app are not synchronous, as a result, they need a synchronized point to check a drones status.
Drones will update their status every second, and API call's to issue commands will be made every 10 - 30 seconds. There will be 5 drones in this prototype but I would like a solution that can scale to 50 drones.
Considered Solution
My solution would be that of a relational database - using a separate table with a 'request_lock' field, this field uses a row-level lock.
When an API call is made it checks if this field is true, if true the request is rejected. If it is false it sets the field to true performs the necessary condition checks and then sets the 'request_lock' field to false when once the command has reached the drone.
I am concerned the status update frequency from each drone does not fit a relational database model and won't scale well. Am I on the right track, or should I be looking to include a NoSQL database in some way to handle status updates?
Thank you to anyone who takes the time to answer.
There are a lot of questions here, so I'll try to pick what seems to be most important:
I am concerned the status update frequency from each drone does not fit a relational database model ..
Should I use a relational or non-relational database?
First, let's calculate the maximum number of drone status updates, per second.
Drones will update their status every second, and API call's [sic] to issue commands will be made every 10 - 30 seconds. There will be 5 drones in this prototype but I would like a solution that can scale to 50 drones.
50 drones * 1 drone-update per second = 50 drone-updates per second
50 drones * (10 / 60) drone-commands per second = 8.3 drone-commands per second
So, can a relational database handle ~60 queries per second?
Yes. Assuming reasonable query complexity, this is within the ability of a traditional relational database. I would not expect the database to need extraordinary system resources, either.
If you'd like to confirm this level of performance with a benchmark, I'd recommend a tool like pgbench.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
We have a requirement where we need to process 10,000 transactions once daily in an offline (non real time mode).
Which of the 2 options are preferable
A batch file with 10,000 rows sent once a day and processed
or
An API call in small batches (as I am presuming sending 10K rows at once is not an option).
I was advised by my architects that option 1 is preferable and an API would only make sense when batch sizes are small - as the disadvantage of 2 is that the person calling the API has to break the payload down into small chunks when they have all the information available to them at once.
I am keen to see how "2" could be a viable option so any comments/suggestion to help make the case would be very helpful.
Thanks
Rahul
This is not a full answer. However, I would like to mention one reason in favor of REST API: Validation. This is better managed through the API. Once the file is dropped into an FTP location, it will be your responsibility to validate the format of the file. Will it be easy to route a "bad" file back to its source with a message to explain the bounce back?
With an API call, if the representation coming in does not adhere to a valid schema e.g. XML, json, etc. then your service can respond with a: "400 Bad Request" http status code. This keeps the responsibility of sending data in a valid format with the consumer of the service and helps to achieve a better separation of concerns.
Additional reasoning for a REST API:
Since your file contains transactions, each record should be atomic (If this were not true e.g. there are relationships between the records in the file, then those records should not be considered "transactions"). Therefore, chunking the file up into smaller batches should be trivial.
Regardless, you can define a service that accepts transactions in batch and respond with an HTTP status code of "202 Accepted". A 202 code indicates that the request was received and will be processed asynchronously. Therefore, the response can also contain callback links to check the status of individual transactions; or the batch as a whole. At that point, you would be implementing HATEOAS (Hypermedia as the Engine of Application State) and be in a position to automate the entire process and report on status.
Alteratively with batch files, if the file passes an upfront format validation check, then you'll still have to process each transaction individually downstream. Some records may load, others may not. My assumption is the records that fail to load would still need to be handled. And, you may need to provide the users a view of what succeeded vs. failed. Now, this can all be handled outside the REST API. However, the API pattern is simple and elegant IMHO to this purpose.
Using Batch Process is always a better idea. you can trigger batch process using REST API.
With Batch processing you can always send an email with msg "improper file format" or you can also send "Which records processed and which did not" . With Rest you cannot keep track records and transactions.
As mentioned in above comment you can use Rest API to trigger a batch Process asynchronously and send the status response using HATEOAS.
SPRING BATCH + SPring REST using SPring BOOT
I have the same question and all answer I found the same subjective answer. I would like put some ideas to compare both concepts:
Batch solution requires more storage than REST API. You will need
store your results on intermediate storage area, and write it on an
open format. Perhaps you can compress it, but you are changing
storage with processing.
REST API could use more network bandwidth than batch solution, only
if the intermediate storage is not in network drive. Fetch request,
and query pooling could require a lot of network bandwidth,
but could be solved with web-hooks or web-sockets.
REST API is easiest to automatic recovery than batch solution. REST
API response code can help to take automatic decision to recover
from a FAIL. And you reduce the number of services required to
identify it. If the network is down an email could fail as REST API.
And REST API help you to define a good API on these cases.
REST API can manage high number of rows as any other TCP protocol
(as FTP). But in case of any fail you will need logic to manage it.
It means the REST API will require a chunk enabled protocol too. For
batch service, this logic is in FTP protocol, but with his own
logic, not your business logic.
Batch service does not require to reserve an instance all time
(CPU, IP address, port, etc), just
run when it is needed. You will need a scheduler to start it, or men
force. Or a man to restart it if it fails. Again, out of scheduler,
it is not natural to automatize.
Batch service does not require more security setup from developer
side: REST API must take care about authentication. Also, must think
on injection or other attack methods. REST API could be use helper
services to prevent all of this, but it means more configuration.
Batch services are easy to deploy. Batch services could run on your
machine, or a server and run it when business need. REST API requires
continues health check, use a deployment strategy to keep it up, take
care about DNS configuration, etc. Check if your company give you all
this services.
If this solution is for your company, check what your company is
doing. Right now there is a common policy to move to REST API, but
if your support team do not know about it but has a lot of
experience with batch solution, could be a good idea do not improve.
I have a WCF service that will serve multiple clients.
They will have operation like 'Accept Suggested Match' or 'Reject Suggested Match'.
I would like all the operations to be run serially, but I am afraid this will have an impact on performance.
From what I saw - the default (and most used) instancing is 'Per Call' and the default concurrency mode is single.
So is it true I would need to use 'Single' mode in the concurrency ?
And how bad of an impact is it ?
(I estimate tens of clients using the system at the most).
Is there any way to enjoy both worlds ?
Use parallel computing (the service communicates with a database) and still perform the operations serially ?
In this post I read about 'Instance mode = Per Call and Concurrency = Single':
For every client instance, a single thread will be allocated.
For every method call, a new service instance will be created.
A single thread will be used to serve all WCF instances generated from a single client instance.
It seems like this doesn't guarantee to me that operation calls will be performed serially !
For example, if this happens :
CLIENT ALPHA calls 'Operation A'
CLIENT ALPHA calls 'Operation B'
CLIENT BETA calls 'Operation C'
From what I read - it says 'A single thread will be used to serve all WCF instances generated from a single client instance'. This sounds to me like it means that CALL 1 and CALL 2 would be performed serially, but CALL 3 might be performed between them, because it is from a different client thread.
Is this true ? Is there no way to make sure that calls are handled at the order they arrive ?
And can someone tell me if it is a bad practice or are there real world scenarios where it is accepted and recommended to use this method of communication in WCF.
Thank you very much
I think there are 3 ways to answer this question
A direct answer
An alternative approach for situations where you control the clients and the server
An alternative approach where you don't control the clients
I'd like to give answers 2 and 3 first (because I think they are better) but I will give a direct answer at the end...I promise!
I believe your problem is not that you need to process messages in the order they are received by the service, it is that since you have independent clients, you cannot guarantee that they are received by the service in the same order they are sent from the clients.
Consider your example of clients ALPHA and BETA. Even though client ALPHA might send the request to call operation 2 before client BETA calls operation C, you have no way of knowing what order they will be received by the service. This means that even if you process them at the server in the order they are received, this might still be the "wrong" order.
If you agree with this, carry on reading. If not, go straight to the direct answer at the bottom of this post ;o)
Approach 1: Use WS-Transactions
If you control both the service and the clients, and if the clients are capable of implementing WS-* SOAP protocols (e.g. .Net WCF clients). Then you could use the WS-Transaction protocol to make sure that operations from a single client that are intended to be processed in a single long-running transaction are definitely done that way.
For details of how to do WS-Transcation with WCF, look at
http://msdn.microsoft.com/en-us/library/ms752261.aspx
Following you example with clients ALPHA and BETA, this would enable client ALPHA to
Start a transaction
Call operation 1
Call operation 2
Commit the transaction
The effect of the ALPHA starting a transaction on the client is to flow that transaction through to the server so it can create a DB transaction which guarantees that operation 1 and operation 2 complete together or not at all.
If client BETA calls operation 3 while this is going on, it will be forced to wait until the transaction commits or aborts. Note that this might cause operation 3 to time out in which case BETA would need to handle that error.
This behaviour will work with any InstanceContextMode or ConcurrencyMode.
However, the drawback is that these kind of long running, distributed transactions are not generally good for scalability. Today you have 10s of client, but will that grow in the future? And will the clients always be able to implement WS-Transaction? If so, then this could be a good approach.
Approach 2: Use an "optimistic" type approach
If you need to support many different client types (e.g. phones, browsers etc.) then you could just allow the operations to happen in any order and make sure that the service and client logic can handle failures.
This might sound bad, but
you can use database transactions on the server side (not the WS-Transaction ones I mentioned above) to make sure your database is always consistent
each client can synchronise its own calls so that in your example, ALPHA would wait for call 1 to complete OK before doing call 2
The drawback here is that you need to think carefully about what failure logic you need and make sure the client and service code is robust and behaves appropriately.
In your example, I mentioned that ALPHA can synchronise its own calls, but if BETA calls operation 3 after ALPHA calls operation 1 but before it calls operation 2, it could be that
Operation 2 will fail (e.g. operation 3 has deleted a record that operation 2 is trying to update) in which case you need to handle that failure.
Operation 3 will overwrite operation 3 (e.g. operation 3 updates a record and the operation 2 tries to update the same record). In this case you need to decide what to do. Either you can let operation 2 succeed in which case the updates from BETA are lost. Or you could have operation 2 detect that the record has changed since it was read by ALPHA and then fail, or maybe inform the user and have them decide if they want to overwrite the changes.
For a discussion on this in the context of MS Entity Framework see
http://msdn.microsoft.com/en-us/library/bb738618.aspx
for a general discussion of it, see
http://en.wikipedia.org/wiki/Optimistic_concurrency_control
Again, in WCF terms, this will work with any ConcurrencyMode and InstanceContextMode
This kind of approach is the most common one used where scalability is important. The drawback of it is that the user experience can be poor (e.g. they might have their changes overwritten without being aware of it).
Direct answer to the original question
If you definately need to make sure messages are processed in series, I think you are looking for InstanceContextMode.Single
http://msdn.microsoft.com/en-us/library/system.servicemodel.instancecontextmode.aspx
This means that a single instance of the service class is created for the lifetime of the service application. If you also use ConcurrencyMode = Single or ConcurrencyMode = Reentrant this then means that your service will only process one message at a time (because you have a single instance and it is single threaded) and the messages will be processed in the order they are received.
If you use ConcurrencyMode = multiple then your single service instance will process multiple messages at the same time on different threads so the order of processing is not guaranteed.
How bad a performance impact this will have will depend on how long each service call takes to execute. If you get 10 per second and each takes 0.1 second it will be fine. If you get 20 per second and each normally takes 0.1 seconds, you will see a 100% increase in the average time.
I created a RESTful service using WCF which calculates some value and then returns a response to the client.
I am expecting a lot of traffic so I am not sure whether I need to manually implement queues or it is not neccessary in order to process all client requests.
Actually I am receiving measurements from clients which have to be stored to the database - each client sends a measurement every 200 ms so if there are a multiple clients there could be a lot of requests.
And the other operation performed on received data. For example a client could send an instruction "give me the average of the last 200 measurements" so it could take some time to calculate this value and in the meantime the same request could come from another client.
I would be very thankful if anyone could give any advice on how to create a reliable service using WCF.
Thanks!
You could use the MsmqBinding and utilize the method implemented by eedsi9n. However, from what I'm gathering from this post is that you're looking for something along the lines of a pub/sub type of architecture.
This can be implemented with the WSDualHttpBinding which allows subscribers to subscribe to events. The publisher will then notify the user when the action is completed.
Therefore you could have Msmq running behind the scenes. The client subscribes to the certain events, then perhaps it publishes a message that needs to be processed. THe client sits there and does work (because its all async) and when the publisher is done working on th message it can publish an event (The event your client subscribed to) letting you know that its done. That way you don't have to implement a polling strategy.
There are pre-canned solutions for this as well. Such as NService Bus, Mass Transit, and Rhino Bus.
If you are using Web Service, Transmission Control Protocol (TCP/IP) will act as the queue to a certain degree.
TCP provides reliable, ordered
delivery of a stream of bytes from one
program on one computer to another
program on another computer.
This guarantees that if client sends packet A, B, then C, the server will received it in that order: A, B, then C. If you must reply back to the client in the same order as request, then you might need a queue.
By default maximum ASP.NET worker thread is set to 12 threads per CPU core. So on a dual core machine, you can run 24 connections at a time. Depending on how long the calculation takes and what you mean by "a lot of traffic" you could try different strategies.
The simplest one is to use serviceTimeouts and serviceThrottling and only handle what you can handle, and reject the ones you can't.
If that's not an option, increase hardware. That's the second option.
Finally you could make the service completely asynchronous. Implement two methods
string PostCalc(...) and double GetCalc(string id). PostCalc accepts the parameters, stuff them into a queue (or a database) and returns a GUID immediately (I like using string instead of Guid). The client can use the returned GUID as a claim ticket and call GetCalc(string id) every few seconds, if the calculation has not finished yet, you can return 404 for REST. Calculation must now be done by a separate process that monitors the queue.
The third option is the most complicated, but the outcome is similar to that of the first option of putting cap on incoming request.
It will depend on what you mean by "calculates some value" and "a lot of traffic". You could do some load testing and see how the #requests/second evolves with the traffic.
There's nothing WCF specific here if you are RESTful
the GET for an Average would give a URI where the answer would wait once the server finish calculating (if it is indeed a long operation)
Regarding getting the measurements - you didn't specify the freshness needed (i.e. when you get a request for an average - how fresh do you need the results to be) Also you did not specify the relative frequency of queries vs. new measurements
In any event you can (and IMHO should) use the queue (assuming measuring your performance proves it) behind the endpoint. If you change the WCF binding you might still be RESTful but will not benefit from the standard based approach of REST over HTTP