I do not get the purpose of concurrent messages for saga. I'd expect it to behave more like an actor. So all the messages with the same CorrelationId are processed sequentially. The whole purpose of saga is orchestration of a long running process, so why does parallel message processing matter?
Can you give a legit example where handling messages concurrently for the saga instance is beneficial compared to the sequential mode?
Or do I understand it wrong, and concurrency just means several different saga instances running in parallel?
The reason to ask is this fragment from NServiceBus docs:
The main reason for avoiding accessing data from external resources is possible contention and inconsistency of saga state. Depending on persister, the saga state is retrieved and persisted with either pessimistic or optimistic locking. If the transaction takes too long, it's possible another message will come in that correlates to the same saga instance. It might be processed on a different (concurrent) thread (or perhaps a scaled out endpoint) and it will either fail immediately (pessimistic locking) or while trying to persist the state (optimistic locking). In both cases the message will be retried.
There's none, messages for the single saga instance need to be processed sequentially. There's nothing special about saga configuration in MassTransit, you really want to use a separate endpoint for it and set the concurrency limit to one.
But that would kill the performance for processing messages for different saga instances. To solve this, keep the concurrently limit higher than one and use the partitioning filter by correlation id. Unfortunately, the partitioning filter requires by-message configuration, so you'd need to configure the partitioning for all messages that the saga consumes.
But it all depends on the use-case. All the concurrency issues are resolved by retries when using the persistence-based optimistic concurrency, which is documented per saga persistence provider. Certainly, it produces some noise by retrying database operations, but if the number of retries is under control, you can just keep it as it is.
If you hit tons of retries due to massive concurrent updates, you can revert to partitioning your saga.
Related
To keep it short, here is a simplified situation:
I need to implement a queue for background processing of imported data files. I want to dedicate a number of consumers for this specific task (let's say 10) so that multiple users can be processed at in parallel. At the same time, to avoid problems with concurrent data writes, I need to make sure that no one user is processed in multiple consumers at the same time, basically all files of a single user should be processed sequentially.
Current solution (but it does not feel right):
Have 1 queue where all import tasks are published (file_queue_main)
Have 10 queues for file processing (file_processing_n)
Have 1 result queue (file_results_queue)
Have a manager process (in this case in node.js) which consumes messages from file_queue_main one by one and decides to which file_processing queue to distribute that message. Basically keeps track of in which file_processing queues the current user is being processed.
Here is a little animation of my current solution and expected behaviour:
Is RabbitMQ even the tool for the job? For some reason, it feels like some sort of an anti-pattern. Appreciate any help!
The part about this that doesn't "feel right" to me is the manager process. It has to know the current state of each consumer, and it also has to stop and wait if all processors are working on other users. Ideally, you'd prefer to keep each process ignorant of the others. You're also getting very little benefit out of your processing queues, which are only used when a processor is already working on a message from the same user.
Ultimately, the best solution here is going to depend on exactly what your expected usage is and how likely it is that the next message is from a user that is already being processed. If you're expecting most of your messages coming in at any one time to be from 10 users or fewer, what you have might be fine. If you're expecting to be processing messages from many different users with only the occasional duplicate, your processing queues are going to be empty much of the time and you've created a lot of unnecessary complexity.
Other things you could do here:
Have all consumers pull from the same queue and use some sort of distributed locking to prevent collisions. If a consumer gets a message from a user that's already being worked on, requeue it and move on.
Set up your queue routing so that messages from the same user will always go to the same consumer. The downside is that if you don't spread the traffic out evenly, you could have some consumers backed up while others sit idle.
Also, if you're getting a lot of messages in from the same user at once that must be processed sequentially, I would question if they should be separate messages at all. Why not send a single message with a list of things to be processed? Much of the benefit of event queues comes from being able to treat each event as a discrete item that can be processed individually.
If the user has a unique ID, or the file being worked on has a unique ID then hash the ID to get the processing queue to enter. That way you will always have the same user / file task queued on the same processing queue.
I am not sure how this will affect queue length for the processing queues.
I have a question related to a tricky situation in an event-driven system that I want to ask for advise. Here is the situation:
In our system, I use redis as a memcached database, and kafkaa as message queues. To increase the performance of redis, I use lua scripting to process data, and at the same time, push events into a blocking list of redis. Then there will be a process to pick redis events in that blocking list and move them to kafka. So in this process, there are 3 steps:
1) Read events from redis list
2) Produce in batch into kafka
3) Delete corresponding events in redis
Unfortunately, if the process dies between 2 and 3, meaning that after producing all events into kafka, it doesn't delete corresponding events in redis, then after that process is restarted, it will produce duplicated events into kafka, which is unacceptable. So does any one has any solution for this problem. Thanks in advance, I really appreciate it.
Kafka is prone to reprocess events, even if written exactly once. Reprocessing will almost certainly be caused by rebalancing clients. Rebalancing might be triggered by:
Modification of partitions on a topic.
Redeployment of servers and subsequent temporary unavailabilty of clients.
Slow message consumption and subsequent recreation of client by the broker.
In other words, if you need to be sure that messages are processed exactly once, you need to insure that at the client. You could do so, by setting a partition key that ensures related messages are consumed in a sequential fashion by the same client. This client could then maintain a databased record of what he has already processed.
Hi I have with this code
SendMessageToMyQueue();
UpdateStatusInDbThatMessageWasSent();
that sometimes message is processed before status is updated which I would like to avoid.
My question is if I wrap that two line with a transaction like this:
using(var tr = new TransactionScope())
{
SendMessageToMyQueue();
UpdateStatusInDbThatMessageWasSent();
tr.Compleate();
}
will be guaranteed that there will be a lock on MyQueue created and this lock will be not released until UpdateStatusInDbThatMessageWasSent will update the status?
also if I add try catch with rollback and updating status fails, will the message be removed from MyQueue ?
There is no such thing as lock on a queue. The message, however, will be processed transactionally, if the following conditions are met. By transactionally, I mean that the message will be returned to the queue if an unhandled exception is thrown. The conditions to make this happen are:
Your database can enlist and take part in a distributed transaction. Not every database out there does. Some Document databases have none (in case of MongoDB) or sketchy (in case of RavenDB) support for DTC.
Your transport also supports distributed transactions. If you go with a broker type transports, SQL Server Transport is your best bet and on Bus type transports MSMQ is a good choice. Transports like Azure ServiceBus or RabbitMQ have very limited transactions support and do not support distributed transactions.
You'll need to run Distributed Transaction Coordinator service configured and running.
Two other things to note:
What if you're using a transport that lacks DTC support? Most of the time, you are better off if you can design your system to be idempotent. Outbox feature of NServiceBus allows you simulate DTC to some extent.
When a message is picked from the queue, processed, and returned to the queue due to an exception, it might end up being in a different place in the queue. You need to design for messages arriving out of order when designing a message-based architecture.
With all said above, exactly-once delivery guarantees are always a hot topic and disputed.
I'm using NServiceBus to handle some asynchronous tasks. Occasionally I have a task where I need to process 10,000 records, so this takes a few hours.
My problem is that when I handle these records all together, I cannot use NServiceBus default transaction handling.
Also - if I split these records up into 10,000 smaller messages, they will clog up MSMQ for a few hours, and users who are expecting functions to take a few minutes, will be waiting hours.
Is there a way in NServiceBus to prioritise different messages?
I'd consider breaking it down into smaller batches (not necessarily one message per record) and having a separate endpoint service specifically for this process so that other stuff is not held up. If breaking it into batches and you care about the when they all complete then I'd recommend using a saga to track that state.
What is the intended usage of ForwardRecievedMessagesTo?
I read some where that it is to support auditing. Is there any harm in using it as a solution to ensure that messages have been processed and if not reprocessing them? lets say a message was sent to queue_A#server_A and also forwarded to q_All#server_All and before the message was handled, machine_A died irrecoverably. In such a case, I could have a handler pick up messages from q_All#sever_All and check against a database table if the message has been processed. If not reprocess(publish or send) the message or save it in a database table.
Also, what is the performance implication of using forwardreceivedmessageto? How is it different from journalling?
Yes, I am trying to not use msmq clustering.
The feature is there to support auditing. If your machine dies during processing then the messages will backup at the sending machine and would continue to flow after the machine recovered. This means you must size the disk on the sending machine appropriately. You could leverage auditing to accomplish this and the overhead would be minimal. The implication would be the time it would take to complete the distributed transaction to the other machine where your audit queue lives which should be very small.