Hi I have with this code
SendMessageToMyQueue();
UpdateStatusInDbThatMessageWasSent();
that sometimes message is processed before status is updated which I would like to avoid.
My question is if I wrap that two line with a transaction like this:
using(var tr = new TransactionScope())
{
SendMessageToMyQueue();
UpdateStatusInDbThatMessageWasSent();
tr.Compleate();
}
will be guaranteed that there will be a lock on MyQueue created and this lock will be not released until UpdateStatusInDbThatMessageWasSent will update the status?
also if I add try catch with rollback and updating status fails, will the message be removed from MyQueue ?
There is no such thing as lock on a queue. The message, however, will be processed transactionally, if the following conditions are met. By transactionally, I mean that the message will be returned to the queue if an unhandled exception is thrown. The conditions to make this happen are:
Your database can enlist and take part in a distributed transaction. Not every database out there does. Some Document databases have none (in case of MongoDB) or sketchy (in case of RavenDB) support for DTC.
Your transport also supports distributed transactions. If you go with a broker type transports, SQL Server Transport is your best bet and on Bus type transports MSMQ is a good choice. Transports like Azure ServiceBus or RabbitMQ have very limited transactions support and do not support distributed transactions.
You'll need to run Distributed Transaction Coordinator service configured and running.
Two other things to note:
What if you're using a transport that lacks DTC support? Most of the time, you are better off if you can design your system to be idempotent. Outbox feature of NServiceBus allows you simulate DTC to some extent.
When a message is picked from the queue, processed, and returned to the queue due to an exception, it might end up being in a different place in the queue. You need to design for messages arriving out of order when designing a message-based architecture.
With all said above, exactly-once delivery guarantees are always a hot topic and disputed.
Related
I do not get the purpose of concurrent messages for saga. I'd expect it to behave more like an actor. So all the messages with the same CorrelationId are processed sequentially. The whole purpose of saga is orchestration of a long running process, so why does parallel message processing matter?
Can you give a legit example where handling messages concurrently for the saga instance is beneficial compared to the sequential mode?
Or do I understand it wrong, and concurrency just means several different saga instances running in parallel?
The reason to ask is this fragment from NServiceBus docs:
The main reason for avoiding accessing data from external resources is possible contention and inconsistency of saga state. Depending on persister, the saga state is retrieved and persisted with either pessimistic or optimistic locking. If the transaction takes too long, it's possible another message will come in that correlates to the same saga instance. It might be processed on a different (concurrent) thread (or perhaps a scaled out endpoint) and it will either fail immediately (pessimistic locking) or while trying to persist the state (optimistic locking). In both cases the message will be retried.
There's none, messages for the single saga instance need to be processed sequentially. There's nothing special about saga configuration in MassTransit, you really want to use a separate endpoint for it and set the concurrency limit to one.
But that would kill the performance for processing messages for different saga instances. To solve this, keep the concurrently limit higher than one and use the partitioning filter by correlation id. Unfortunately, the partitioning filter requires by-message configuration, so you'd need to configure the partitioning for all messages that the saga consumes.
But it all depends on the use-case. All the concurrency issues are resolved by retries when using the persistence-based optimistic concurrency, which is documented per saga persistence provider. Certainly, it produces some noise by retrying database operations, but if the number of retries is under control, you can just keep it as it is.
If you hit tons of retries due to massive concurrent updates, you can revert to partitioning your saga.
I'm using Azure Table storage for persistence and Azure Service Bus for transport and I would like to know what transactions are in place within a saga's handler? Is it the same as a normal handler?
I'm asking because I'm seeing database changes (SqlBulkCopy which normally enlists in the ambient transaction) happening multiple times. I'm accessing the database directly from the Saga in this scenario to 'single thread' the handling of the messages, but it doesn't seem to be working.
Azure Service Bus transport supports the following transport transactions levels only
SendsAtomicWithReceive (default)
ReceiveOnly
None
It does not support Transaction Scope level. Which is what you're looking for.
Why is that? Azure Service Bus does not allow any ambient transaction to take place. Any business-related data operation will be excluded from the handler transaction. To avoid duplicate business data writes you will need to implement idempotency to ensure the same business data is not written more than once when messages are retried.
When my system's data changes I publish every single change to at least 4 different consumers (around 3000 messages a second) so I want to use a message broker.
Most of the consumers are responsible to update their database tables with the change.
(The DBs are different - couch, mysql, etc therefor solutions such as using their own replication mechanism or using db triggers is not possible)
questions
Does anyone have an experience with data replication between DBs using a message broker?
is it a good practice?
What do I do in case of failures?
Let's say, using RabbitMQ, the client removed 10,000 messages from the queue, acked, and threw an exception each time before handling them. Now they are lost. Is there a way to go back in the queue?
(re-queueing them will mess their order ).
Is using rabbitMQ a good practice? Isn't the ability to go back in the queue as in Kafka important to fail scenarios?
Thanks.
I don't have experience with DB replication using message brokers, but maybe this can help put you in the right track:
2. What do I do in case of failures?
Let's say, using RabbitMQ, the client removed 10,000 messages from the
queue, acked, and threw an exception each time before handling them.
Now they are lost. Is there a way to go back in the queue?
You can use dead lettering to avoid losing messages. I'd suggest to not ack until you are sure the consumers have processed them successfully, unless it is a long-running task. In case of failure, basic.reject instead of basic.ack to send them to a dead-letter queue. You have a medium throughput, so gotta be careful with that.
However, the order is not guaranteed. You'll need to implement a manual mechanism to recover them in the order they were published, maybe by using message headers with some sort of timestamp or id mechanism, to re-process them in the correct order.
What is the intended usage of ForwardRecievedMessagesTo?
I read some where that it is to support auditing. Is there any harm in using it as a solution to ensure that messages have been processed and if not reprocessing them? lets say a message was sent to queue_A#server_A and also forwarded to q_All#server_All and before the message was handled, machine_A died irrecoverably. In such a case, I could have a handler pick up messages from q_All#sever_All and check against a database table if the message has been processed. If not reprocess(publish or send) the message or save it in a database table.
Also, what is the performance implication of using forwardreceivedmessageto? How is it different from journalling?
Yes, I am trying to not use msmq clustering.
The feature is there to support auditing. If your machine dies during processing then the messages will backup at the sending machine and would continue to flow after the machine recovered. This means you must size the disk on the sending machine appropriately. You could leverage auditing to accomplish this and the overhead would be minimal. The implication would be the time it would take to complete the distributed transaction to the other machine where your audit queue lives which should be very small.
Does NServiceBus automatically attempt to redeliver messages if handling fails? And if it does, is there a limit in the number of times delivery can be attempted?
NSB will enlist in a distributed transaction and if it fails it will retry the configured number of times. Look at the MsmqTransport config section.
EDIT: A distributed transaction begins as soon as you peek or receive a message from MSMQ. All of the work you do in a message handler will be included in the transaction and it is governed by the Distributed Transaction Coordinator. The DTC will also include things like DB transactions if you are updating DBs and so on.
If say a DB update fails, the whole thing rolls back and the message is put back on the queue.