I'm using Azure Table storage for persistence and Azure Service Bus for transport and I would like to know what transactions are in place within a saga's handler? Is it the same as a normal handler?
I'm asking because I'm seeing database changes (SqlBulkCopy which normally enlists in the ambient transaction) happening multiple times. I'm accessing the database directly from the Saga in this scenario to 'single thread' the handling of the messages, but it doesn't seem to be working.
Azure Service Bus transport supports the following transport transactions levels only
SendsAtomicWithReceive (default)
ReceiveOnly
None
It does not support Transaction Scope level. Which is what you're looking for.
Why is that? Azure Service Bus does not allow any ambient transaction to take place. Any business-related data operation will be excluded from the handler transaction. To avoid duplicate business data writes you will need to implement idempotency to ensure the same business data is not written more than once when messages are retried.
Related
Hi I have with this code
SendMessageToMyQueue();
UpdateStatusInDbThatMessageWasSent();
that sometimes message is processed before status is updated which I would like to avoid.
My question is if I wrap that two line with a transaction like this:
using(var tr = new TransactionScope())
{
SendMessageToMyQueue();
UpdateStatusInDbThatMessageWasSent();
tr.Compleate();
}
will be guaranteed that there will be a lock on MyQueue created and this lock will be not released until UpdateStatusInDbThatMessageWasSent will update the status?
also if I add try catch with rollback and updating status fails, will the message be removed from MyQueue ?
There is no such thing as lock on a queue. The message, however, will be processed transactionally, if the following conditions are met. By transactionally, I mean that the message will be returned to the queue if an unhandled exception is thrown. The conditions to make this happen are:
Your database can enlist and take part in a distributed transaction. Not every database out there does. Some Document databases have none (in case of MongoDB) or sketchy (in case of RavenDB) support for DTC.
Your transport also supports distributed transactions. If you go with a broker type transports, SQL Server Transport is your best bet and on Bus type transports MSMQ is a good choice. Transports like Azure ServiceBus or RabbitMQ have very limited transactions support and do not support distributed transactions.
You'll need to run Distributed Transaction Coordinator service configured and running.
Two other things to note:
What if you're using a transport that lacks DTC support? Most of the time, you are better off if you can design your system to be idempotent. Outbox feature of NServiceBus allows you simulate DTC to some extent.
When a message is picked from the queue, processed, and returned to the queue due to an exception, it might end up being in a different place in the queue. You need to design for messages arriving out of order when designing a message-based architecture.
With all said above, exactly-once delivery guarantees are always a hot topic and disputed.
I was trying to get some info on WCF transactions, and I did manage to get info on how to use them. What I didn't get much info on is Why/When to use them.
What is the difference between database transactions and WCF transactions? Is there any specific case when either of these approach is preferred over the other?
By WCF transactions what you are really asking about is Microsoft's implementation of the WS-AtomicTransaction web service extension standard.
Why/When to use them
Similar to using a database transaction to guarantee consistency within the database, WS-AtomicTransaction is used to guarantee consistency within a larger, distributed system, based on communication over SOAP 1.2 services. This distributed system may or may not include database writes, but more often than not it will do.
Transactions propagated to the service from clients will cause the internal code of the service to execute within the context of the clients transaction.
So, in that same way a database transaction can wrap multiple database writes into a single unit of work, a WCF transaction can wrap multiple service calls into a single unit of work, so that a failure in one will roll back the others.
This, as you can imagine, is hugely costly from a resource perspective, so these kinds of cross-network transactions should be rarely (if ever) used unless absolutely necessary.
Quick overview of our topology:
Web sites sending commands to an nServiceBus server, which accepts the commands and then publishes the correct pub/sub events. This service also has message handlers that can do some process against the DB in response to the command, for instance:
1 user registers on web site
2 web site sends nServicebus command to nServicebus service on another server.
3 nServicebus server has a handler for that specific type of command, which logs something to the database and sends a welcome email
Since instituting this architecture we started to get deadlocks on the DB. I have traced it down to MSDTC on the database server. If I turn that service OFF on the database server nServicebus starts throwing up errors, which to me shows that nServiceBus has been enlisting the DB update in the transaction.
I don't wish this to happen, I want to handle the DB failing myself, I only want the transaction to ensure the message is delivered to my nServicebus proxy service. I don't want a transaction from the web all the way through 2 servers to the DB and back.
Any suggestions?
EDIT: this post provides some clues, however I'm not entirely sure it's the proper way to proceed.. NServiceBus - Problem with using TransactionScopeOption.Suppress in message handler
EDIT2: The reason that we want the DB work outside the scope of the transaction is that the intent is to 'asynchronously' process these commands on another server so as not to slow down the web site and/or cause users to wait for these long running aggregation commands. If the DB is within the scope of the transaction, is that blocking execution on the website at the point where the original command is fired to the distributor? Is there a better nServicebus architecture for this scenario? We want the command to fire quickly and return control to the web site so the user can quickly proceed and not have to wait for our longish running DB command, which is updating aggregate counts and sending emails etc.
I wouldn't recommend having the DB work outside the context of the NServiceBus transaction. Instead, try reducing the isolation level of the transactions. This can be done by calling:
.IsolationLevel(System.Transactions.IsolationLevel.ReadCommited)
in the fluent configuration. You'll have to put this after .MsmqTransport() in v2.6. In v3.0 you can put this call almost anywhere.
RESPONSE TO EDIT2:
Just using NServiceBus will achieve your objective of not slowing down the website, regardless of the level of the transactions run on the other server. The use of transactions is to provide a guarantee that messages won't be lost in case of failure and also that you won't have to write your own deduplication logic.
We are using JPA 1.0 for ORM based operations and we want to have JTA datasource for our application. We are having only 1 database to which our application will connect.
We start our transaction boundary in controller class and it goes till DAO layer controller--> BOImpl--> DAO.
In websphere application server admin console when I am defining datasource should I use non-XA datasource or XA-Datasource.
My understanding is that for single datasource I should not use XADatasource.
Please let me know what should I need to use.
For a single resource (like a single DB) you indeed do not need an XA-datasource.
On the other hand, bear in mind that most JTA/JTS implementations actually recognize that there is only 1 resource participating in a transactions, so the overhead for XA would be minimal or none then. There can also be additional participants in the transactions that you might now not think about, like sending JMS messages.
But if you're really sure you only have 1 resource participating, you can safely go for non-XA.
I hope your doubt may be clear by now, but here is more information on that just in case.
The typical XA resources are databases, messaging queuing products such as JMS or WebSphere MQ, mainframe applications, ERP packages, or anything else that can be coordinated with the transaction manager. XA is used to coordinate what is commonly called a two-phase commit (2PC) transaction. The classic example of a 2PC transaction is when two different databases need to be updated atomically. Most people think of something like a bank that has one database for savings accounts and a different one for checking accounts. If a customer wants to transfer money between his checking and savings accounts, both databases have to participate in the transaction or the bank risks losing track of some money.
The problem is that most developers think, "Well, my application uses only one database, so I don't need to use XA on that database." This may not be true. The question that should be asked is, "Does the application require shared access to multiple resources that need to ensure the integrity of the transaction being performed?" For instance, does the application use Java 2 Connector Architecture adapters or the Java Message Service (JMS)? If the application needs to update the database and any of these other resources in the same transaction, then both the database and the other resource need to be treated as XA resources.
I need to build Identity server like Microsoft's http://login.live.com.
To handle failover I will have multiple web servers nodes. The plan is that all database write operations are done by sending messages to the database server. Database will be mirrored or replicated. The idea is that database subscribes to the write operations but that other nodes subscribe also. That way other nodes do not need to read from database and can update their caches.
I am just starting to learn the service bus architecture and what is not clear to me is how to handle failover scenario for the service bus.
Question:
If database server is not available, what will happen with the published messages ?
Will they be stored somewhere and where ?
Do I need additional machine or a cluster to handle failover of the service bus?
I read that SQL Server can be used as a message store but can I use durable MSMQ? I am queuing messages to be able to write them to the database so why would I store them to the DB first just to take them and write them again? OR, I am getting this wrong and DB is only used for the list of subscriptions and not for the Messages?
Whe implementing this kind of architecture, you should look at applying the principles of CQRS - queries (is this user/pwd combo valid) should not be done via the bus; commands (change pwd, forgot pwd) are sent via the bus, not published as events. While internally you will likely use events to keep the command and query sides in sync, this doesn't involve the client.
Queries can be done using simple ado.net against the replicated-read-slaves of your DB - what's known as the persistent view model in CQRS. If you like, you can put some simple WCF in front of that too.
When using MSMQ, all messages are delivered via store-and-forward. That means that they're first stored on the client before being delivered to the server, so if the server is down, the messages sit on the client waiting. For fault-tolerance, you will want your messages to be recoverable (written to disk) - this is the default in NServiceBus but not the default of standard MSMQ (don't know about MassTransit). You don't need the database for this.
In NServiceBus, the bus is not installed on a separate machine so you don't need to deal with its availability independently of the rest of the system. It's only when you look at scaling our your command processing to more nodes that you might consider using the message-based load balancer in NServiceBus (called the distributor) which, for high availability, should be installed on a cluster or fault-tolerant hardware.
This will depend on how it is setup, but in MassTransit you can leave the subscription active so the message will still be delivered to the queue for the DB. When the DB is active again, you can read the messages in the queue.
Each service connected to a service bus, in MassTransit, has an active queue for itself. The messages will be stored there.
I think this is a "it depends"... MassTransit has support for other MQs than MSMQ but is really built around MSMQ. We have no experienced great support for things such as failover from MSMQ. However, everything will continue to run without fault if the subscription service (i.e. the bus) fails - the services already know who to talk to. It's only when a change in a consumer (subscribe or unsubscribe) where this becomes a problem. For me, that's an event that happens almost never.
With MassTransit, we use the DB to store the subscription states but all the messages are stored in MSMQ.
If you'd like more details in one of these responses or have additional questions about MT, you can join us on the mailing list: http://groups.google.com/group/masstransit-discuss.