When using NHibernate in web applications, I will usually let my IoC container take care of opening and closing an ISession per request and commit/rollback the transaction. The nature of HTTP makes it very easy to define a clear Unit-of-Work in such applications.
Now, I have been tasked with putting together a small program, which will be invoked regularly by a task scheduler, for sending out newsletters. The concepts of both newsletters and subscribers are already well defined entities in our domain model, and sending a newsletter to all subscribers would involve doing something similar to this:
var subscribers = _session
.QueryOver<Subscription>()
.Where(s => !s.HasReceivedNewsletter)
.List();
foreach (var subscriber in subscribers)
{
SendNewsletterTo(subscriber);
subscriber.HasReceivedNewsletter = true;
}
Notice how each Subscriber object is updated within the loop, recording that she has now received the newsletter. The idea is, that if the mail sending program should crash, it can be restarted and continue sending newsletters from where it left off.
The problem I am facing, is in defining and implementing the Unit-of-Work pattern here. I will probably need to commit changes to the database by the end of each iteration of the loop. Simply wrapping the loop body with a using (var trans = _session.BeginTransaction()) block seems to be extremely expensive in running time, and I also seem to experience locking issues between this long running process and other (web) applications using the same database.
After reading some articles and documentation on NHibernate transactions, I have come to think, that I might need to detach the list of subscribers from the session to avoid the locking issues, and reattach each to a fresh session in the loop body. I am not sure how this will work for performance, though.
So, NHibernate experts, how would you design and implement a long running job like this?
Don't you want to use asynchronous durable messaging here? Something like NServiceBus, Rhino Service Bus or MassTransit. It seems you don't have to send a lot of messages as soon as possible, so I think you should do it asynchronously with 1 durable message per user basis
Don't you think that Stateless session with no transaction will do better here?
There's no problem having multiple transactions in a session. It's appropriate here to scope the transaction to updating a single subscriber because it's an independent operation. Depending on the number of subscribers and the likelihood of failure, it might be best to grab a small number of subscribers at a time.
foreach (var subscriber in subscribers)
{
using (var txn = _session.BeginTransaction())
{
try
{
SendNewsletterTo(subscriber);
subscriber.HasReceivedNewsletter = true;
txn.Commit();
}
catch (Exception ex)
{
txn.Rollback();
// log exception, clean up any actions SendNewsletterTo has taken if needed
// Dispose of session and start over
}
}
}
Related
I have a situation where I have an asp.net core application which registers a subscription client to a topic on startup (IHostedService), this subscription client essentially has a dictionary of callbacks that need to be fired whenever it detects a new message in a topic with an id (this id is stored on the message properties). This dictionary lives throughout the lifetime of the application, and is in memory.
Everything works fine on a single instance of the asp.net core app service on azure, as soon as I scale up to 2, I notice that sometimes the callbacks in the subscription are not firing. This makes sense, as we have two instances now, each with its own dictionary store of callbacks.
So I updated the code to check if the id of the subscription exists, if not, abandon message, if yes, get the callback and invoke it.
public async Task HandleMessage(Microsoft.Azure.ServiceBus.Message message, CancellationToken cancellationToken)
{
var queueItem = this.converter.DeserializeItem(message);
var sessionId = // get the session id from the message
if (string.IsNullOrEmpty(sessionId))
{
await this.subscriptionClient.AbandonAsync(message.SystemProperties.LockToken);
return;
}
if (!this.subscriptions.TryGetValue(sessionId, out var subscription))
{
await this.subscriptionClient.AbandonAsync(message.SystemProperties.LockToken);
return;
}
await subscription.Call(queueItem);
// subscription was found and executed. Complete message
await this.subscriptionClient.CompleteAsync(message.SystemProperties.LockToken);
}
However, the problem still occurs. My only guess is that when calling AbandonAsync, the same instance is picking up the message again?
I guess what I am really trying to ask is, if I have multiple instances of a topic subscription client all pointing to the same subscriber for the topic, is it possible for all the instances to get a copy of the message? Or is that not guaranteed.
if I have multiple instances of a topic subscription client all pointing to the same subscriber for the topic, is it possible for all the instances to get a copy of the message? Or is that not guaranteed.
No. If it's the same subscription all clients are pointing to, only one will be receiving that message.
You're running into an issue of scaling out with competing consumers. If you're scaling out, you never know what instance will pick the message. And since your state is local (in memory of each instance), this will fail from time to time. Additional downside is the cost. By fetching messages on the "wrong" instance and abandoning, you're going to pay higher cost on the messaging side.
To address this issue you either need to have a shared/centralized or change your architecture around this.
I managed to solve the issue by making use of service bus sessions. What I was trying to do with the dictionary of callbacks is basically a session manager anyway!
Service bus sessions allow me to have multiple instances of a session client all pointing to the same subscription. However, each instance will only know or care about the sessions it is currently dealing with.
I am trying to understand what is the right pattern to deal with RabbitMQ deliveries in the context of distributed database transaction.
To make this simple, I will illustrate my ideas in pseudocode, but I'm in fact using Spring AMQP to implement these ideas.
Anything like
void foo(message) {
processMessageInDatabaseTransaction(message);
sendMessageToRabbitMQ(message);
}
Where by the time we reach sendMessageToRabbitMQ() the processMessageInDatabaseTransaction() has successfully committed its changes to the database, or an exception has been thrown before reaching the message sending code.
I know that for the sendMessageToRabbitMQ() I can use Rabbit transactions or publisher confirms to guarantee that Rabbit got my message.
My interest is understanding what should happen when things go south, i.e. when the database transaction succeeded, but the confirmation does not arrive after certain amount of time (with publisher confirms) or the Rabbit transaction fails to commit (with Rabbit transaction).
Once that happens, what is the right pattern to guarantee delivery of my message?
Of course, having developed idempotent consumers, I have considered that I could retry the sending of the messages until Rabbit confirms success:
void foo(message) {
processMessageInDatabaseTransaction(message);
retryUntilSuccessFull {
sendMessagesToRabbitMQ(message);
}
}
But this pattern has a couple of drawbacks I dislike, first, if the failure is prolonged, my threads will start to block here and my system will eventually become unresponsive. Second, what happens if my system crashes or shuts down? I will never deliver these messages then since they will be lost.
So, I thought, well, I will have to write my messages to the database first, in pending status, and then publish my pending messages from there:
void foo(message) {
//transaction commits leaving message in pending status
processMessageInDatabaseTransaction(message);
}
#Poller(every="10 seconds")
void bar() {
for(message in readPendingMessagesFromDbStore()) {
sendPendingMessageToRabbitMQ(message);
if(confirmed) {
acknowledgeMessageInDatabase(message);
}
}
}
Possibly sending the messages multiple times if I fail to acknowledge the message in my database.
But now I have introduced other problems:
The need to do I/O from the database to publish a message that 99% time would have successfully being published immediately without having to check the database.
The difficulty of making the poller closer to real time delivery since now I have added latency to the publication of the messages.
And perhaps other complications like guarantee delivery of events in order, poller executions stepping into one another, multiple pollers, etc.
And then I thought well, I could make this a bit more complicated like, I can publish from the database until I catch up with the live stream of events and then publish real time, i.e. maintain a buffer of size b (circular buffer) as I read based on pages check if that message is in buffer. If so then switch to live subscription.
To this point I realized that how to do this right is not exactly evident and so I concluded that I need to learn what are the right patterns to solve this problem.
So, does anyone has suggestions on what is the right ways to do this correctly?
While RabbitMQ cannot participate in a truly global (XA) transaction, you can use Spring Transaction management to synchronize the Database transaction with the Rabbit transaction, such that if either update fails, both transactions will be rolled back. There is a (very) small timing hole where one might commit but not the other so you do need to deal with that possibility.
See Dave Syer's Javaworld Article for more details.
When Rabbit fails to receive a message (for whatever reason, but in my experience only because the service is down or unavailable) you should be in a position to catch an error. At this point, you can make a record of that - and any subsequent - failed attempt in order to retry when Rabbit becomes available again. The quickest way of doing this is just logging the message details to file, and iterating over to re-send when appropriate.
As long as you have that file, you've not lost your messages.
Once messages are inside Rabbit, and you have faith in the rest of the architecture, it should be safe to assume that messages will end up where they are supposed to be, and that no further persistence work needs doing at your end.
i have a azure worker role with an nservicebus host 4.7.5 . This host sends events on azure servicebus transport and to a topic. Is there a way to either delay the sending of the event or setting some properties to make sure that the message appears after a delay on the topic subscription? The host sends out events after it notices a change in the primary database. There are several secondary databases into which the primary data write is replicated. The receivers are also azure worker roles that use nservicebus host and have subscription to the topics.
By the time the receivers receive the message, due to replication lag the secondaries may have out of sync data.
one option is to use primary database to read but that is a route which i dont want to take.
Would it be possible to fail-early in your subscription endpoints and let the retries take care of it? You can fine-tune the retry times/delays to make sure your secondary databases are updated before the message is retried.
You still need to find the best way to lookup your data from the database and a way to differentiate between the version in the event. You could use version numbers or last update dates in case of updates, or just lookup by an identifier in case of creation.
The endpoint reading data off the secondary database might have a event handler like this:
public class CustomerCreationHandler : IHandlesMessage<CustomerCreated>
{
public void Handle(CustomerCreated #event)
{
var customer = Database.Load(#event.CustomerId);
if(customer == null)
{
throw new CustomerNotFoundException("Customer was not found.");
}
//Your business logic goes here
}
}
You can control how many times the event handler will retry and how much delay there'll be between each attempt. In this case, the message will be retried by First-Level retries and then handed over to Second-Level retries which is configured below.
class ProvideConfiguration :
IProvideConfiguration<SecondLevelRetriesConfig>
{
public SecondLevelRetriesConfig GetConfiguration()
{
return new SecondLevelRetriesConfig
{
Enabled = true,
NumberOfRetries = 2,
TimeIncrease = TimeSpan.FromSeconds(10)
};
}
}
Alternatively, instead of just publishing the event, you can send a deferred message to the same endpoint to then publish the actual event after certain amount of time is passed.
I have a WCF MSMQ service (hosted in a windows service). My method has the TransactionScopeRequired attribute on it.
I am using Nhibernate to save my data to my database. I want to make sure I close each Nhibernate session after each call.
I was using the following approach (using the castle facility) in my data access
using(var session = sessionManager.OpenSession())
using(var transaction = session.BeginTransaction())
{
// do work
transaction.Commit()
}
But when my main service method exits I am getting an error because I have already disposed of the Nhibernate session and I think the DTC needs this to do its commit.
My question is:
What would be the best way to close the Nhibernate session - after the DTC has committed (i.e after i have exited my service method?).
Thank you.
If you wrap your code in the following
using (TransactionScope sc = new TransactionScope(TransactionScopeOption.Suppress))
{
// code here
sc.Complete();
}
Then NHibernate will not enlist in the ambient transaction and therefore DTC will not have a dependency on the database transaction.
This is a hunch as you haven't supplied the error details in your question.
EDIT
Of course by following this advice your database commit will not be performed under the same transaction as the dequeue action so if there is a failure in your database this may or may not cause the dequeue transaction to roll the message being processed back onto the queue, so you risk dropping messages in this way. You can compensate for this in various ways or you can just run the risk if the cost of dropped messages are not high.
The below text is an effort to expand and add color to this question:
How do I prevent a misbehaving client from taking down the entire service?
I have essentially this scenario: a WCF service is up and running with a client callback having a straight forward, simple oneway communication, not very different from this one:
public interface IMyClientContract
{
[OperationContract(IsOneWay = true)]
void SomethingChanged(simpleObject myObj);
}
I'm calling this method potentially thousands of times a second from the service to what will eventually be about 50 concurrently connected clients, with as low latency as possible (<15 ms would be nice). This works fine until I set a break point on one of the client apps connected to the server and then everything hangs after maybe 2-5 seconds the service hangs and none of the other clients receive any data for about 30 seconds or so until the service registers a connection fault event and disconnects the offending client. After this all the other clients continue on their merry way receiving messages.
I've done research on serviceThrottling, concurrency tweaking, setting threadpool minimum threads, WCF secret sauces and the whole 9 yards, but at the end of the day this article MSDN - WCF essentials, One-Way Calls, Callbacks and Events describes exactly the issue I'm having without really making a recommendation.
The third solution that allows the service to safely call back to the client is to have the callback contract operations configured as one-way operations. Doing so enables the service to call back even when concurrency is set to single-threaded, because there will not be any reply message to contend for the lock.
but earlier in the article it describes the issue I'm seeing, only from a client perspective
When one-way calls reach the service, they may not be dispatched all at once and may be queued up on the service side to be dispatched one at a time, all according to the service configured concurrency mode behavior and session mode. How many messages (whether one-way or request-reply) the service is willing to queue up is a product of the configured channel and the reliability mode. If the number of queued messages has exceeded the queue's capacity, then the client will block, even when issuing a one-way call
I can only assume that the reverse is true, the number of queued messages to the client has exceeded the queue capacity and the threadpool is now filled with threads attempting to call this client that are now all blocked.
What is the right way to handle this? Should I research a way to check how many messages are queued at the service communication layer per client and abort their connections after a certain limit is reached?
It almost seems that if the WCF service itself is blocking on a queue filling up then all the async / oneway / fire-and-forget strategies I could ever implement inside the service will still get blocked whenever one client's queue gets full.
Don't know much about the client callbacks, but it sounds similar to generic wcf code blocking issues. I often solve these problems by spawning a BackgroundWorker, and performing the client call in the thread. During that time, the main thread counts how long the child thread is taking. If the child has not finished in a few milliseconds, the main thread just moves on and abandons the thread (it eventually dies by itself, so no memory leak). This is basically what Mr.Graves suggests with the phrase "fire-and-forget".
Update:
I implemented a Fire-and-forget setup to call the client's callback channel and the server no longer blocks once the buffer fills to the client
MyEvent is an event with a delegate that matches one of the methods defined in the WCF client contract, when they connect I'm essentially adding the callback to the event
MyEvent += OperationContext.Current.GetCallbackChannel<IFancyClientContract>().SomethingChanged
etc... and then to send this data to all clients, I'm doing the following
//serialize using protobuff
using (var ms = new MemoryStream())
{
ProtoBuf.Serializer.Serialize(ms, new SpecialDataTransferObject(inputData));
byte[] data = ms.GetBuffer();
Parallel.ForEach(MyEvent.GetInvocationList(), p => ThreadUtil.FireAndForget(p, data));
}
in the ThreadUtil class I made essentially the following change to the code defined in the fire-and-foget article
static void InvokeWrappedDelegate(Delegate d, object[] args)
{
try
{
d.DynamicInvoke(args);
}
catch (Exception ex)
{
//THIS will eventually throw once the client's WCF callback channel has filled up and timed out, and it will throw once for every single time you ever tried sending them a payload, so do some smarter logging here!!
Console.WriteLine("Error calling client, attempting to disconnect.");
try
{
MyService.SingletonServiceController.TerminateClientChannelByHashcode(d.Target.GetHashCode());//this is an IContextChannel object, kept in a dictionary of active connections, cross referenced by hashcode just for this exact occasion
}
catch (Exception ex2)
{
Console.WriteLine("Attempt to disconnect client failed: " + ex2.ToString());
}
}
}
I don't have any good ideas how to go and kill all the pending packets the server is still waiting to see if they'll get delivered on. Once I get the first exception I should in theory be able to go and terminate all the other requests in some queue somewhere, but this setup is functional and meets the objectives.