I have a process whereby an admin must be alerted and the message automatically retried if some business logic is not meet.
Currently what I did is I throw and Exception to force NServiceBus to retry the message.
I have a feeling this is not what I am supposed to do. Is this the proper way of doing it?
public void Handle(ImportantCmd message)
{
//do some awesome business logic here
..a business logic is not meet..
//send email alert in case of error
Bus.Publish<SendEmailCmd>(email =>
{
email.To = "pooradmin#awesomecompany.com";
email.Title = "Important title";
email.Body = "Important message";
});
//then force NServiceBus to retry
throw new Exception("Blah blah...., retrying this message.");
}
Update: I would like an admin to be alerted whenever some condition is not met and he/she should be able to see all messages that are affected (perhaps in a dedicated queue?) and possibly retry them.
Basically our service depends on an external service. This external service occasionally could return erroneous respond (but if we retry it might work). That is why I am alerting the admin and at the same time retrying them.
Given your update (i'm assuming the admin will not alter the message) i would say you can use the FLR (First Level retry) and SLR(Second Level Retry) to retry the messages as the web service you are calling will eventually be able to process your message.
If that fails, the message will end up in the error queue.
You can monitor the error queue, by polling ServiceControl using it's API (if you use the platform installer it will install ServiceControl with NServiceBus) or subscribing to the MessageFailed event ServiceControl is publishing like this spike code more on David's blog .
Here is a link about SLR
Check Out David's book
The retry mechanism of NServiceBus (driven by throwing an exception) is supposed to be for infrastructure problems (deadlocks, servers unavailable, outright bugs, etc.) that a developer would need to look at. That way transient failures (deadlocks, web service down) is taken care of on an automatic retry, and permanent errors (whoops looks like I divided by zero!) go to an error queue for a developer to figure out and take administrative action.
Now, if your endpoint is transactional, your code above will not work as expected because either everything in the message handler is in the transaction. That means if you throw an exception, your Bus.Publish (or Bus.Send, and you can't/shouldn't publish a command) will not actually happen.
Really, I don't understand what sort of business logic would require an alert and a retry. Can you elaborate? What is it that makes your business logic so non-deterministic based on the incoming message? And can anything be done about that?
But at the end of the day, this business logic sounds like it's part of a business process, which should stay expressed in messages, not in errors and retry. So if a condition means you need to notify someone and so something else, publish a ThingHappened event (a subscriber can send an email) and then have another handler do whatever is necessary to handle that business process. If that means that, in the future, a new command comes through with largely the same data, then so be it.
Related
I am trying to understand what is the right pattern to deal with RabbitMQ deliveries in the context of distributed database transaction.
To make this simple, I will illustrate my ideas in pseudocode, but I'm in fact using Spring AMQP to implement these ideas.
Anything like
void foo(message) {
processMessageInDatabaseTransaction(message);
sendMessageToRabbitMQ(message);
}
Where by the time we reach sendMessageToRabbitMQ() the processMessageInDatabaseTransaction() has successfully committed its changes to the database, or an exception has been thrown before reaching the message sending code.
I know that for the sendMessageToRabbitMQ() I can use Rabbit transactions or publisher confirms to guarantee that Rabbit got my message.
My interest is understanding what should happen when things go south, i.e. when the database transaction succeeded, but the confirmation does not arrive after certain amount of time (with publisher confirms) or the Rabbit transaction fails to commit (with Rabbit transaction).
Once that happens, what is the right pattern to guarantee delivery of my message?
Of course, having developed idempotent consumers, I have considered that I could retry the sending of the messages until Rabbit confirms success:
void foo(message) {
processMessageInDatabaseTransaction(message);
retryUntilSuccessFull {
sendMessagesToRabbitMQ(message);
}
}
But this pattern has a couple of drawbacks I dislike, first, if the failure is prolonged, my threads will start to block here and my system will eventually become unresponsive. Second, what happens if my system crashes or shuts down? I will never deliver these messages then since they will be lost.
So, I thought, well, I will have to write my messages to the database first, in pending status, and then publish my pending messages from there:
void foo(message) {
//transaction commits leaving message in pending status
processMessageInDatabaseTransaction(message);
}
#Poller(every="10 seconds")
void bar() {
for(message in readPendingMessagesFromDbStore()) {
sendPendingMessageToRabbitMQ(message);
if(confirmed) {
acknowledgeMessageInDatabase(message);
}
}
}
Possibly sending the messages multiple times if I fail to acknowledge the message in my database.
But now I have introduced other problems:
The need to do I/O from the database to publish a message that 99% time would have successfully being published immediately without having to check the database.
The difficulty of making the poller closer to real time delivery since now I have added latency to the publication of the messages.
And perhaps other complications like guarantee delivery of events in order, poller executions stepping into one another, multiple pollers, etc.
And then I thought well, I could make this a bit more complicated like, I can publish from the database until I catch up with the live stream of events and then publish real time, i.e. maintain a buffer of size b (circular buffer) as I read based on pages check if that message is in buffer. If so then switch to live subscription.
To this point I realized that how to do this right is not exactly evident and so I concluded that I need to learn what are the right patterns to solve this problem.
So, does anyone has suggestions on what is the right ways to do this correctly?
While RabbitMQ cannot participate in a truly global (XA) transaction, you can use Spring Transaction management to synchronize the Database transaction with the Rabbit transaction, such that if either update fails, both transactions will be rolled back. There is a (very) small timing hole where one might commit but not the other so you do need to deal with that possibility.
See Dave Syer's Javaworld Article for more details.
When Rabbit fails to receive a message (for whatever reason, but in my experience only because the service is down or unavailable) you should be in a position to catch an error. At this point, you can make a record of that - and any subsequent - failed attempt in order to retry when Rabbit becomes available again. The quickest way of doing this is just logging the message details to file, and iterating over to re-send when appropriate.
As long as you have that file, you've not lost your messages.
Once messages are inside Rabbit, and you have faith in the rest of the architecture, it should be safe to assume that messages will end up where they are supposed to be, and that no further persistence work needs doing at your end.
I understand NServiceBus's retry mechanism to be primarily for connectivity problems or database deadlock problems, which is great and I love it for that.
However, I would like to configure NServiceBus' retry mechanism to not bother with a retry if the exception is typeof(ApplicationException). My code throws this kind of exception when there is a broken business rule (like a customer on hold), so no matter how many times this message is retried by NServiceBus' quick-retry mechanism, it will fail. This scenario requires that users take action on the data and then use ServiceInsight to re-queue the message for processing.
Can this be done?
I would reconsider using your application logic to inform users about this type of errors using Reply or Return in your handler, that should be located in the catch (ApplicationException) section. Then users change the data and send the message again using your application, not ServiceInsight. In this case, do not re-throw the ApplicationException in your catch block and this will prevent NServiceBus from retrying your message handling.
My requirement is to make the Subscriber pause processing the messages depending on whether a web service is up or not. So, when the web service is down, the messages should keep coming to the subscriber queue from Publisher and keep piling up until the web service is up again. (These messages should not go to the error queue, but stay on the Subscriber queue.)
I tried to use unsubscribe, but the publisher stops sending messages as the unsubscribe seems to clear the subscription info on RavenDB. I have also tried setting the MaxConcurrencyLevel on the Transport class, if I set the worker threads to 0, the messages coming to Subscriber go directly to the error queue. Finally, I tried Defer, which seems to put the current message in audit queue and creates a clone of the message and sends it locally to the subscriber queue when the timeout is completed. Also, since I have to keep checking the status of service and keep defering, I cannot control the order of messages as I cannot predict when the web service will be up.
What is the best way to achieve the behavior I have explained? I am using NServiceBus version 4.5.
It sounds like you want to keep trying to handle a message until it succeeds, and not shuffle it back in the queue (keep it at the top and keep trying it)?
I think your only pure-NSB option is to tinker with the MaxRetries setting, which controls First Level Retries: http://docs.particular.net/nservicebus/msmqtransportconfig. Setting MaxRetries to a very high number may do what you are looking for, but I can't imagine doing so would be a good practice.
Second Level Retries will defer the message for a configurable amount of time, but IIRC will allow other messages to be handled from the main queue.
I think your best option is to put retry logic into your own code. So the handler can try to access the service x number of times in a loop (maybe on a delay) before it throws an exception and NSB's retry features kick in.
Edit:
Your requirement seems to be something like:
"When an MyEvent comes in, I need to make a webservice call. If the webservice is down, I need to keep trying X number of times at Y intervals, at which point I will consider it a failure and handle a failure condition. Until I either succeed or fail, I will block other messages from being handled."
You have some potentially complex logic on handling a message (retry, timeout, error condition, blocking additional messages, etc.). Keep in mind the role that NSB is intended to play in your system: communication between services via messaging. While NSB does have some advanced features that allow message orchestration (e.g. sagas), it's not really intended to be used to replace Domain or Application logic.
Bottom line, you may need to write custom code to handle your specific scenario. A naive solution would be a loop with a delay in your handler, but you may need to create a more robust in-memory collection/queue that holds messages while the service is down and processes them serially when it comes back up.
The easiest way to achieve somewhat the required behavior is the following:
Define a message handler which checks whether the service is available and if not calls HandleCurrentMessageLater and a message handler which does the actual message processing. Then you specify the message handler order so that the handler which checks the service availability gets executed first.
public interface ISomeCommand {}
public class ServiceAvailabilityChecker : IHandleMessages<ISomeCommand>{
public IBus Bus { get; set; }
public void Handle(ISomeCommand message) {
try {
// check service
}
catch(SpecificException ex) {
this.Bus.HandleCurrentMessageLater();
}
}
}
public class ActualHandler : IHandleMessages<ISomeCommand>{
public void Handle(ISomeCommand message) {
}
}
public class SomeCommandHandlerOrdering : ISpecifyMessageHandlerOrdering{
public void SpecifyOrder(Order order){
order.Specify(First<ServiceAvailabilityChecker>.Then<ActualHandler>());
}
}
With that design you gain the following:
You can check the availability before the actual business code is invoked
If the service is not available the message is put back into the queue
If the service is available and your business code gets invoked but just before the ActualHandler is invoked the service becomes unavailable you get First and Second Level retries (and again the availability check in the pipeline)
I have a service that handles messages that persists data to an external system. If (a.k.a. when) the writing of this data to the external system fails, or normal monitoring strategy will alert system admins of the failure.
I would like to also notify the user who submitted the message that there is a delay in processing their request.
Where/How is the best way to accomplish this scenario? I've looked into the IManageMessageFailures, but it seems that will bypass the SLR functionality.
Starting with NServiceBus version 5.1 now has the ability to use Reactive Extensions to observe when a message is sent to an error queue. From there, you can log, email, or whatever best meets your needs.
http://docs.particular.net/nservicebus/subscribing-to-push-based-error-notifications
Why don't you try and separate the two concerns?
Manage the 3rd party interaction in a saga, and if it fails, send a failure notification message (you can use timeout to cater for no proper reply).
Well I've been doing this NServiceBus project for a while and once I got it working for PubSub I then spent the rest of the time on the actual workflow logic. However, I can see a serious issue which I want to get around (or rather learn how to handle correctly).
A publisher publishes a message to the storage queues of any subscribers as far as I understand. Great. But what happens when the subscriber isn't running (I've read other posts about this and they don't seem to be asking the same question).
Scenario - I get the publisher to Publish a message when no subscribers are running (attached/requested messages to be relayed to them).. I then find that.. the message is "gone" just simply isn't there! where did it go? Did the publisher say "hey, no one's subscribing to this, so I wont bother publishing it?", shouldn't it NOT do that and require at least one subscriber?
Can anyone shed any light on this? (nservicenewbie)
You should publish an event that has happened - a statement of fact, that other handler may or may not be interested in. It's perfectly valid to have zero subscribers! If this is not the case then maybe you should be Send()ing a command instead of Publish()ing an event.
If you are using a persistent subscription storage, start the subscriber up once and it will always be subscribed. If the subscriber is offline, messages for it will pile up in its Input Queue, ready to be processed when the subscriber comes back online.
If you're just testing with NServiceBus, the NServiceBus.Host.exe is running in the Lite profile, which uses in-memory (non-persistant) subscription storage, which would result in what you are seeing.
Ah ha! Well though it's not always an error to have no subscriber for a message type, there is a way to handle it.
In your publisher simply modify the:
IBus Bus
To use (you will need NServiceBus.Core.dll and the NS NServiceBus.Unicast):
IUnicastBus Bus
Then you can attach an handler to:
Bus.NoSubscribersForMessage += .......
This can then put the message in an error queue.. or perhaps retry forever.. or publish something else etc.. etc.. what ever you want. Thus ensuring there is nothing lost where your particular system (from a business perspective) requires an outcome