I have an endpoint that has a message handler which does some FTP work.
Since the ftp process can take some time, I encapsulated the ftp method within a TransactionScope with TransactionScopeOption.Suppress to prevent the transaction timeout exceptions.
Doing this got rid of the timeout exceptions, however the handler was fired 5 times
(retries is set to 5 in my config file)
The files were ftp'd ok, but they were just ftp'd 5 times.
the handler look like it is re-fired after 10 or 11 minutes.
some test code looks as follows:
public void Handle(FtpMessage msg)
{
using (TransactionScope t = new TransactionScope(TransactionScopeOption.Suppress))
{
FtpFile(msg);
}
}
Any help would be greatly appreciated.
thanks.
If this truly is an FTP communication that cannot be completed within the transaction timeout, another approach would be to turn this into a Saga.
The Saga would be started by FtpMessage, and in the handler it would start the FTP work asynchronously, whether in another thread, or via another process, or whatever, and store enough information in saga data to be able to look up the progress later.
Then it would request a timeout from the TimeoutManager for however long makes sense. Upon receiving that timeout, it would look up the state in saga data, and check on the FTP work in progress. If it's completed, mark the saga as complete, if not, request another timeout.
Alternatively, you could have a process wrapping the FTP communication that hosts its own Bus but does not have any message handlers of its own. It could receive its FTP information via the command line (including requesting endpoint), do its work, and then send a message back to the requesting endpoint saying it is complete. Then you would not have to wait for a timeout to move on with the process.
I'd recommend configuring that endpoint as non-transactional rather than trying to suppress the transaction. Do that by including .IsTransactional(false) in your initialization code if self-hosting or by implementing IConfigureThisEndpoint, AsA_Client when using the generic host.
My guess is that by not completing the inner scope you're causing the outer scope, created by NSB, to rollback. This will cause NSB to retry your FtpMessage.
Try to add: t.Complete(); after your call to FtpFile and see if that does it for you.
Edit: After rereading your question I realized that this won't solve your timeout issue. Have you tried to increase the timeout? (10 min is the default maxValue in machine.config so you can't set it to higher without modifying machine.config)
Related
We have a long running saga process which includes timeouts, when the timeout kicks in the first thing that happens is a call to an external data source.
We're wondering if it's appropriate to query the source directly and have the timeout hit the error queues if the source is down (or if some other issue comes up), or should we have the timeout create and send a message which is handled and from there query the source (and thus the message hits error queues if a problem arises) and then reply back to the original sender
We feel our NserviceBus code is a bit overly complex and are looking for ways to simplify it, and we're wondering if this is a good chance to do so.
public void Timeout(TimeoutEvent event)
{
bus.send(ExternalServiceCallCmd cmd)
}
public void handle(ExternalServiceCallCmd cmd)
{
manager.CallToExternalService();
}
If the call to the external service fails, the ExternalServiceCallCmd gets retried and eventually end up on error queues.
We're wondering if we can simplify like this:
public void Timeout(TimeoutEvent event)
{
manager.CallToExternalService();
}
and if the call to the external service fails, the TimeoutEvent would be retried and end up on error queues if necessary
Well, the first rule of Sagas is don't do I/O in saga handler, send commands to worker handler instead...
Other than interacting with its own internal state, a saga should not access a database, call out to web services, or access other resources - neither directly nor indirectly by having such dependencies injected into it.
Full details are provided here: https://docs.particular.net/nservicebus/sagas/#accessing-databases-and-other-resources-from-a-saga
Does that answer your question?
Please feel free to contact support support#particular.net to get more details :-)
We are having an issue with recovery for messages originating from Sagas.
When a Saga sends a message for processing, the message handler can sometimes fail with an exception. We currently use a try/catch and when an exception is thrown, we "Reply" with a failed message to the Saga. The issue with this approach is that Recoverability retries don't happen since we are handling the error in the message handler.
My thought was to add custom logic to the pipeline and if the Command message implements some special Interface, the custom logic would send a failed message response to the Saga if an exception occurs (after the retries fails), but I'm not sure where to plug into the pipeline that would allow me to send messages after retries fails.
Is this a valid approach? If not, how can I solve for Saga to Handler failure messages after retries?
You can use immediate dispatch to not wait for a handler to complete.
However, I would like to suggest an alternate approach. Why not create a Timeout in the saga? If the reply from the processing-handler isn't received within a certain TimeSpan, you take an alternate path. The processing-handler gets 5 minutes and if it doesn't respond within 5 minutes, we do something else. If it still responds after 6 minutes, we know we've already taken the alternate path (use a boolean flag or so and store that inside the saga data) and put aside the reply that arrived too late.
If you want to start a discussion based on this, check our community platform.
I have an NServiceBus handler that creates a new sql connection and new sql command.
However, the command that is executed is not being committed to the database until after the whole process is finished.
It's like there is a hidden sql transaction in the handler itself.
I moved my code into a custom console application without nservicebus and the sql command executed and saved immediately. Unlike in nservicebus where it doesn't save until the end of the handler.
Indeed every handler is wrapped in a transaction, the default transaction guarantee is relying on DTC. That is intentional :)
If you disable it then you might get duplicate messages or lose some data, so that must be done carefully. You can disable transactions using endpoint configuration API instead of using options in connection string.
Here you can find more information about configuration and available guarantees http://docs.particular.net/nservicebus/transports/transactions.
Unit of work
Messages should be processed as a single unit of work. Either everything succeeds or fails.
If you want to have multiple units of work executed then
create multiple endpoints
or send multiple messages
This also has the benefit that these can potentially be processed in parallel.
Please note, that creating multiple handlers WILL NOT have this effect. All handlers on the same endpoint will be part of the same unit of work thus transaction.
Immediate dispatch
If you really want to send a specific message when the sending of the message must not be part of the unit of work then you can immediately send it like this:
using (new TransactionScope(TransactionScopeOption.Suppress))
{
var myMessage = new MyMessage();
bus.Send(myMessage);
}
This is valid for V5, for other versions its best to look at the documentation:
http://docs.particular.net/nservicebus/messaging/send-a-message#dispatching-a-message-immediately
Enlist=false
This is a workaround that MUST NOT be used to circumvent a specific transactional configuration as is explained very well by Tomasz.
This can result in data corruption because the same messsage can be processed multiple times in case of error recovery while then the same database action will be performed again.
Found the solution.
In my connection string I had to add Enlist=False
As mentioned by #wlabaj Setting Enlist=False will indeed make sure that a transaction opened in the handler will be different from transaction used by the transport to receive/send messages.
It is however important to note that it changes the message processing semantics. By default, when DTC is used, receive/send and any transactional operations inside a handler will be commited/rolled-back atomically. With Enlist=False it's not the case so it's possible that there will be more than one handler transaction being committed for the same message. Consider following scenario as a sample case when it can happen:
message is received (transport transaction gets started)
message is successfully processed inside the handler (handler transaction committed successfully)
transport transaction fails and message is moved back to the input queue
message is received second time
message is successfully processed inside the handler
...
The behavior with Enlist-False setting is something that might a be desirable behavior in your case. That being said I think it's worth clarifying what are the consequences in terms of message processing semantics.
i am having a handler for a NserviceBus command, in its handler, we have a process which is reading an xml, but that is taking time. So now what happens is that the time when the xml is being read, the command handler fires up again and it fires number of times which I have configured as max retries, 5 in this case.
This is some code :-
class SendHandler :
IHandleMessages<SendCommand>
{
public SendHandler()
{
------some code
}
public void Handle(SendCommand message)
{ *1
-----there is some code which is taking time
-----the line number (*1) is fired again and again by the time processing takes place
}
Please help.
Check your First and Second level retry settings (FLR/SLR
As suggested in comments, try on a single command message in your queue.
If handler is not done with XML processing, check if it exceeds the default transaction timeout. If so, review the approach you're taking with handler since you might need a long(er) running process.
As far as I know, you can't specify FLR/SLR settings at a message level. They can only be specified at an endpoint level. If an endpoint is handling different messages, it is better to group them based on the message SLA. So, separate the message types with differing SLA to a different endpoint. Can you please post what your use case is in the Google groups here: https://groups.google.com/forum/#!forum/particularsoftware and we can chime in? Thanks.
I have a producer sending durable messages to a RabbitMQ exchange. If the RabbitMQ memory or disk exceeds the watermark threshold, RabbitMQ will block my producer. The documentation says that it stops reading from the socket, and also pauses heartbeats.
What I would like is a way to know in my producer code that I have been blocked. Currently, even with a heartbeat enabled, everything just pauses forever. I'd like to receive some sort of exception so that I know I've been blocked and I can warn the user and/or take some other action, but I can't find any way to do this. I am using both the Java and C# clients and would need this functionality in both. Any advice? Thanks.
Sorry to tell you but with RabbitMQ (at least with 2.8.6) this isn't possible :-(
had a similar problem, which centred around trying to establish a channel when the connection was blocked. The result was the same as what you're experiencing.
I did some investigation into the actual core of the RabbitMQ C# .Net Library and discovered the root cause of the problem is that it goes into an infinite blocking state.
You can see more details on the RabbitMQ mailing list here:
http://rabbitmq.1065348.n5.nabble.com/Net-Client-locks-trying-to-create-a-channel-on-a-blocked-connection-td21588.html
One suggestion (which we didn't implement) was to do the work inside of a thread and have some other component manage the timeout and kill the thread if it is exceeded. We just accepted the risk :-(
The Rabbitmq uses a blocking rpc call that listens for a reply indefinitely.
If you look the Java client api, what it does is:
AMQChannel.BlockingRpcContinuation k = new AMQChannel.SimpleBlockingRpcContinuation();
k.getReply(-1);
Now -1 passed in the argument blocks until a reply is received.
The good thing is you could pass in your timeout in order to make it return.
The bad thing is you will have to update the client jars.
If you are OK with doing that, you could pass in a timeout wherever a blocking call like above is made.
The code would look something like:
try {
return k.getReply(200);
} catch (TimeoutException e) {
throw new MyCustomRuntimeorTimeoutException("RabbitTimeout ex",e);
}
And in your code you could handle this exception and perform your logic in this event.
Some related classes that might require this fix would be:
com.rabbitmq.client.impl.AMQChannel
com.rabbitmq.client.impl.ChannelN
com.rabbitmq.client.impl.AMQConnection
FYI: I have tried this and it works.