Handling multiple push notifications from Exchange Server in WCF WF service, I am getting the following exception and the WF aborts:
"Some context on the correlation handler was not consumed properly". No documentation on this error anywhere afai can see.
Full message:
System.ServiceModel.FaultException`1[System.ServiceModel.ExceptionDetail]: Some context on the correlation handler was not consumed properly. Make sure that the handler was initialized properly by the runtime and the workflow has a Send followed by Receive or ReceiveReply activity. (Fault Detail is equal to An ExceptionDetail, likely created by IncludeExceptionDetailInFaults=true, whose value is:
System.InvalidOperationException: Some context on the correlation handler was not consumed properly. Make sure that the handler was initialized properly by the runtime and the workflow has a Send followed by Receive or ReceiveReply activity.
at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
at System.ServiceModel.Activities.Dispatcher.ControlOperationInvoker.ControlOperationAsyncResult.End(Object[]& outputs, IAsyncResult result)
at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeEnd(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage7(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)).
OK I figured this out. What was happening, to cut a long story short, was a race condition where a Receive with both query correlation and Request-Reply correlation was handling multiple messages with the same correlation ID by sitting in a do-while loop. The aim was to process the first one of many messages with the same id, discarding the rest. The first message caused a parallel process to start by raising a flag for a waiting Sequence (with the Send Reply a second later), while subsequent messages were discarded (with the Send Reply occuring immediately).
The problem was that the second message was coming in before the first SendReply had passed. This caused the correlation initialiser to overwrite the Request Reply correlation handler that the first one needed to use. It seems that this is what causes the above exception. I fixed this temporarily by making the SendReply happen immediately, though when I have time I will look at using multiple Correlation Handlers for the Receive Reply. (A dictionary of handlers based on some message id??)
Related
We are having an issue with recovery for messages originating from Sagas.
When a Saga sends a message for processing, the message handler can sometimes fail with an exception. We currently use a try/catch and when an exception is thrown, we "Reply" with a failed message to the Saga. The issue with this approach is that Recoverability retries don't happen since we are handling the error in the message handler.
My thought was to add custom logic to the pipeline and if the Command message implements some special Interface, the custom logic would send a failed message response to the Saga if an exception occurs (after the retries fails), but I'm not sure where to plug into the pipeline that would allow me to send messages after retries fails.
Is this a valid approach? If not, how can I solve for Saga to Handler failure messages after retries?
You can use immediate dispatch to not wait for a handler to complete.
However, I would like to suggest an alternate approach. Why not create a Timeout in the saga? If the reply from the processing-handler isn't received within a certain TimeSpan, you take an alternate path. The processing-handler gets 5 minutes and if it doesn't respond within 5 minutes, we do something else. If it still responds after 6 minutes, we know we've already taken the alternate path (use a boolean flag or so and store that inside the saga data) and put aside the reply that arrived too late.
If you want to start a discussion based on this, check our community platform.
I am working on RabbitMQ integration. I have a microservice which receives messages from other services. I am currently looking into how to handle messages which encounter exceptions during processing.
The scenario could be:
ServiceA sends message to engine's queue.
Engine processes the message received.
During processing, engine encountered an exception (say a NullPointerException)
Engine returns the message to ServiceA for reprocessing
ServiceA holds the message until the exception in the engine is resolved (resending to engine can be manually triggered)
I bumped into Spring AMQP documentation about Publisher Returns but I could not totally grasp the context. I would like to know how this works and if this could be a solution to address above item #4. Or is there other solution for this?
Thank you in advance!
For #4 on your list the solution is quite simple - don't acknowledge the message automatically, rather then when the processing is finished. In that way -
if the client (subscriber) dies (for whatever reason) during processing of the message then that message is re-queued (so sent to ServiceA for reprocessing in your case).
If you want to explicitly re-queue the message you could do negative acknowledgment (search for it here).
In any case of re-queuing (manual or automatic) you should be careful that the single message that causes subscribers to die doesn't end up being processed forever by subscriber(s), that is - make sure that the exception that happened during processing was a random and not a guaranteed event. Example for this would be a message containing invalid XML - you process it, see it's invalid, handle the exception and re-queue, but then again another (or the same) subscriber gets it, and handles the same exception since the content of the message and the XML inside it didn't change and so on...
I have an orchestration that calls a WCF Service, which is a 2-way request/response send port. Like so:
I've been trying to figure out a way to get the orchestration to persist in a state BEFORE the send shape. This is in hopes that when there are exceptions, or the endpoint is down, I can resume the orchestration and have it try to resend to the endpoint.
I've tried wrapping the send/receive shapes in all sorts of scope shapes in order to get it to persist above the send shape, but to no avail. I've read Seroter's blog up and down and tried all the things suggested in his post:
http://seroter.wordpress.com/2007/01/02/orchestration-handling-of-suspended-messages/
For example, if I take down the endpoint and send to it I get a
System.ServiceModel.ServiceActivationException: The requested service, 'https://my-service.svc' could not be activated.
The service instance will remain suspended until administratively resumed or terminated.
If resumed the instance will continue from its last persisted state and may re-throw the same unexpected exception.
Which is to be expected. As it says, the instance may re-throw the same exception when resumed (which it does no matter what I've tried).
Things I've tried:
Wrapping the Send shape in a atomic scope, but it still get suspended after the exception is thrown.
Wrapping both Send and Receive shapes in atomic scope, but get an error when compiling:
Error 1 an atomic scope may not contain or call a service or scope that contains both the send and the corresponding receive of a requestresponse operation on a 'uses' port or servicelink
Tried wrapping the Send and Receive shapes in a Long-Running scope and catching the exception, then using a suspend shape - but this just results in a dehydrated instance that is waiting for a response from the WCF Service.
I'm not sure of anything else I could try. Basically what I'm asking is, how can I get the orchestration to persist at the "conWrapper" (or anywhere above the sendRCMR...) shape seen in the included image?
For what you're describing, you don't really need to force an Orchestration Persistence.
If the endpoint is down, or the Adapter throws any other error, the Messaging instance will first go into retry mode, then suspend. That's likely the 'instance' the error message you included is referring to.
Once the endpoint is running again, you resume the Messaging instance.
This is the most common scenario.
Have a variable defaulted to false.
Put a long running scope around the send/receive shapes and a expression shape that set the variable to true after the send, have catch blocks for all exceptions (usually one for SOAP, one for System.Exception and a General) to send out an alert and then have a suspend shape after the alert is sent.
Around the scope put a looping shape, while false, loop.
This will allow you to resume the Orchestration until it succeeds in sending.
I have a requirement for an endpoint to receive commands from a client and also to subscribe to events from another endpoint such as:
1- the received command is tried only once then sent to the error queue if an exception occurred
2- the received event is tried indefinitely until it is processed
Could the MaxRetries bet set differently depending on the message type?
NServiceBus MaxRetries setting is to handle things like deadlocks, so not really what you want for this scenario.
What you want is to use SLRs to handle this situation.
To filter based on an exception type, have a look at http://andreasohlund.net/2012/09/26/disabling-second-level-retries-for-specific-exceptions/
Hope this helps!
I looked into NSB source and I notice that MaxRetries can't take different values for different messages in NSB 3.3. Happily you can override class that forward messages to error queue and implement your own version that checking if failed message is event and then instead forward to error queue you can send it again to current endpoint.
Hopefully this is a simple question but i need to verify that my assumption is correct: If i send 4 messages in one batch send and one of the 4 messages causes a fault and fails retries in its handler does that single message get forwarded to the error queue or does the entire batch message get placed into the error queue?
Common sense tells me that the single message would be moved to the error queue as the batch message has been unwrapped and delegated to its handlers.
The transaction boundary is the handler and therefore each message has its own set of retries. The only complexity to this is that if you are using a pipeline of message handlers you also have to consider that if any of the handlers fail for a given message, a retry will occur.