Sometimes I got MassTransit.ConsumerCanceledException in my consumers.
MT-Fault-ExceptionType: MassTransit.ConsumerCanceledException
MT-Fault-Message: The operation was canceled by the consumer
Why and when does Masstransit occur this exception? how can I resolve that?
for testing, I wrote the below code. but I didn't get this exception. and the message was consumed without a problem.
public async Task Consume(ConsumeContext context)
{
await Task.Delay(TimeSpan.FromMinutes(10));
}
The ConsumerCanceledException is thrown when a consumer throws an OperationCanceledException (or TaskCanceledException since it's a subclass). This represents a consumer-induced cancellation, vs. being canceled by MassTransit itself.
It's likely a method call in your consumer or dependency is throwing either of the above exceptions.
Related
Is there a way to specify timeout value when publishing messages using the MassTransit Library. How to handle scenarios when message broker goes down. Now it seems that Publish call waits indefinitely. It would be nice to control these behavior. Should we rely on cancellation token? Timeout implementation might be better.
You can pass a CancellationToken to Publish. If canceled, an OperationCanceledException will be thrown. If you want to use a timeout, you can create a CancellationTokenSource with a timeout.
using var source = new CancellationTokenSource(TimeSpan.FromSeconds(30));
await bus.Publish(message, source.Token);
I have written a program which requires multiple queues interaction - means consumer of one queue writes message to another queue and same program has consumer to take action on that queue.
Problem: How to handle network time-out issues with queue while sending messages asynchronously using spring rabbit ampq library?or RabbitTemplate.send() function must throw an exception if there are network issues.
Currently, I have implemented RabbitTemplate.send() that returns immediately and working fine. But, If network is down, send function returns immediately, doesn't throw any exception and client code assumes success. As a result, i have in-consistent state in DB that message is successfully processed. Please note that call to send function is wrapped inside transactional block and goal is if queue writing fails, DB commit must also rollback. I am exploring following solutions but no success:
Can we configure rabbitTemplate to throw run-time exception if any network connectivity issue so that client call is notified? Please suggest how to do this.
Shall we use synchronous SendAndReceive function call but it leads to delay in processing? Another problem, observed with this function, my consumer code gets notification while sendAndReceive function is still blocked for writing message to queue. Please advise if we can delay notification to queue unless sendAndReceive function is returned. But call to SendAndReceive() was throwing an amqp exception if network was down which we were able to capture, but it has cost associated related to performance.
My application is multi-threaded, if multiple threads are sending message using sendAndReceive(), how spring-amqp library manages queue communication? Does it internally creates channel per request? If messages are delivered via same channel, it would impact performance a lot for multi-threaded application.
Can some-one share sample code for using SendAndReceive function with best-practices?
Do we have any function in spring-amqp library to check health of RabbitMQ server before submitting send function call? I explored rabbitTemplate.isRunning() but not getting proper result. If any specific configuration required, please suggest.
Any other solution to consider for guaranteed message delivery or handle network time-out issues to throw runtime exceptions to client..
As per Gary comment below, I have set: rabbitTemplate.setChannelTransacted(true); and it makes call sync. Next part of problem is that if I have transaction block on outer block, call to RabbitTemplate.send() returns immediately. I expect transaction block of outer function must wait for inner function to return, otherwise, ii don't get expected result as my DB changes are persisted though we enabled setChannelTransacted to true. I tried various Transaction propagation level but no success. Please advise if I am doing anything wrong and review transactional propagation settings as below
#Transactional
public void notifyQueueAndDB(DBRequest dbRequest) {
logger.info("Updating Request in DB");
dbService.updateRequest(dbRequest));
//Below is call to RabbitMQ library
mqService.sendmessage(dbRequest); //If sendMessage fails because of network outage, I want DB commit also to be rolled-back.
}
MQService defined in another library of project, snippet below.
#Transactional( propagation = Propagation.NESTED)
private void sendMessage(......) {
....
rabbitTemplate.send(this.queueExchange, queueName, amqpMessage);
}catch (Exception exception) {
throw exception
}
Enable transactions so that the send is synchronous.
or
Use Publisher confirms and wait for the confirmation to be received.
Either one will be quite a bit slower.
I am implementing fail over solution for messages published with Masstransit when the actual broker (RabbitMQ) is down.
The idea is to grab failed messages store them somewhere and then republish when the broker is up and running.
Possible solution is to use PublishObserver with implementation of PublishFault method.
Version of Masstransit is 5.5.5
public Task PublishFault<T>(PublishContext<T> context, Exception exception) where T : class
{
var message = context.Message; // message is null
..... // logic to save fault message in persistent storage
}
Expected result is to have an access to context.Message
Actual result - the context.Message is null;
This has been fixed in the develop version of MassTransit, which should be released at some point (as 5.5.6).
https://github.com/MassTransit/MassTransit/pull/1546
My requirement is to make the Subscriber pause processing the messages depending on whether a web service is up or not. So, when the web service is down, the messages should keep coming to the subscriber queue from Publisher and keep piling up until the web service is up again. (These messages should not go to the error queue, but stay on the Subscriber queue.)
I tried to use unsubscribe, but the publisher stops sending messages as the unsubscribe seems to clear the subscription info on RavenDB. I have also tried setting the MaxConcurrencyLevel on the Transport class, if I set the worker threads to 0, the messages coming to Subscriber go directly to the error queue. Finally, I tried Defer, which seems to put the current message in audit queue and creates a clone of the message and sends it locally to the subscriber queue when the timeout is completed. Also, since I have to keep checking the status of service and keep defering, I cannot control the order of messages as I cannot predict when the web service will be up.
What is the best way to achieve the behavior I have explained? I am using NServiceBus version 4.5.
It sounds like you want to keep trying to handle a message until it succeeds, and not shuffle it back in the queue (keep it at the top and keep trying it)?
I think your only pure-NSB option is to tinker with the MaxRetries setting, which controls First Level Retries: http://docs.particular.net/nservicebus/msmqtransportconfig. Setting MaxRetries to a very high number may do what you are looking for, but I can't imagine doing so would be a good practice.
Second Level Retries will defer the message for a configurable amount of time, but IIRC will allow other messages to be handled from the main queue.
I think your best option is to put retry logic into your own code. So the handler can try to access the service x number of times in a loop (maybe on a delay) before it throws an exception and NSB's retry features kick in.
Edit:
Your requirement seems to be something like:
"When an MyEvent comes in, I need to make a webservice call. If the webservice is down, I need to keep trying X number of times at Y intervals, at which point I will consider it a failure and handle a failure condition. Until I either succeed or fail, I will block other messages from being handled."
You have some potentially complex logic on handling a message (retry, timeout, error condition, blocking additional messages, etc.). Keep in mind the role that NSB is intended to play in your system: communication between services via messaging. While NSB does have some advanced features that allow message orchestration (e.g. sagas), it's not really intended to be used to replace Domain or Application logic.
Bottom line, you may need to write custom code to handle your specific scenario. A naive solution would be a loop with a delay in your handler, but you may need to create a more robust in-memory collection/queue that holds messages while the service is down and processes them serially when it comes back up.
The easiest way to achieve somewhat the required behavior is the following:
Define a message handler which checks whether the service is available and if not calls HandleCurrentMessageLater and a message handler which does the actual message processing. Then you specify the message handler order so that the handler which checks the service availability gets executed first.
public interface ISomeCommand {}
public class ServiceAvailabilityChecker : IHandleMessages<ISomeCommand>{
public IBus Bus { get; set; }
public void Handle(ISomeCommand message) {
try {
// check service
}
catch(SpecificException ex) {
this.Bus.HandleCurrentMessageLater();
}
}
}
public class ActualHandler : IHandleMessages<ISomeCommand>{
public void Handle(ISomeCommand message) {
}
}
public class SomeCommandHandlerOrdering : ISpecifyMessageHandlerOrdering{
public void SpecifyOrder(Order order){
order.Specify(First<ServiceAvailabilityChecker>.Then<ActualHandler>());
}
}
With that design you gain the following:
You can check the availability before the actual business code is invoked
If the service is not available the message is put back into the queue
If the service is available and your business code gets invoked but just before the ActualHandler is invoked the service becomes unavailable you get First and Second Level retries (and again the availability check in the pipeline)
I have a Workflow Service that has a few receives that listen to MSMQ queues. I would like to implement the following behavior:
If a correlation exception occurs (ie - workflow instance is gone), throw away the message.
If an InstanceLockException occurs (ie - this workflow instance is doing something on another server), put the message in the retry queue.
I have tried putting TransactedReceiveScope around the Receive activities, but it will put the message in the retry queue on a correlation error. Plus, it causes a lot of problems under heavy load.
Without the TransactedReceiveScope, the message is thrown away if there is an InstanceLockException.
Is there a way to implement this behavior (maybe via behavior extensions)?
You can implement a IErrorHandler for WCF to catch all unhandled exceptions whether thrown by your app or by WCF. The thing you have to be careful about with the netMsmqBinding is that throwing a fault in this handler means the message has been "successfully" processed and it will taken off the queue. In your case when InstanceLockException occurs, you'll have to let it remain unhandled if you want the built-in MSMQ 4 retry handling to occur. You'll also need to allow the PoisonMessageException to remain unhandled for proper MSMQ retrying to occur.
I'm not familiar with using Workflow, but knowing how MSMQ and WCF work you could try this
When a CorrelationException occurs:
Catch the exception
Return from your service method
Since your service method doesn't throw an exception it will think the message was successfully processed and remove it from the queue.
When a InstanceLockException occurs:
Catch the exception
rethrow the exception
Since your service method throws and exception it will think the message was not successfully processed and move it to the retry queue.
I think you'll have to create a WCF custom behavior to catch those exceptions.