We have a system in production where one of the message processing services is hanging up.
Symptoms:
Service using 100% CPU
Restart of service doesn't resolve
A message handler is called but never finished
We've looked at the code and it appears to be endless loop bug in the message handler. Is there any way to configure NServiceBus to timeout handler processing and move the message to the error queue if the handler doesn't complete in a specified time?
Was the message being processed on the infinite loop copied wholesale from the error queue?
I have seen this behavior before.
When you copy a message from the error queue back onto the input queue manually it does something weird with the message header which results in this hapenning.
Related
I am doing a POC to work with RabbitMQ and have a questions about how to listen to queues conditionally!
We are consuming messaging from a queue and once consumed, the message will be involved in an upload process that takes longer times based on the file size. And as the file sizes are larger, sometimes the external service we invoke running out of memory if multiple messages are consumed and upload process is continuing for the previous messages.
That said, we would like to only consume the next message from the queue once the current/previous message is processed completely. I am new to JMS and wondering how to do it.
My current thought is, the code flow will manually pull the next message from the queue when it completes the process of previous message as the flow knows that it has completed the processing but if that listener is only used in code flow to manually call, how it will pull the very first message!
The JMS spec says that message consumers work sequentially:
The session used to create the message consumer serializes the
execution of all message listeners registered with the session
If you create a MessageListener and use that with your consumer, the JMS spec states the listener's onMessage will be called sequentially, i.e. once per message after each message has been processed by the listener. So in effect each message waits until the previous has completed.
I have a service which tasks worker processes via RabbitMQ. The messages are sent with a TTL, and the worker will not ack the message until it successfully completes the task sent in the message.
The tasking process will monitor workers for timeouts, and if a worker exceeds the timeout it will be terminated. Since the message isn't ack'd, the message is re-queued immediately and the next worker will pick up the message (this is useful in my scenario, as workers are unreliable and may fail but subsequent attempts typically succeed.
However, I would also like the ability to cancel a message. Terminating and re-creating the worker process is the normal procedure (it's single threaded, so I can't send a separate 'cancel' message to the worker). However, doing so leads to the message immediately re-queueing if the TTL has not been exceeded.
The only suggested solution I've found is here, which suggested a separate data source which checks if a message is still valid. However, that answer is both a) old and b) inconvenient.
Does RabbitMQ offer a means to cancel a message once it's been placed into the queue?
Unfortunately rabbitmq does not have a way to cancel a message.
Without the ability to send a "cancel" message to your consumer, you may have to do something like what that other post suggests.
Another option to consider: message processing should be idempotent. That is, processing the same message more than once should only cause the desired result to occur once (the first time it is processed).
Idempotence is often achieved through the use of a correlationid in messaging. You can attach a correlationid to your message, then check a database or other service to see if that message should still be processed. If you want to "cancel" the message, you would update the other database/service with that specific correlationid to say "this one has been processed already" or "has been canceled" or something like that.
If I queue up a bunch of tasks into ServiceStack MQ, they are picked up by registered handlers for processing as expected. However if my handler application exits during processing or crashes, the message seems to be gone for ever.
I see that it does support retries, and a dead letter queue, but are messages only retried if they explicitly throw and exception from the handler?
Is there any way to have the framework handle keeping the message around until a success result is returned from the handler?
Hopefully this is a simple question but i need to verify that my assumption is correct: If i send 4 messages in one batch send and one of the 4 messages causes a fault and fails retries in its handler does that single message get forwarded to the error queue or does the entire batch message get placed into the error queue?
Common sense tells me that the single message would be moved to the error queue as the batch message has been unwrapped and delegated to its handlers.
The transaction boundary is the handler and therefore each message has its own set of retries. The only complexity to this is that if you are using a pipeline of message handlers you also have to consider that if any of the handlers fail for a given message, a retry will occur.
I have a nServicebus application,now my application reads messages from handler and remove the messages from bus automatically. But what I want is that after retriving messages from Handler it should not remove messages from bus.And after my message processing which is done by my code, I will remove the message from bus. It means I have to control the messages.
Thank in advance
Susanta Samanta
NServiceBus runs all handlers in a TransactionScope, that means that if your code throws an exception the message will be put back in the queue and retried later. So you shouldn't worry about catching any exceptions/managing how messages get removed from the queue, NSB takes care of that for you.
If you can provide some more info on what you're trying to do I might be able to help you some more?