Is it possible to use Dead Letter Exchanges to handle both retrying a task and archiving tasks that failed more than retry_max number of times?
Most articles I've read set it up so that when a task fails, it is nacked so the task is sent to a retry-queue via a DLX. retry-queue has a TTL and is configured to send the dead message back to worker-queue.
What I would like to do is to send the message to a different queue, e.g.: dead-queue, when the maximum number of retries fails for manual intervention later. Is this possible using nack?
The diagram in the second answer here suggests that I can somehow nack the message when the max retries is exceeded and send it to a "Fail Exchange" but I'm not sure how I can nack the message and have it delivered to a different exchange.
Related
I'm trying to understand how to set the redelivery time for basic.nacked messages. Some important info is that I'm using quorum queues with a redelivery amount of 5 times. Consider the following scenario:
What is happening Now:
Producer sends message: Message X
Consumer handles Message X and runs into an error, in the error handler I use basic.nack()
Message is resent to original queue. Consumer immediately handles that task again.
This process repeats until the redelivery amount has been reached and then it's dead-lettered.
What I actually want:
I want the message that is requeued to wait a few seconds (3-5 sec or something) before it's once again handled by a consumer. I want to do this due to the fact that I'm using OCC & there are instances where delaying the message redelivery solves consistency issues. (for the people wondering why in god's name I need this).
I was able to do this with NATS streaming server, but I don't know how to implement it with rabbitMQ.
Additional info: I'm using amqplib (typescript) as the client and prefetch is set to 10 globally. I'm using AWS MQ for Rabbit as my rabbitMQ host
As far as i know, there isn't a way to add RabbitMQ Delayed Message Plugin to AWS MQ.
You can:
create a new dead_letter_queue with x-message-ttl option, with value you need (3-5 secs)
for this dead_letter_queue dead letter exchange will be your original exchange
create dead_letter_exchange connected with dead_letter_queue
Workflow:
Consumer nack message
Message goes to dead_letter_exchange
From dead_letter_exchange it goes to dead_letter_queue
In dead_letter_queue message waits x-message-ttl time
Message marks as dead and goes to your original exchange (and then to your original queue)
There are some of the messages got stuck in UnAck state in RabbitMQ. Is there any way to move them to ready state without restarting the consumer application or without restarting the RabbitMQ server?
Unacked state literally means messages are being consumed and awaiting for Acknowledgement i.e. status update. If your messages are stuck in this state, it mostly likely means your consumers have not provided appropriate acknowledgements for those message.
You can provide acknowledgements in the following ways.
ack the message. This signals to RabbitMQ that the message has been successfully processed/consumed and can be pop from the queue. See https://www.rabbitmq.com/amqp-0-9-1-quickref.html#basic.ack
reject or nack the message. This signals that the message was not processed correctly and should be either "dead-lettered" or "re-queue", depending on the message/queue configuration. See https://www.rabbitmq.com/amqp-0-9-1-quickref.html#basic.reject
Alternatively, you can also set a TTL for your messages, in which case, they will be automatically reject if their time in the queue exceed their TTL. See here https://www.rabbitmq.com/ttl.html.
I have a basic setup where a consumer is listening to a "/queue/work" for items. The consumer is meant to only consume one item at a time.
to do so i NACK any other item received concurrently and also UNSUBSCRIBE from the queue while the item is getting processed.
if i now SUBSCRIBE to the same queue again, the NACKed messages are not redelivered to the client - unless i drop the whole connection and reconnect the session - which is not exactly what i want to do since it impacts other subscriptions too :(
is there another way to implement this "take one item - ignore while busy" pattern ?
According to the STOMP Protocol Spec, the server does not re-deliver the NACKed message to the client that sent the NACK.
When you re-subscribe, try using a different id header value that is not used by any other subscriber.
To answer my own question - the way to implement it without ever having to NACK a message which produces the redelivery problem is to use a combination of transactions and prefetch settings:
if setting the "prefetch-count": 1 in the STOMP header - the server will only allow one message "in flight" on the channel before sending the next. This means that the client has to ACK/NACK the message before a new one is sent.
So instead of sending an ACK only if the "job" is completed - we instead start a transaction when we receive the message - ACK the message immediately - and COMMIT the transaction when the job is done. This way "failed" jobs are redelivered correctly.
I'm trying to implement a work queue architecture using RabbitMQ. I have a single sender application and multiple consumers.
I use manual ack on the consumers, so in case of failure in handling a request, it will be re-queued for another consumer to handle.
I was wondering what would happen if all the consumers return nack on a specific request. Is there a way to recognize this behavior and mark the request as 'dead' so it's rerouted to the dead letter exchange? In such a case, I'd like to have a separate consumer open on the queue bound to the dead letter exchange and receive all the messages that failed to be handled by any consumer (for logging purposes or executing this request's task locally, without distributed consumers).
Another question I had. When requeueing the request upon receiving NACK from a consumer, will it try to send this request to other consumers or will it try to send to the first available, even if it's the one that already nacked the request?
Thanks
no there is no such a feature in RabbitMQ. You may handle exceptions, and for specific exception send message to the dead queue or if know maximum time that message must live, configure TTL on queue.
if you nack message, it will go to the next AVAILABLE consumer
Can I retry a message N times and then send it to a dead queue without making ack and republishing the message from the consumer?
The only way I can think of is to use multiple queues with dlx setup which fills the next retry queue like this:
test ---> test.retries.1 ---> ... ---> test.retries.N ---> test.dead
Is this ok? I am not sure want I mean by ok. I've started playing with rabbitmq recently. Let's say is this a common setup? Are there any disadvantages?
Is there another way? Maybe a plugin that adds a counter to basic.reject and does the same thing?
Side note: I want to known this because I distrust the idea of having a consumer that will acknowledge a message (even though he cannot process it) and then publish it again. At the end you will end up with multiple liers that will publish a message and from time to time fetch it immediately before everyone else "just to be sure" and.. you'll make them remember.. (and they won't) [this also happens in the scenario with the multiple retry queues but at least the broker will control where the message is going not the consumer]
basic.reject with requeue + TTL
You have one queue and you requeue the message multiple times on failure and when the ttl expires you can setup a dlx.
basic.reject with multiple queues
On failure you always do basic.reject without requeue and use the dlx to send the message to the next retry queue:
test ---> test.retries.1 ---> ... ---> test.retries.N ---> test.z_dead
At the moment I am using this approach with only 1 retry queue and I have a special queue that receives certain messages from the dlx and sends me an email. (In my case a message is acknowledged in few hours)
basic.reject with counting of the number of retries
When you do basic.reject without requeue and use a dlx you can check x-death header added by the dlx to determine the number of retries.
Here is how it is done in sneakers - a ruby gem:
---> test (queue)
|
| test.retry (exchange)
|
---> test.retry (queue - wait for some time with ttl)
|
| test.retry.requeue (exchange)
|
---> test (queue)
At the end you count how many times you have passed through the test queue and when you exceed your retry count, you'll have to acknowledge the message (maybe after publishing it somewhere so you could be notified for the error).