Is there a way to ensure when you NACK a message and it gets sent to a dead letter exchange/queue that it will be persistent? It seems by default even if the original message is set as persistent with SetPersistent(true), the dead lettered message does not follow suite.
RabbitMQ doesn't modify the properties of Dead Letter'ed messages, except for the expiration one: https://github.com/rabbitmq/rabbitmq-server/blob/master/src/rabbit_dead_letter.erl#L74
If the message was published with deliver_mode=2, then it should be published as persistent into the Dead Letter Queue
Related
We've got our queues configured to send dead letter messages (nack'ed messages specifically) to a dead letter exchange that routes them by their original topic to individual dead letter queues. This all works great and when messages are nack'ed they're sent to the correct dead letter queue.
The trouble comes in when we shovel those messages back from the dlq to the normal queue, where they get nack'ed again. For some reason, this second time through they just disappear instead of being sent back to the dead letter exchange.
I assume there's some sort of "circular message routing" detection going on, but can't find anything like that. Inspecting the messages the second time through gives all the expected headers so I'm not sure what such a thing could even be based on. Any suggestions of where to look next or if rabbit has such a thing would be greatly appreciated!
If it's necessary, our consumers are written in python using the pika library for communication.
Assuming that you have the following queues/exchanges:
Exchanges
global_exchange - your main exchange
DLX - another exchange specifically for dead-letters
Queues
queue - your main queue within global_exchange. Contains arguments=x-dead-letter-exchange: 'DLX'
queue.dlq - your dead-letter queue within global_exchange
Bindings
test_message routing_key bound to queue and queue.dlq
Finally, I assume that you are using the shovel plugin on the queue.dlq management page like this, to move messages from queue.dlq into queue:
Here is how the routing works when you send a message with test_message as the routing_key to the global_exchange:
Message lands in queue from the binding on test_message
Consumer nack's (nack or reject doesn't matter) the message, thus dead-lettering it
The x-dead-letter-exchange argument sends it to DLX with routing_key= test_message
Because of the queue.dlq binding, that queue receives the message
When you use that particular management panel to shovel messages back into queue, it uses the default exchange. This changes the routing key. So the 2nd receipt of the message has a routing key that is equal to the name of the queue you are shoveling into.
Since you do not have an x-dead-letter-routing-key configured, the message is dead-lettered to the current routing key:
If this is not set, the message's own routing keys will be used.
So on the result of the shovel, this is how it is routed:
Message appears in queue with routing_key = queue
Since there is no x-dead-letter-routing-key configured, it dead letters to DLX with routing_key = queue
No binding to queue in DLX, message dropped
There 2 potential workarounds:
Add another binding to queue.dlq to routing_key = queue
Manually configure the x-dead-letter-routing-key on queue to always send to the same routing key on dead-letter no matter what message was originally sent to it and ensure there is a binding to it within DLX
I have an a queue that has x-expires set. The issue I am having is that I need to do further processing on the messages that are in the queue IF the queue expires. My initial idea was to set x-dead-letter-exchange on the queue. But, when the queue expires, the messages just vanish without making it to the dead-letter exchange.
How can I dead-letter, or otherwise process, messages that are in a queue that expires?
As suggested in the comments, you cannot do this by relying only on the x-expire feature. But a solution that worked in a similar case I had was to:
Use x-message-ttl to make sure messages die if not consumed in a timely manner,
Assign a dead letter exchange to the queue where all those messages will be routed,
Use x-expires to set the queue expiration to a value higher than the TTL of the messages,
(and this is the tricky part) Assuming you have control over your consumers, before the last consumer goes offline, delete the binding to your "dying" queue, potentially through a REST API call - this will prevent new messages from being routed to the queue.
This way the messages that were published before the last consumer died were already processed, existing messages will be dead-lettered before the queue expires, and new messages cannot come into the queue.
You need to add a new dead letter queue that is bound to your dead letter exchange with the binding routing key set as the original queue name. In this way all expired messages sent to the dead letter exchange are routed to the dead letter queue.
If I use publisher confirms, I can be (reasonably) sure that a message sent to an exchange on the RabbitMQ server, and which received ACK from the RabbitMQ server is not lost even if the RabbitMQ server crashes (power outage for example).
However what happens when a message arrives at a dead letter exchange after a manual rejection in the consumer? (channel.basicReject, I use Spring AMQP.)
Can I still be sure that in the case in which the original message is dequeued from the queue to which the consumer is listening, and the RabbitMQ server subsequently crashes, I will eventually find the message, after the RabbitMQ server is restarted, in the queues which are bound to the dead letter exchange (if normally the message would have arrived there)?
If the answer is negative, is there a way to ensure that this is the case?
As #GaryRussell suggested, I posted a similar question on rabbitmq-users Google group.
Here is the answer I got from Daniil Fedotov
"Hi,
There is no delivery guarantees in place. Dead lettering does not check if the message was enqueued or saved to disk.
Dead-lettering does not use publisher confirms or any other confirm mechanisms.
It's not that easy to implement reliable dead-lettering from one queue to another and there are plans to address this issue eventually, but it may take a while.
If you want to safely reject messages from the consumer without a risk of losing them - you can publish them from the consumer application manually to the dead-letter queue, wait for the confirmation and then reject."
I've a service A which is publishing message to Queue(Q-A).
I've a dead letter queue(DLQ) bounded to DLX with DLRK.
Queue A is bounded to an exchange(E-A) with a routing key(RA).
I've also set x-letter-exchange(DLX) and x-dead-letter-routing-key(DLRK) on Q-A with ttl-per-message on this queue to 60 seconds
The DLQ is also set with x-letter-exchange(E-A) and x-dead-letter-routing-key(DLRK) with ttl-per-message on this queue to 60 seconds.
With above configuration I'm trying to route the message to DLQ from Q-A after ttl expires and vice versa.
On the consumer side which is another service, I throw AMQPRejectAndDontRequeueException with defaultRequeueRejected set to fals.
The above configuration works fine when the consumer is up and throws the
exception.
But I'm trying to limit my queue size to 1 and then publish 3 messages to the Q-A and also shutting down the consumer. I see all the three messages placed in both Q-A and DLQ and eventually all the messages are dropped.
But if I don't set the queue limit to 1 or start the consumer, everything works fine.
I've also set the x-overflow to reject-publish and when there is overflow, I get a nack at the publisher and then I've a scheduler which publish it again to Q-A.
Note: Both exchanges are Direct and I'm using routing keys to bind it to respective queue.
Kindly, let me know if I'm missing something here and let me know need to share my config
After digging through, I think i finally found the answer from the link Dead-lettering dead-lettered messages in RabbitMQ
answer by pinepain
It is possible to form a cycle of dead-letter queues. For instance, this can happen when a queue dead-letters messages to the default exchange without specifiying a dead-letter routing key. Messages in such cycles (i.e. messages that reach the same queue twice) will be dropped if the entire cycle is due to message expiry.
So I think to solve the problem I need to create another consumer to consume from dead letter queue and publish it back to original queue from the consumer and not directly ttl from the dead letter queue. Please correct me if my understanding is right.
I may have arrived at this too late, But I think I can help you with this.
Story:
You want a retry queue to send dead messages to and retrieve and re-queue them in the main queue after a certain amount of time.
Solution:
Declare your main queue and bind it to an exchange. We call them main_queue and main_exchange and add this feature to the main_queue: x-dead-letter-exchange: retry_exchange
Create your retry queue and bind it to another exchange. We call these retry_queue and retry_exchange and add these features to the retry queue: x-dead-letter-exchange: main_exchange and x-message-ttl: 10000
With this combination, dead messages from main_queue will be sent to retry_queue and after 10 seconds they will be sent again to the main_queue which will they last indefinitely until a consumer declares them dead.
Note: This method works only if you publish your messages to the exchange and not directly in the queue.
I'm trying to implement a work queue architecture using RabbitMQ. I have a single sender application and multiple consumers.
I use manual ack on the consumers, so in case of failure in handling a request, it will be re-queued for another consumer to handle.
I was wondering what would happen if all the consumers return nack on a specific request. Is there a way to recognize this behavior and mark the request as 'dead' so it's rerouted to the dead letter exchange? In such a case, I'd like to have a separate consumer open on the queue bound to the dead letter exchange and receive all the messages that failed to be handled by any consumer (for logging purposes or executing this request's task locally, without distributed consumers).
Another question I had. When requeueing the request upon receiving NACK from a consumer, will it try to send this request to other consumers or will it try to send to the first available, even if it's the one that already nacked the request?
Thanks
no there is no such a feature in RabbitMQ. You may handle exceptions, and for specific exception send message to the dead queue or if know maximum time that message must live, configure TTL on queue.
if you nack message, it will go to the next AVAILABLE consumer