Here's what we have here:
Topic Exchange DLE, which is intended to be a Dead-Letter Exchange
Topic Exchange E, which is the "main" Exchange
Several Queues (EQ1, ..., EQn) bound to E (and initialized with x-dead-letter-exchange = DLE), each with own Routing Key. These queues are the ones being consumed from.
For each EQn, there's a DLEQn (initialized with x-dead-letter-exchange = E and x-message-ttl = 5000), bound to DLE with the same routing key as EQn. These queues are not being consumed from
What I want is the following: if a consumer cannot process a message from EQn, it Nacks the message with requeue: false and it gets to the DLEQn - that is, to an appropriate queue on the Dead-Letter Exchange. Now, I want this message to sit on the DLEQn for some time and then get routed back to the original queue EQn to be processed again.
Try as I might, I could not get the "redelivery to the original queue" working. I see that messages sit in the DLEQn with all the right headers and Routing Key intact, but after TTL expires they just vanish into thin air.
What am I doing wrong here?
Yes, you can do this. We are currently doing this in production and it works great. The code is too long to include here but I will show you the diagram I created that represents the process. The basic idea is that the First DLX has a TTL, once that TTL expires the message goes into a 2nd queue to be re-sent back into the original.
RabbitMQ detects message flow cycling (E -> DLE -> E -> DLE ...) and silently drops messages:
From DLX manual (Routing Dead-Lettered Messages section):
It is possible to form a cycle of dead-letter queues. For instance, this can happen when a queue dead-letters messages to the default exchange without specifiying a dead-letter routing key. Messages in such cycles (i.e. messages that reach the same queue twice) will be dropped if the entire cycle is due to message expiry.
That post is pretty old, but it took me days to find a solution for a similar problem, so I thought I should share my solution here.
We're receiving messages in TargetQueue (no TTL!!!, bound to TargetExchange) which may be nacked by the consumer. TargetQueue has a DLX defined (RetryExchange), which in turn has bound a corresponding queue (RetryQueue, with a TTL of 60 secs, TargetExchange defined as DLX).
So if the consumer nacks a message from TargetQueue, it gets queued up in the RetryQueue and because of the TTL, the message gets nacked again and requeued in the original TargetQueue. The clue was, that TargetQueue may not have a TTL defined, otherwise a message like this appears in the RabbitMQ log:
Dead-letter queues cycle detected: [<<"TargetQueue">>,<<"RetryQueue">>,<<"TargetQueue">>]
So in the end the solution is pretty straight forward (and only needs one consumer). I got the final inspiration from https://medium.com/#igkuz/ruby-retry-scheduled-tasks-with-dead-letter-exchange-in-rabbitmq-9e38aa39089b
Related
We've got our queues configured to send dead letter messages (nack'ed messages specifically) to a dead letter exchange that routes them by their original topic to individual dead letter queues. This all works great and when messages are nack'ed they're sent to the correct dead letter queue.
The trouble comes in when we shovel those messages back from the dlq to the normal queue, where they get nack'ed again. For some reason, this second time through they just disappear instead of being sent back to the dead letter exchange.
I assume there's some sort of "circular message routing" detection going on, but can't find anything like that. Inspecting the messages the second time through gives all the expected headers so I'm not sure what such a thing could even be based on. Any suggestions of where to look next or if rabbit has such a thing would be greatly appreciated!
If it's necessary, our consumers are written in python using the pika library for communication.
Assuming that you have the following queues/exchanges:
Exchanges
global_exchange - your main exchange
DLX - another exchange specifically for dead-letters
Queues
queue - your main queue within global_exchange. Contains arguments=x-dead-letter-exchange: 'DLX'
queue.dlq - your dead-letter queue within global_exchange
Bindings
test_message routing_key bound to queue and queue.dlq
Finally, I assume that you are using the shovel plugin on the queue.dlq management page like this, to move messages from queue.dlq into queue:
Here is how the routing works when you send a message with test_message as the routing_key to the global_exchange:
Message lands in queue from the binding on test_message
Consumer nack's (nack or reject doesn't matter) the message, thus dead-lettering it
The x-dead-letter-exchange argument sends it to DLX with routing_key= test_message
Because of the queue.dlq binding, that queue receives the message
When you use that particular management panel to shovel messages back into queue, it uses the default exchange. This changes the routing key. So the 2nd receipt of the message has a routing key that is equal to the name of the queue you are shoveling into.
Since you do not have an x-dead-letter-routing-key configured, the message is dead-lettered to the current routing key:
If this is not set, the message's own routing keys will be used.
So on the result of the shovel, this is how it is routed:
Message appears in queue with routing_key = queue
Since there is no x-dead-letter-routing-key configured, it dead letters to DLX with routing_key = queue
No binding to queue in DLX, message dropped
There 2 potential workarounds:
Add another binding to queue.dlq to routing_key = queue
Manually configure the x-dead-letter-routing-key on queue to always send to the same routing key on dead-letter no matter what message was originally sent to it and ensure there is a binding to it within DLX
I have an a queue that has x-expires set. The issue I am having is that I need to do further processing on the messages that are in the queue IF the queue expires. My initial idea was to set x-dead-letter-exchange on the queue. But, when the queue expires, the messages just vanish without making it to the dead-letter exchange.
How can I dead-letter, or otherwise process, messages that are in a queue that expires?
As suggested in the comments, you cannot do this by relying only on the x-expire feature. But a solution that worked in a similar case I had was to:
Use x-message-ttl to make sure messages die if not consumed in a timely manner,
Assign a dead letter exchange to the queue where all those messages will be routed,
Use x-expires to set the queue expiration to a value higher than the TTL of the messages,
(and this is the tricky part) Assuming you have control over your consumers, before the last consumer goes offline, delete the binding to your "dying" queue, potentially through a REST API call - this will prevent new messages from being routed to the queue.
This way the messages that were published before the last consumer died were already processed, existing messages will be dead-lettered before the queue expires, and new messages cannot come into the queue.
You need to add a new dead letter queue that is bound to your dead letter exchange with the binding routing key set as the original queue name. In this way all expired messages sent to the dead letter exchange are routed to the dead letter queue.
I've a service A which is publishing message to Queue(Q-A).
I've a dead letter queue(DLQ) bounded to DLX with DLRK.
Queue A is bounded to an exchange(E-A) with a routing key(RA).
I've also set x-letter-exchange(DLX) and x-dead-letter-routing-key(DLRK) on Q-A with ttl-per-message on this queue to 60 seconds
The DLQ is also set with x-letter-exchange(E-A) and x-dead-letter-routing-key(DLRK) with ttl-per-message on this queue to 60 seconds.
With above configuration I'm trying to route the message to DLQ from Q-A after ttl expires and vice versa.
On the consumer side which is another service, I throw AMQPRejectAndDontRequeueException with defaultRequeueRejected set to fals.
The above configuration works fine when the consumer is up and throws the
exception.
But I'm trying to limit my queue size to 1 and then publish 3 messages to the Q-A and also shutting down the consumer. I see all the three messages placed in both Q-A and DLQ and eventually all the messages are dropped.
But if I don't set the queue limit to 1 or start the consumer, everything works fine.
I've also set the x-overflow to reject-publish and when there is overflow, I get a nack at the publisher and then I've a scheduler which publish it again to Q-A.
Note: Both exchanges are Direct and I'm using routing keys to bind it to respective queue.
Kindly, let me know if I'm missing something here and let me know need to share my config
After digging through, I think i finally found the answer from the link Dead-lettering dead-lettered messages in RabbitMQ
answer by pinepain
It is possible to form a cycle of dead-letter queues. For instance, this can happen when a queue dead-letters messages to the default exchange without specifiying a dead-letter routing key. Messages in such cycles (i.e. messages that reach the same queue twice) will be dropped if the entire cycle is due to message expiry.
So I think to solve the problem I need to create another consumer to consume from dead letter queue and publish it back to original queue from the consumer and not directly ttl from the dead letter queue. Please correct me if my understanding is right.
I may have arrived at this too late, But I think I can help you with this.
Story:
You want a retry queue to send dead messages to and retrieve and re-queue them in the main queue after a certain amount of time.
Solution:
Declare your main queue and bind it to an exchange. We call them main_queue and main_exchange and add this feature to the main_queue: x-dead-letter-exchange: retry_exchange
Create your retry queue and bind it to another exchange. We call these retry_queue and retry_exchange and add these features to the retry queue: x-dead-letter-exchange: main_exchange and x-message-ttl: 10000
With this combination, dead messages from main_queue will be sent to retry_queue and after 10 seconds they will be sent again to the main_queue which will they last indefinitely until a consumer declares them dead.
Note: This method works only if you publish your messages to the exchange and not directly in the queue.
I have planned to delay the processing of messages in queue by following these two links link1 link2. So, as suggested in the link. I have declared the original queue with the x-dead-letter-exchange and x-dead-letter-routing-key args. Which published the messages to the so called dead-letter-queue when message either failed to get processed by consumer or ttl happen or queue length exceed. Now in the dead-letter-queue similar args have been set along with the ttl parameter. Which is suppose to republish the messages to the original queue after ttl exceed. But the problem is it is dropping all the messages.
Moreover, there is a catch here. If i explicitly publish the failed messages from original queue to dead-letter-queue. Then after ttl it republish the messages to the original queue. Why is it so and how do i make it work. So that dead-letter-queue republishes the messages to the original queue instead of dropping. I am using RabbitMQ 3.0.0.
FYI, I have created both the exchanges of direct type along with the routing key
When a queue has a TTL setup that means that messages in that queue will be sent to the dead-letter-exchange (DLX) associated with that queue after the TTL has expired. If the queue has no DLX assigned then the messages go into the bit bucket.
If you want to send messages back into the queue from which they came to be re-processed then you need to have the setup that I described in this post.
Dead-lettering dead-lettered messages in RabbitMQ
Hopefully that is helpful for you.
Suppose your original exchange is x.notification and is bind to the queue q.A with routing queue A. And your dead-letter-exchange namae is dlx.notification. Now in the queue q.A set ttl the time interval you want to wait and dead-lleter-exchange as dlx.notification. Now create another queue dlq.A to route the expired message from dlx.notification into dlq.A with routing key "A". I think thats all you need to do to achive your goal.
We are building a solution in which we are publishing message to a time-out queue. After TTL expiry messages are pushed to main queue for re-processing.
We are setting up counter value so that messages will be tried for x no. of times for the redelivery.
Solution is working fine. But the scenario is when the message on the head position is highest TTL is not expired, other messages of lower expiry will not be re-published (to main queue).
Is this understanding correct ? If Yes what is the solution so that each message re-processed just after TTL.
Appreciating answers / viewpoint.
Thanks.
If you use per-queue message TTL, then message expires and get removed from queue from head to tail (in the same order they was published).
When you use per-message TTL, then messages removed from queue only when they reach queue head, so situation when expired messages still reside in the middle of queue is normal. Such messages will not be send to consumer, and will be deadlettered (or dropped), but due to strict FIFO nature or RabbitMQ's queues that will happen as written above, when they reach queue head and delay before removal may be greater than actual message TTL. For example, if there are two message, first with TTL=10sec and the second one with TTL=1sec, second message will be deadlettered also in 10sec while it stay after first one.
To deal with messages that has different TTL, common workaround is to declare few queues, each for messages with same TTL or almost same, say, with precision 10sec. Actual precision may vary while it very application-specific and somehow empirical value.
If you will pick separate per-TTL queues, use per-queue TTL rather than per-message TTL for ease of messages workflow and to prevent disambiguation of understanding what happens with messages. Developers after you will thank you for that.
To re-process messages after their TTL use Dead Letter Exchanges, but beware of cycled messages problem: if RabbitMQ broker detects that your messages workflow cycled (messages get published to same exchange with the same routing key after it was deadlettered from it), it will silently drop message.
the queue ttl is simple enough and working fine.
but set per message ttl is not working expectly: each message publish to online consumer just after ttl.
why rabbitmq provide this feature? for which biz scenario?