We've got our queues configured to send dead letter messages (nack'ed messages specifically) to a dead letter exchange that routes them by their original topic to individual dead letter queues. This all works great and when messages are nack'ed they're sent to the correct dead letter queue.
The trouble comes in when we shovel those messages back from the dlq to the normal queue, where they get nack'ed again. For some reason, this second time through they just disappear instead of being sent back to the dead letter exchange.
I assume there's some sort of "circular message routing" detection going on, but can't find anything like that. Inspecting the messages the second time through gives all the expected headers so I'm not sure what such a thing could even be based on. Any suggestions of where to look next or if rabbit has such a thing would be greatly appreciated!
If it's necessary, our consumers are written in python using the pika library for communication.
Assuming that you have the following queues/exchanges:
Exchanges
global_exchange - your main exchange
DLX - another exchange specifically for dead-letters
Queues
queue - your main queue within global_exchange. Contains arguments=x-dead-letter-exchange: 'DLX'
queue.dlq - your dead-letter queue within global_exchange
Bindings
test_message routing_key bound to queue and queue.dlq
Finally, I assume that you are using the shovel plugin on the queue.dlq management page like this, to move messages from queue.dlq into queue:
Here is how the routing works when you send a message with test_message as the routing_key to the global_exchange:
Message lands in queue from the binding on test_message
Consumer nack's (nack or reject doesn't matter) the message, thus dead-lettering it
The x-dead-letter-exchange argument sends it to DLX with routing_key= test_message
Because of the queue.dlq binding, that queue receives the message
When you use that particular management panel to shovel messages back into queue, it uses the default exchange. This changes the routing key. So the 2nd receipt of the message has a routing key that is equal to the name of the queue you are shoveling into.
Since you do not have an x-dead-letter-routing-key configured, the message is dead-lettered to the current routing key:
If this is not set, the message's own routing keys will be used.
So on the result of the shovel, this is how it is routed:
Message appears in queue with routing_key = queue
Since there is no x-dead-letter-routing-key configured, it dead letters to DLX with routing_key = queue
No binding to queue in DLX, message dropped
There 2 potential workarounds:
Add another binding to queue.dlq to routing_key = queue
Manually configure the x-dead-letter-routing-key on queue to always send to the same routing key on dead-letter no matter what message was originally sent to it and ensure there is a binding to it within DLX
Related
Using RabbitMQ 3.7.16, with spring-amqp 2.2.3.RELEASE.
Multiple clients publish messages to the DataExchange topic exchange in our RabbitMQ server, using a unique routing key. In the absence of any bindings, the exchange will route all the messaged to the data.queue.generic through the AE.
When a certain client (client ID 1 and 2 in the diagram) publishes lots of messages, in order to scale the consumption of their messages independently from other clients, we are starting consumers and assign them to only handle a their client ID. To achieve this, each client-consumer is defining a new queue, and it binds it to the topic exchange with the routing key events.<clientID>.
So scaling up is covered and works well.
Now when the messages rate for this client goes down, we would like to also scale down its consumers, up to the point of removing all of them. The intention is to then have all those messages being routed to the GenericExchange, where there's a pool of generic consumers taking care of them.
The problem is that if I delete data.queue.2 (in order to remove its binding which will lead to new messages being routed to the GenericExchange) all its pending messages will be lost.
Here's a simplified architecture view:
It would be an acceptable solution to let the messages expire with a TTL in the client queue, and then dead letter them to the generic exchange, but then I also need to stop the topic exchange from routing new messages to this "dying" queue.
So what options do I have to stop the topic exchange from routing messages to the client queue where now there's no consumer connected to it?
Or to explore another path - how to dead letter messages in a deleted/expired queue?
If the client queue is the only one with a matching binding as your explanation seems to suggest, you can just remove the binding between the exchange and the queue.
From then on, all new messages for the client will go through the alternate exchange, your "generic exchange", to be processed by your generic consumers.
As for the messages left over in the client queue, you could use a shovel to send them back to the topic exchange, for them to be routed to the generic exchange.
This based on the assumption the alternate exchange is internal. If it's not internal, you can target it directly with the shovel.
As discussed with Bogdan, another option to resolve this while ensuring no message loss is occuring is to perform multiple steps:
remove the binding between the specific queue and the exchange
have some logic to have the remaining messages be either consumed or rerouted to the generic queue
if the binding removal occurs prior to the consumer(s) disconnect, have the last consumer disconnect only once the queue is empty
if the binding removal occurs after the last consumer disconnect, then have a TTL on messages with alternate exchange as the generic exchange
depending on the options selected before, have some cleanup mecanism to remove the lingering empty queues
I have an a queue that has x-expires set. The issue I am having is that I need to do further processing on the messages that are in the queue IF the queue expires. My initial idea was to set x-dead-letter-exchange on the queue. But, when the queue expires, the messages just vanish without making it to the dead-letter exchange.
How can I dead-letter, or otherwise process, messages that are in a queue that expires?
As suggested in the comments, you cannot do this by relying only on the x-expire feature. But a solution that worked in a similar case I had was to:
Use x-message-ttl to make sure messages die if not consumed in a timely manner,
Assign a dead letter exchange to the queue where all those messages will be routed,
Use x-expires to set the queue expiration to a value higher than the TTL of the messages,
(and this is the tricky part) Assuming you have control over your consumers, before the last consumer goes offline, delete the binding to your "dying" queue, potentially through a REST API call - this will prevent new messages from being routed to the queue.
This way the messages that were published before the last consumer died were already processed, existing messages will be dead-lettered before the queue expires, and new messages cannot come into the queue.
You need to add a new dead letter queue that is bound to your dead letter exchange with the binding routing key set as the original queue name. In this way all expired messages sent to the dead letter exchange are routed to the dead letter queue.
I wonder if the following scenario is possible:
Create an exchange of type direct
Publish a message to that exchange with routing key rk1
After that:
Create a queue which accepts messages with routing key rk1
Consume message published to exchange
It seems like if there is no queue present, the message is dropped and there is no way to receive it.
So basically I want to be able to produce messages when there are no consumers present. And consume them some time later.
It seems like if there is no queue present, the message is dropped and there is no way to receive it.
Yes, this is correct, but it's only part of the story.
A message queue is the storage location for messages published to the server. A consumer is a designated connection set to receive messages put into a queue. The exchange is simply a location to push messages. It contains the routing semantics to determine which messages wind up in the queues on the server. When a message cannot be routed to a queue and/or consumer, there are various semantics that can apply, but the default is that the message is dropped.
Options for dealing with unroutable messages:
Alternate exchange - designates a different exchange where messages can be dumped if they cannot be routed to a queue on the current exchange. This can be thought of similar to how TCP/IP works when a destination host is not reachable on the current subnet, and traffic is forwarded to the gateway. Note that a queue must be bound to the alternate exchange for the message to be dumped into. A typical case might be to have it configured as fanout exchange with one queue to trap all messages sent into the alternate exchange.
Mandatory or Immediate - return a message back to the sender if it can't be delivered. The server does not store the message.
Mandatory designates that the message must be deliverable to a queue at the time it is published. If the message is not routable, the publisher will receive a basic.return.
Immediate designates that, in addition to being deliverable, must be immediately routed to a consumer on a particular queue (e.g. it's not good enough that it be dumped in a queue for pickup later - it has to be delivered to the end consumer right now.
In every case, if there is no queue, the server cannot store the message.
The entity queue is the one that is supposed to hold the messages , so without a queue the messages will be lost.
However in case you do not create any exchange with appropriate routing key you may leverage dead lettering feature in rabbitmq.
Another solution could be to declare the queue with the binding after the exchange and before publishing the message; this way the message will be routed and stored, but you may have to add some TTLs ( https://www.rabbitmq.com/ttl.html ).
I've a service A which is publishing message to Queue(Q-A).
I've a dead letter queue(DLQ) bounded to DLX with DLRK.
Queue A is bounded to an exchange(E-A) with a routing key(RA).
I've also set x-letter-exchange(DLX) and x-dead-letter-routing-key(DLRK) on Q-A with ttl-per-message on this queue to 60 seconds
The DLQ is also set with x-letter-exchange(E-A) and x-dead-letter-routing-key(DLRK) with ttl-per-message on this queue to 60 seconds.
With above configuration I'm trying to route the message to DLQ from Q-A after ttl expires and vice versa.
On the consumer side which is another service, I throw AMQPRejectAndDontRequeueException with defaultRequeueRejected set to fals.
The above configuration works fine when the consumer is up and throws the
exception.
But I'm trying to limit my queue size to 1 and then publish 3 messages to the Q-A and also shutting down the consumer. I see all the three messages placed in both Q-A and DLQ and eventually all the messages are dropped.
But if I don't set the queue limit to 1 or start the consumer, everything works fine.
I've also set the x-overflow to reject-publish and when there is overflow, I get a nack at the publisher and then I've a scheduler which publish it again to Q-A.
Note: Both exchanges are Direct and I'm using routing keys to bind it to respective queue.
Kindly, let me know if I'm missing something here and let me know need to share my config
After digging through, I think i finally found the answer from the link Dead-lettering dead-lettered messages in RabbitMQ
answer by pinepain
It is possible to form a cycle of dead-letter queues. For instance, this can happen when a queue dead-letters messages to the default exchange without specifiying a dead-letter routing key. Messages in such cycles (i.e. messages that reach the same queue twice) will be dropped if the entire cycle is due to message expiry.
So I think to solve the problem I need to create another consumer to consume from dead letter queue and publish it back to original queue from the consumer and not directly ttl from the dead letter queue. Please correct me if my understanding is right.
I may have arrived at this too late, But I think I can help you with this.
Story:
You want a retry queue to send dead messages to and retrieve and re-queue them in the main queue after a certain amount of time.
Solution:
Declare your main queue and bind it to an exchange. We call them main_queue and main_exchange and add this feature to the main_queue: x-dead-letter-exchange: retry_exchange
Create your retry queue and bind it to another exchange. We call these retry_queue and retry_exchange and add these features to the retry queue: x-dead-letter-exchange: main_exchange and x-message-ttl: 10000
With this combination, dead messages from main_queue will be sent to retry_queue and after 10 seconds they will be sent again to the main_queue which will they last indefinitely until a consumer declares them dead.
Note: This method works only if you publish your messages to the exchange and not directly in the queue.
Here's what we have here:
Topic Exchange DLE, which is intended to be a Dead-Letter Exchange
Topic Exchange E, which is the "main" Exchange
Several Queues (EQ1, ..., EQn) bound to E (and initialized with x-dead-letter-exchange = DLE), each with own Routing Key. These queues are the ones being consumed from.
For each EQn, there's a DLEQn (initialized with x-dead-letter-exchange = E and x-message-ttl = 5000), bound to DLE with the same routing key as EQn. These queues are not being consumed from
What I want is the following: if a consumer cannot process a message from EQn, it Nacks the message with requeue: false and it gets to the DLEQn - that is, to an appropriate queue on the Dead-Letter Exchange. Now, I want this message to sit on the DLEQn for some time and then get routed back to the original queue EQn to be processed again.
Try as I might, I could not get the "redelivery to the original queue" working. I see that messages sit in the DLEQn with all the right headers and Routing Key intact, but after TTL expires they just vanish into thin air.
What am I doing wrong here?
Yes, you can do this. We are currently doing this in production and it works great. The code is too long to include here but I will show you the diagram I created that represents the process. The basic idea is that the First DLX has a TTL, once that TTL expires the message goes into a 2nd queue to be re-sent back into the original.
RabbitMQ detects message flow cycling (E -> DLE -> E -> DLE ...) and silently drops messages:
From DLX manual (Routing Dead-Lettered Messages section):
It is possible to form a cycle of dead-letter queues. For instance, this can happen when a queue dead-letters messages to the default exchange without specifiying a dead-letter routing key. Messages in such cycles (i.e. messages that reach the same queue twice) will be dropped if the entire cycle is due to message expiry.
That post is pretty old, but it took me days to find a solution for a similar problem, so I thought I should share my solution here.
We're receiving messages in TargetQueue (no TTL!!!, bound to TargetExchange) which may be nacked by the consumer. TargetQueue has a DLX defined (RetryExchange), which in turn has bound a corresponding queue (RetryQueue, with a TTL of 60 secs, TargetExchange defined as DLX).
So if the consumer nacks a message from TargetQueue, it gets queued up in the RetryQueue and because of the TTL, the message gets nacked again and requeued in the original TargetQueue. The clue was, that TargetQueue may not have a TTL defined, otherwise a message like this appears in the RabbitMQ log:
Dead-letter queues cycle detected: [<<"TargetQueue">>,<<"RetryQueue">>,<<"TargetQueue">>]
So in the end the solution is pretty straight forward (and only needs one consumer). I got the final inspiration from https://medium.com/#igkuz/ruby-retry-scheduled-tasks-with-dead-letter-exchange-in-rabbitmq-9e38aa39089b