With the RabbitMQ Admin (v3), I tried to create a queue that will send dead letter messages to the default exchange, with a routing key "MyErrorRoutingKey" on which is binded an error queue.
So in the administration interface, I left the "dead letter exchange" blank.
I just wonder if it is normal to have the following message when trying to create the queue:
406 PRECONDITION_FAILED - invalid arg 'x-dead-letter-routing-key' for
queue 'MyQueue' in vhost '/': routing_key_but_no_dlx_defined
It seems possible to route dead letter messages to the default exchange because further in the documentation it is said:
It is possible to form a cycle of dead-letter queues. For instance,
this can happen when a queue dead-letters messages to the default
exchange without specifiying a dead-letter routing key. Messages in
such cycles (i.e. messages that reach the same queue twice) will be
dropped.
So how I am supposed to route messages to the default exchange? Unlike the "cyclic useless dead-letter" described above, I want to be able to specify the routing key so that my messages are not lost.
Old question but no-one seems to have answered so I'll give it a shot.
I was having trouble with the exact same error using the web UI but I was able to get around it by setting the "x-dead-letter-exchange" and "x-dead-letter-routing-key" manually as custom arguments instead of using the provided fields.
Sounds to me like routing_key_but_no_dlx_defined says that you should not define a dead-letter-routing-key if you are not defining a dead-letter-exchange, which sort of makes sense too.
I'm not sure exactly what you're trying to do with the default exchange, but that too has a name so maybe you can just set the dead-letter-exchange also...
Regarding your last quote from the docs it means that if you have setup a cycle using dead-letter-exchanged, i.e:
QUEUE-A > DEAD-LETTER-QUEUE > QUEUE-A
...RabbitMQ will drop the message as it comes back to QUEUE-A if it leaves the DEAD-LETTER-QUEUE because of a timeout.
If such a delayed retry is what you want you'll have to manually queue your message to the DEAD-LETTER-QUEUE currently, but there's a indication that RabbitMQ may let you have such cycles later on (http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2013-April/026489.html).
Related
I wonder if the following scenario is possible:
Create an exchange of type direct
Publish a message to that exchange with routing key rk1
After that:
Create a queue which accepts messages with routing key rk1
Consume message published to exchange
It seems like if there is no queue present, the message is dropped and there is no way to receive it.
So basically I want to be able to produce messages when there are no consumers present. And consume them some time later.
It seems like if there is no queue present, the message is dropped and there is no way to receive it.
Yes, this is correct, but it's only part of the story.
A message queue is the storage location for messages published to the server. A consumer is a designated connection set to receive messages put into a queue. The exchange is simply a location to push messages. It contains the routing semantics to determine which messages wind up in the queues on the server. When a message cannot be routed to a queue and/or consumer, there are various semantics that can apply, but the default is that the message is dropped.
Options for dealing with unroutable messages:
Alternate exchange - designates a different exchange where messages can be dumped if they cannot be routed to a queue on the current exchange. This can be thought of similar to how TCP/IP works when a destination host is not reachable on the current subnet, and traffic is forwarded to the gateway. Note that a queue must be bound to the alternate exchange for the message to be dumped into. A typical case might be to have it configured as fanout exchange with one queue to trap all messages sent into the alternate exchange.
Mandatory or Immediate - return a message back to the sender if it can't be delivered. The server does not store the message.
Mandatory designates that the message must be deliverable to a queue at the time it is published. If the message is not routable, the publisher will receive a basic.return.
Immediate designates that, in addition to being deliverable, must be immediately routed to a consumer on a particular queue (e.g. it's not good enough that it be dumped in a queue for pickup later - it has to be delivered to the end consumer right now.
In every case, if there is no queue, the server cannot store the message.
The entity queue is the one that is supposed to hold the messages , so without a queue the messages will be lost.
However in case you do not create any exchange with appropriate routing key you may leverage dead lettering feature in rabbitmq.
Another solution could be to declare the queue with the binding after the exchange and before publishing the message; this way the message will be routed and stored, but you may have to add some TTLs ( https://www.rabbitmq.com/ttl.html ).
In my system, I use Topic Exchanges with lots of consumer queues. Each queue has it's own non-unique routing key (f.e. 'add.#' for all new entities or just '#' to consume all events).
I want to add support for retrying failed messages with some delay. The biggest issue that I see with Dead Letter Exchange approach is to send a message directly to the queue in which it failed. Routing keys for Queues are not unique, and even if I resubmit a message to the Exchange with the original routing key, it will be consumed by other queues.
One solution is having a "retry" exchange and every application will be subscribed to it with unique routing key (f.e. original queue name). But it sounds too complicated and I want to hide this infrastructure complexity from developers.
I came up with the idea to have a filter that will check the 'x-death' header, get the first queue (the queue where the error occurred in a first place), and process a message only for the appropriate queue. Otherwise - acknowledge the message.
Is it possible to implement this behavior with Spring AMQP? I'm looking into MessagePostProcessor, but how to Acknowledge a message from it?
If you really worry about only the target queue, so you need to consider a variant with republishing in the default exchange which has these abilities:
The default exchange is implicitly bound to every queue, with a routing key equal to the queue name. It is not possible to explicitly bind to, or unbind from the default exchange. It also cannot be deleted.
Pay attention to the routing key equal to the queue name part. I would consider to deal with a AmqpHeaders.CONSUMER_QUEUE and use its value as a routing key for republishing to the default exchange ("") during retry process.
My problem:
I pull off a message from a RabbitMQ-Queue. I try to process this message and realize that it can't be processed yet. So i would like to add it back to the queue and let it return only on a specific time + 5000ms. Unfortunately that is more challenging than i thought.
What i've tried:
RabbitMQ Dead Letter Attributes -> My issue here is, even though the manual says that the default exchange is binded to every queue it doesnt forward it according to the routing criteria. I've tried to add expires = "5000" and x-dead-letter-routing-key = "queuename" also "x-dead-letter-exchange = "" as the default exchange should work. The only part which works is the expires part. The message will disappear and go into the dark. This also occurs with the dead-letter-exchange beeing amq.direct including the binding on the targeted queue.
Open gaps for me:
Where i'm a bit left in the dark is if the receivers have to be dead letter queues and if i the dead letter queue is a basic queue with extended functionality. It is also not clear if those parameters (x-dead-letter..) are only for DLX Queues. I would like to do this delayed delivery persistent and purely via. the message attributes and not via. queue configurations (only if required).
I've searched on the web and checked many different dead-letter infos. Im trying to build a micro-service like architecture while using RabbitMQ as the delivery mechanism (i use processes which take their work from the queue and forward it). I would believe other people here have the same running already but i couldn't find any blogs about this.
I had to come to the conclusion that on the message level it is not possible.
I've created now for each queue which is in use a separate queue ("name.delayed") , where i can add the message with the argument "expiration" = 5000
The queue settings itself has to be a dead letter queue routing it to the queue "name"
Is there a way in EasyNetQ to set the routing key [x-dead-letter-routing-key] argument when creating a Queue? (as far as I can see you can only set a DeadLetterExchange.)
IQueue updateCacheQueue = advancedBus.QueueDeclare(name: "UpdateCache", deadLetterExchange: "UpdatesDeadLetter");
RabbitMQ assumes that exchanges are superior to queues. You can create an exchange that delivers to exactly one queue, and thus your DLQ addressing issue is solved. Should you decide you need to take additional actions in the future (e.g. store the message for potential reprocessing AND ALSO alert operations via email), you can do that in the exchange without mucking up the queue processor.
I Added another parameter to the QueueDeclare method and created a pull request, and you can set it after version 0.40.6.355
Is there a good best practice about what kind of messages an application is allowed to reject?
My understanding is that all messages which can't be handled should be rejected to the dead letter queue - no matter if the problem is a syntax error or a semantic error in the message or if the application is temporarily not able to handle the message (for instance because the db just went down).
Of course - if the app already knows upfront that it will not be able to handle a message (DB down), it should stop accepting messages.
So what's the common understanding / best practice?
My response is with respect to WebSphere MQ:
A Dead Letter Queue (DLQ for short) is a place where messages that could not be delivered to their destination are put. Messages can be put on the DLQ by queue managers, message channel agents (MCAs), and applications. All messages on the DLQ must be prefixed with a dead-letter header structure, MQDLH. The MQDLH header is automatically fixed when queue manager or MCAs put messages whereas applications must prefix the MQDLH explicitly.
As far applications are concerned, if they are unable to handle the message, say for example the message format is not understood, they can put the message to a BACKOUT queue instead of a DLQ. A BACKOUT queue is just like any normal queue where messages rejected by applications can be put. The advantage of BACKOUT queue is that you can specify a BACKOUT queue on a per queue basis and the messages put there need not have MQDLH header prefixed.
An application can be written to read the messages from BACKOUT and route them back to the target queue as it is. However the messages in a DLQ require additional processing to remove the MQDLH before they are put onto a target queue.