I have just learned I have a high number of unroutable messages being published to the default exchange.
Because this is the default exchange, I am having a hard time tracking down the source of these unroutable messages. If this was not the default exchange I could find the publishing client by searching my code base by exchange name. Additionally, I can't effectively bind to the default exchange because I do not know the routing keys of the messages being published.
Is there any way I can find these messages without starting/stopping all my clients one at a time?
Related
I wonder if the following scenario is possible:
Create an exchange of type direct
Publish a message to that exchange with routing key rk1
After that:
Create a queue which accepts messages with routing key rk1
Consume message published to exchange
It seems like if there is no queue present, the message is dropped and there is no way to receive it.
So basically I want to be able to produce messages when there are no consumers present. And consume them some time later.
It seems like if there is no queue present, the message is dropped and there is no way to receive it.
Yes, this is correct, but it's only part of the story.
A message queue is the storage location for messages published to the server. A consumer is a designated connection set to receive messages put into a queue. The exchange is simply a location to push messages. It contains the routing semantics to determine which messages wind up in the queues on the server. When a message cannot be routed to a queue and/or consumer, there are various semantics that can apply, but the default is that the message is dropped.
Options for dealing with unroutable messages:
Alternate exchange - designates a different exchange where messages can be dumped if they cannot be routed to a queue on the current exchange. This can be thought of similar to how TCP/IP works when a destination host is not reachable on the current subnet, and traffic is forwarded to the gateway. Note that a queue must be bound to the alternate exchange for the message to be dumped into. A typical case might be to have it configured as fanout exchange with one queue to trap all messages sent into the alternate exchange.
Mandatory or Immediate - return a message back to the sender if it can't be delivered. The server does not store the message.
Mandatory designates that the message must be deliverable to a queue at the time it is published. If the message is not routable, the publisher will receive a basic.return.
Immediate designates that, in addition to being deliverable, must be immediately routed to a consumer on a particular queue (e.g. it's not good enough that it be dumped in a queue for pickup later - it has to be delivered to the end consumer right now.
In every case, if there is no queue, the server cannot store the message.
The entity queue is the one that is supposed to hold the messages , so without a queue the messages will be lost.
However in case you do not create any exchange with appropriate routing key you may leverage dead lettering feature in rabbitmq.
Another solution could be to declare the queue with the binding after the exchange and before publishing the message; this way the message will be routed and stored, but you may have to add some TTLs ( https://www.rabbitmq.com/ttl.html ).
In my system, I use Topic Exchanges with lots of consumer queues. Each queue has it's own non-unique routing key (f.e. 'add.#' for all new entities or just '#' to consume all events).
I want to add support for retrying failed messages with some delay. The biggest issue that I see with Dead Letter Exchange approach is to send a message directly to the queue in which it failed. Routing keys for Queues are not unique, and even if I resubmit a message to the Exchange with the original routing key, it will be consumed by other queues.
One solution is having a "retry" exchange and every application will be subscribed to it with unique routing key (f.e. original queue name). But it sounds too complicated and I want to hide this infrastructure complexity from developers.
I came up with the idea to have a filter that will check the 'x-death' header, get the first queue (the queue where the error occurred in a first place), and process a message only for the appropriate queue. Otherwise - acknowledge the message.
Is it possible to implement this behavior with Spring AMQP? I'm looking into MessagePostProcessor, but how to Acknowledge a message from it?
If you really worry about only the target queue, so you need to consider a variant with republishing in the default exchange which has these abilities:
The default exchange is implicitly bound to every queue, with a routing key equal to the queue name. It is not possible to explicitly bind to, or unbind from the default exchange. It also cannot be deleted.
Pay attention to the routing key equal to the queue name part. I would consider to deal with a AmqpHeaders.CONSUMER_QUEUE and use its value as a routing key for republishing to the default exchange ("") during retry process.
I have several exchanges that are dumping messages into a single queue for consumption by a client app. When the messages are received, I'd like to be able to see the exchange the message was originally published to so I can add some metadata to the object. Is there anything I can look at in the message properties that can tell me where it came from?
The AMQP 0-9-1 specifications include the name of the exchange the message was published to in the basic.get_ok answer of the basic.get method.
Therefore, you should be able to retrieve such information. It just depends on the client you are using and to what degree it honors the AMQP specifications.
Why do we need routing key to route messages from exchange to queue? Can't we simply use the queue name to route the message? Also, in case of publishing to multiple queues, we can use multiple queue names. Can anyone point out the scenario where we actually need routing key and queue name won't be suffice?
There are several types of exchanges. The fanout exchange ignores the routing key and sends messages to all queues. But pretty much all other exchange types use the routing key to determine which queue, if any, will receive a message.
The tutorials on the RabbitMQ website describes several usecases where different exchange types are useful and where the routing key is relevant.
For instance, tutorial 5 demonstrates how to use a topic exchange to route log messages to different queues depending on the log level of each message.
If you want to target multiple queues, you need to bind them to a fanout exchange and use that exchange in your publisher.
You can't specify multiple queue names in your publisher. In AMQP, you do not publish a message to queues, you publish a message to an exchange. It's the exchange responsability to determine the relevant queues. It's possible that a message is routed to no queue at all and just dropped.
Decoupling queue names from applications is useful for flexibility.
You could establish multiple queues to consume the same message, but queues can't have the same name.
In some cases, message's originator doesn't know the names of queues. (like when you have randomly generated queue names when horizontally scaling a server)
An exchange may be routing messages for more than just one type of consumer. Then you would need some wildcards in your routing keys to route messages to concerned consumers.
I have exactly 2 types of messages that I want to be sent via RabbitMQ. So I have 2 options how I can do this:
sent a message to default empty-named exchange with routing_key corresponding to the queue name
use direct exchange's routing_key parameter corresponding to consumer's routing_key parameter in queue binding
So which option is preferable and why?
A default exchange is a direct exchange. RabbitMQ creates the default exchange by default, but it uses an empty string for the name. If you look at the RabbitMQ AMQP concepts page, under Default Exchange:
The default exchange is a direct exchange with no name (empty string)
pre-declared by the broker.
You can see this by running rabbitmqctl list_exchanges as well:
direct
Foo direct < Same thing as the above
amq.direct direct
amq.fanout fanout
...and so on
As far as I'm aware, there isn't any benefits of using one over the other. I would stick with the default exchange if you only need to route based on routing keys.
Let's say you direct-bind to an exchange broadcasting logs to routing keys of "info", "warn", and "error". Using the default exchange, you would need to create three different queues with those names to receive all logs; and adjustments to which log levels you receive would require changing your queue declarations. By using a named exchange, you can simply change your queue's bindings and continue processing things as normal.
In short, it provides one extra level of abstraction.
As I see it, the default direct exchange give the possibility for the consumers and the producers to not know about each other, by binding a queue (used by a consumer) to an exchange (used by a producer) implicitly using the queue's name.
I use the default direct-exchange for a specific case: the consumer and producers don't know about each other. In my case, each consumer have its proper queue. From the producer, I cannot know by advance which queues are going to be declared and used, as it depends on the consumers. So it is impossible to define the bindings between a custom direct-exchange and the queues on the producer side.
One way to solve it with a custom (user-defined) direct-exchange would be to define the binding-key on consumer side. But it would means to know about the producer from the consumer side as I need to know the exchange name used by the producer.
Therefore, automatically binding a queue by its name on the default direct-exchange makes it possible, in my case, to only declare a queue on consumer side and send message to it from producer by only knowing the queue's name.
Of course it implies to know the queue's name at runtime, when invoking the producer, as it would be required to know the binding-key of a custom direct-exchange (in my case, the queue's name is given by the application using the producer). But when configuring the broker, producers and consumers don't have to know about each other.
some description in rabbitmq web manager.
//default exchange 's Bindings
Default exchange
The default exchange is implicitly bound to every queue,
with a routing key equal to the queue name.
It is not possible to
explicitly bind to, or
unbind from the default exchange.
It also cannot be deleted.