In case when RabbitMQ broker has a fanout exchange with many queues bound to it, where the queues are durable and messages delivered to the exchange are durable/persisted, will each queue store a separate copy for the message or broker may figure out and optimize queued message persistence and not store payload multiple times for each queue in the fan out, but store something like message links/references?
It depends on the size of the message. With persistence, each durable queue that receives a message will have a queue index that references that message. Larger messages (>= 4K by default) will be written to the message store and referenced by the queue index. Thus, each queue bound to your fanout exchange will have a queue index for the message, but only one copy of the message payload is sitting in the message store.
Smaller messages will be stored entirely within the queue index, with no entry in the message store. If your fanout exchange receives a small message, then each bound queue will have its own copy of the entire message.
A good explanation of RabbitMQ message persistence can be found at https://www.rabbitmq.com/persistence-conf.html.
Related
We've got our queues configured to send dead letter messages (nack'ed messages specifically) to a dead letter exchange that routes them by their original topic to individual dead letter queues. This all works great and when messages are nack'ed they're sent to the correct dead letter queue.
The trouble comes in when we shovel those messages back from the dlq to the normal queue, where they get nack'ed again. For some reason, this second time through they just disappear instead of being sent back to the dead letter exchange.
I assume there's some sort of "circular message routing" detection going on, but can't find anything like that. Inspecting the messages the second time through gives all the expected headers so I'm not sure what such a thing could even be based on. Any suggestions of where to look next or if rabbit has such a thing would be greatly appreciated!
If it's necessary, our consumers are written in python using the pika library for communication.
Assuming that you have the following queues/exchanges:
Exchanges
global_exchange - your main exchange
DLX - another exchange specifically for dead-letters
Queues
queue - your main queue within global_exchange. Contains arguments=x-dead-letter-exchange: 'DLX'
queue.dlq - your dead-letter queue within global_exchange
Bindings
test_message routing_key bound to queue and queue.dlq
Finally, I assume that you are using the shovel plugin on the queue.dlq management page like this, to move messages from queue.dlq into queue:
Here is how the routing works when you send a message with test_message as the routing_key to the global_exchange:
Message lands in queue from the binding on test_message
Consumer nack's (nack or reject doesn't matter) the message, thus dead-lettering it
The x-dead-letter-exchange argument sends it to DLX with routing_key= test_message
Because of the queue.dlq binding, that queue receives the message
When you use that particular management panel to shovel messages back into queue, it uses the default exchange. This changes the routing key. So the 2nd receipt of the message has a routing key that is equal to the name of the queue you are shoveling into.
Since you do not have an x-dead-letter-routing-key configured, the message is dead-lettered to the current routing key:
If this is not set, the message's own routing keys will be used.
So on the result of the shovel, this is how it is routed:
Message appears in queue with routing_key = queue
Since there is no x-dead-letter-routing-key configured, it dead letters to DLX with routing_key = queue
No binding to queue in DLX, message dropped
There 2 potential workarounds:
Add another binding to queue.dlq to routing_key = queue
Manually configure the x-dead-letter-routing-key on queue to always send to the same routing key on dead-letter no matter what message was originally sent to it and ensure there is a binding to it within DLX
I have an a queue that has x-expires set. The issue I am having is that I need to do further processing on the messages that are in the queue IF the queue expires. My initial idea was to set x-dead-letter-exchange on the queue. But, when the queue expires, the messages just vanish without making it to the dead-letter exchange.
How can I dead-letter, or otherwise process, messages that are in a queue that expires?
As suggested in the comments, you cannot do this by relying only on the x-expire feature. But a solution that worked in a similar case I had was to:
Use x-message-ttl to make sure messages die if not consumed in a timely manner,
Assign a dead letter exchange to the queue where all those messages will be routed,
Use x-expires to set the queue expiration to a value higher than the TTL of the messages,
(and this is the tricky part) Assuming you have control over your consumers, before the last consumer goes offline, delete the binding to your "dying" queue, potentially through a REST API call - this will prevent new messages from being routed to the queue.
This way the messages that were published before the last consumer died were already processed, existing messages will be dead-lettered before the queue expires, and new messages cannot come into the queue.
You need to add a new dead letter queue that is bound to your dead letter exchange with the binding routing key set as the original queue name. In this way all expired messages sent to the dead letter exchange are routed to the dead letter queue.
I am reading Persistence Configuration, and I have some questions.
I know that queues can be either durable or not, and messages are also divided into two categories: persistent or transient.
As the document says: Persistent messages will be written to disk as soon as they reach the queue.
So, here are my questions:
If I send a message and the message was not distributed to any queues,
will the message be persistent?
If a queue is non-durable, will the message(persistent or transient, not
being consumed) be deleted from disk when the queue is deleted?
If a message is consumed, does rabbitmq delete this messages from
physical storage?
Any ideas on how to test these conditions are appreciated.
If I send a message and the message was not distributed to any queues, will the message be persistent?
No, see mandatory to handle the unroutable messages.
If a queue is non-durable, will the message(persistent or transient, not being consumed) be deleted from disk when the queue is deleted?
yes
If a message is consumed, does rabbitmq delete this messages from physical storage?
When a message is consumed and the status is unacked the message is not removed.
when you send the ack or nack the message is removed from the memory/disk.
I've defined one topic exchange (alarms) and multiple queues, each with its own routing key:
allAlarms, with routing key alarms.#: I want this to be used for receiving all alarms in a monitoring application
alarms_[deviceID], with routing key alarms.[deviceID], where the number of devices can vary at any given time
When sending an alarm from the device, I publish it using the routing key alarms.[deviceID]. The monitoring app, however, only consumes from the allAlarms queue. This leads to the following problem:
The messages in the allAlarms queue have been consumed, while the messages in the remaining queues are ready. Is there a better way of handling messages from multiple consumers? Ideally, I'd like to be able to also send commands back to the devices using the same queues where the devices publish their alarms.
It looks like you have consumers bound to the allAlarms queue but not to any of the alarms_[deviceID] queues.
In AMQP, a single consumer is bound to a single queue by name (and each queue can have multiple consumers bound to it). Messages are delivered to the consumers of a queue in round robin such that for a given message in a queue there is exactly one consumer that will receive the message. That is, consumers cannot listen to multiple queues.
Since you're using a topic exchange, you're correctly routing a single message to multiple queues via the routing key and queue bindings. This means that you can have a consumer for each queue and when a message is delivered to the exchange, each queue will get a copy of the message and each queue will deliver the message to exactly one consumer on each queue.
Thus, if allAlarms is consuming messages, it's because it has a consumer attached to the queue. If any of the alarms_[deviceID] are not consuming messages then they must not have consumers bound to those individual queues. You have to start up consumers for each alarms_[deviceID] by name. That will allow you to also have different consumer logic for different queues.
One last thing:
Ideally, I'd like to be able to also send commands back to the devices using the same queues where the devices publish their alarms.
You don't want to do this using the same queue because there's nothing that will stop the non-device consumers on the queue from picking up those messages.
I believe you're describing RPC over RabbitMQ. For that you will want to publish the messages to the alarms queues with a reply-to header which is the name of a temporary queue. This temp queue is a single-use queue that the consumer will publish to when it's done to communicate back to the device. The device will publish to the alarms exchange and then immediately start listening to the temp queue for a response from the consumer.
For more info on RPC over RabbitMQ check out this tutorial.
I don't think you need any of the queues for the devices - the alarm_[deviceid] queues.
You don't have any consumer code set up on these queues, and the messages are backed up and waiting for you to consume them.
You also haven't mentioned a need to consume messages from these queues. Instead, you are only consuming messages form the alarmAll queue.
Therefore, I would drop all of the alarm_[deviceid] queues and only have the alarmAll queue.
Just publish the alarms through your exchange and route them all to the alarmAll queue and be done with it. No need for any other routing or queues.
I have a rabbitmq cluster used as a working queue. There are 5 kinds of consumers who want to consume exactly the same data.
What I know for now is using fanout exchange to "copy" the data to 5 DIFFERENT queues. And the 5 consumers can consume different queue. This is kind of wasting resources because the data is the same in file queues.
My question is, does rabbitmq support to push the same data to multi consumers? Just like a message need to be acked for a specified times to be deleted.
I got the following answer from rabbitmq email group. In short, the answer is no... and what I did above is the correct way.
http://rabbitmq.1065348.n5.nabble.com/Does-rabbitmq-support-to-push-the-same-data-to-multi-consumers-td36169.html#a36170
... fanout exchange to "copy" the data to 5 DIFFERENT queues. And the 5 consumers can consume different queue. This is kind of wasting resources because the data is the same in file queues.
You can consume with 5 consumers from one queue if you do not want to duplicate messages.
does rabbitmq support to push the same data to multiple consumers
In AMQP protocol terms you publish message to exchange and then broker (RabbitMQ) decide what to do with messages - assume it figured out the queue message intended for (one or more) and then put that message on top of that queue (queues in RabbitMQ are classic FIFO queues which is somehow break AMQP implementation in RabbitMQ). Only after that message may be delivered to consumer (or die due to queue length limit or per-queue or per-message ttl, if any).
message need to be acked for a specified times to be deleted
There are no way to change message body or attributes after message being published (actually, Dead Letter Exchanges extension and some other may change routing key, for example and add,remove and change some headers, but this is very specific case). So if you want to track ack's number you have to re-publish consumed message with changed body or header (depends on where do you plan to store ack's counter, but headers fits pretty nice for this.
Also note, that there are redeliverd message attribute which denotes whether message was already was consumed, but then redelivered. This flag doesn't count redelivers number so it usage is quite limited.