The "RabbitMQ in Action" book on page 19 gives these descriptions of exclusive and auto-delete:
exclusive - When set to true, your queue becomes private and can only
be consumed by your app. This is useful when you need to limit a queue
to only one consumer.
auto-delete - The queue is automatically deleted when the last
consumer unsubscribes. If you need a temporary queue used only by one
consumer, combine auto-delete with exclusive. When the consumer
disconnects, the queue will be removed.
But as far as I can see when using exclusive, auto-delete is redundant. Only exclusive is needed. The RabbitMQ tutorial seems to say that is the case
...once we disconnect the consumer the queue should be deleted. There's
an exclusive flag for that:
result = channel.queue_declare(exclusive=True)
There is no mention in that tutorial about auto-delete and sudo rabbitmqctl list_bindings seems to indicate that the queue is in fact deleted after the receiver goes away.
Well, it is true that exclusive queues will auto-delete when the consumer disconnects (see the documentation pasted below). However, there are cases when you want queues to be non-exclusive, yet still auto-delete (for example, if I want to add another consumer).
exclusive
Exclusive queues may only be accessed by the current connection, and are deleted when that connection closes. Passive declaration of an exclusive queue by other connections are not allowed.
auto-delete
If set, the queue is deleted when all consumers have finished using
it. The last consumer can be cancelled either explicitly or because
its channel is closed. If there was no consumer ever on the queue, it
won't be deleted. Applications can explicitly delete auto-delete
queues using the Delete method as normal.
Personally, I prefer to use neither of these parameters, instead opting for the RabbitMQ queue expiration parameter, which is better if I have a consumer disconnect and then re-connect immediately (or a short time) later; messages are not lost in this case. But, of course it all depends upon your application and requirements.
In contrast to what theMayer described, my testing showed that there is a difference in behavior when auto-delete is toggled while exclusive is set to true.
If auto-delete is set to false, the queue is indeed tied to the connection and will disappear when the connection is terminated.
If auto-delete is set to true, the queue will be deleted after the last consumer is cancelled.
There is a difference between a connection and a consumer. You can be connected, but not consuming a given queue. If you need the queue's lifecycle to be tied to your connection rather than to whether or not you're actively consuming it, set auto-delete to false in conjunction with exclusive=true.
Related
The "RabbitMQ in Action" book on page 19 gives these descriptions of exclusive and auto-delete:
auto-delete - The queue is automatically deleted when the last
consumer unsubscribes. If you need a temporary queue used only by one
consumer, combine auto-delete with exclusive. When the consumer
disconnects, the queue will be removed.
Then, in https://www.rabbitmq.com/ttl.html, gives expeire description:
expires policy controls for how long a queue can
be unused before it is automatically deleted. Unused means the queue
has no consumers, the queue has not been recently redeclared
(redeclaring renews the lease),and basic.get has not been invoked for a duration of at least the expiration period
expalation for auto-delete: the queue is deleted when all consumers have finished using it. The last consumer can be cancelled either explicitly or because its channel is closed. If there was no consumer ever on the queue, it won't be deleted. Applications can explicitly delete auto-delete queues using the Delete method.
explanation for expires: Expiry time can be set for a given queue by setting the x-expires argument to queue.declare, or by setting the expires policy. This controls for how long a queue can be unused before it is automatically deleted. Unused means the queue has no consumers, the queue has not been recently redeclared (redeclaring renews the lease), and basic.get has not been invoked for a duration of at least the expiration period. This can be used, for example, for RPC-style reply queues, where many queues can be created which may never be drained.
I think it would be better to use expires instead of auto-delete because if somehting happens and the subscriber went down for a short time then the msg will not be lost that's a big advantage of using expires and that explains simply the difference between the two.
The plain and simple answer here is that exclusive/auto-delete will delete the queue immediately after the first consumer has disconnected, while expires will delete the queue after a period of inactivity regardless of consumer(s) having connected to it in the past.
I'm not sure how to resiliently handle RabbitMQ messages in the event of an intermittent outage.
I subscribe in a windows service, read the message, then store it my database. If I can't process the record because of the data I publish it to a dead letter queue for a human to address and reprocess.
I am not sure what to do if I have some intermittent technical issue that will fix itself (database reboot, network outage, drive space, etc). I don't want hundreds of messages showing up on dead letter that just needed to wait for a for a glitch but now would be waiting on a human.
Currently, I re-queue the event and retry it once, but it retries so fast the issue is not usually resolved. I thought of retrying forever but I don't want a real issue to get stuck in an infinite loop.
Is a broad topic but from the server side you could persist your messages and make your queues durable, this means that in the eventuality the server gets restarted they won't be lost, check more here How to persist messages during RabbitMQ broker restart?
For the consumer (client) it will depend on how you configure your client, from the docs:
In the event of network failure (or a node crashing), messages can be duplicated, and consumers must be prepared to handle them. If possible, the simplest way to handle this is to ensure that your consumers handle messages in an idempotent way rather than explicitly deal with deduplication.
If a message is delivered to a consumer and then requeued (because it was not acknowledged before the consumer connection dropped, for example) then RabbitMQ will set the redelivered flag on it when it is delivered again (whether to the same consumer or a different one). This is a hint that a consumer may have seen this message before (although that's not guaranteed, the message may have made it out of the broker but not into a consumer before the connection dropped). Conversely if the redelivered flag is not set then it is guaranteed that the message has not been seen before. Therefore if a consumer finds it more expensive to deduplicate messages or process them in an idempotent manner, it can do this only for messages with the redelivered flag set.
Check more here: https://www.rabbitmq.com/reliability.html#consumer
Having trouble wrapping my ahead around the differences between exclusive queue and exclusive consumer, and was wondering if I had this correct.
Let's say I have a queue, consumer1, and consumer2.
My understanding is:
exclusive queue
If queue is exclusive queue and was created by consumer1, only consumer1 can access the queue, and when consumer1 goes down, queue is deleted. It sounds like messages are then lost.
exclusive consumer
If queue is NOT exclusive and was created by consumer1, then both consumer1 and consumer2 can read from it. If consumer1 goes down, consumer2 can pick up where it left off. If consumer1 is exclusive consumer and starts a subscription first, then consumer2 cannot access queue, while it is locked by consumer1.
exclusive queue sounds like a weird use case to me whereas exclusive consumer sounds like the proper solution for "only 1 consumer should process from the queue at a time".
I think the difference between them is that for exclusive queue it can only be consumed by consumers that are on the same connection which is used for declaring the queue. When this connection is closed or disconnected, the exclusive queue is automatically closed by rabbitmq. The use case for it is that you usually want the queue to be destroyed when the exclusive (private) consumer exit or disconnect.
Whereas the use case for exclusive consumer on a normal queue is slightly different. The private consumer may be gone or disconnected but another consumer can be the successor of the next exclusive consumer, and it may not have to be on the same rabbitmq connection as the one used for declaring the queue.
Essentially both use cases are about allowing exclusive access to the queue but the latter is less strict.
I'm trying to setup RabbitMQ in a model where there is only one producer and one consumer, and where messages sent by the producer are delivered to the consumer only if the consumer is connected, but dropped if the consumer is not present.
Basically I want the queue to drop all the messages it receives when no consumer is connected to it.
An additional constraint is that the queue must be declared on the RabbitMQ server side, and must not be explicitly created by the consumer or the producer.
Is that possible?
I've looked at a few things, but I can't seem to make it work:
durable vs non-durable does not work, because it is only useful when the broker restarts. I need the same effect but on a connection.
setting auto_delete to true on the queue means that my client can never connect to this queue again.
x-message-ttl and max-length make it possible to lose message even when there is a consumer connected.
I've looked at topic exchanges, but as far as I can tell, these only affect the routing of messages between the exchange and the queue based on the message content, and can't take into account whether or not a queue has connected consumers.
The effect that I'm looking for would be something like auto_delete on disconnect, and auto_create on connect. Is there a mechanism in rabbitmq that lets me do that?
After a bit more research, I discovered that one of the assumptions in my question regarding x-message-ttl was wrong. I overlooked a single sentence from the RabbitMQ documentation:
Setting the TTL to 0 causes messages to be expired upon reaching a queue unless they can be delivered to a consumer immediately
https://www.rabbitmq.com/ttl.html
It turns out that the simplest solution is to set x-message-ttl to 0 on my queue.
You can not doing it directly, but there is a mechanism not dificult to implement.
You have to enable the Event Exchange Plugin. This is a exchange at which your server app can connect and will receive internal events of RabbitMQ. You would be interested in the consumer.created and consumer.deleted events.
When these events are received you can trigger an action (create or delete the queue you need). More information here: https://www.rabbitmq.com/event-exchange.html
Hope this helps.
If your consumer is allowed to dynamically bind / unbind a queue during start/stop on the broker it should be possible by that way (e.g. queue is pre setup and the consumer binds the queue during startup to an exchange it wants to receive messages from)
I have one direct exchange. There is also one queue, bound to this exchange.
I have two consumers for that queue. The consumers are manually ack'ing the messages once they've done the corresponding processing.
The messages are logically ordered/sorted, and should be processed in that order. Is it possible to enforce that all messages are received and processed sequentially accross consumer A and consumer B? In other words, prevent A and B from processing messages at the same time.
Note: the consumers are not sharing the same connection and/or channel. This means I cannot use <channel>.basicQoS(1);.
Rationale of this question: both consumers are identicall. If one goes down, the other queue starts processing messages and everything keeps working without any required intervention.
One approach to handling failover in a case where you want redundant consumers but need to process messages in a specific order is to use the exclusive consumer option when setting up the bind to the queue, and to have two consumers who keep trying to bind even when they can't get the exclusive lock.
The process is something like this:
Consumer A starts first and binds to the queue as an exclusive consumer. Consumer A begins processing messages from the queue.
Consumer B starts next and attempts to bind to the queue as an exclusive consumer, but is rejected because the queue already has an exclusive consumer.
On a recurring basis, consumer B attempts to get an exclusive bind on the queue but is rejected.
Process hosting consumer A crashes.
Consumer B attempts to bind to the queue as an exclusive consumer, and succeeds this time. Consumer B starts processing messages from the queue.
Consumer A is brought back online, and attempts an exclusive bind, but is rejected now.
Consumer B continues to process messages in FIFO order.
While this approach doesn't provide load sharing, it does provide redundancy.
Even though this is already answered. May be this can help others.
RabbitMQ has a feature known as Single Active Consumer, which matches your case.
We can have N consumers attached to a Queue but only 1 (one) of them will be actively consuming messages from the Queue. Fail-over happens only when active consumer fails.
Kindly take a look at the link https://www.rabbitmq.com/consumers.html#single-active-consumer
Thank you
Usually the point of a MQ system is to distribute workload. Of course, there are some situations where processing of message N depends on result of processing the message N-1, or even the N-1 message itself.
If A and B can't process messages at the same time, then why not just have A or just B? As I see it, you are not saving anything with having 2 consumers in a way that one can work only when the other one is not...
In your case, it would be best to have one consumer but to actually do the parallelisation (not a word really) on the processing part.
Just to add that RMQ is distributing messages evenly to all consumers (in round-robin fashion) regardless on any criteria. Of course this is when prefetch is set to 1, which by default it is. More info on that here, look for "fair dispatch".