Having trouble wrapping my ahead around the differences between exclusive queue and exclusive consumer, and was wondering if I had this correct.
Let's say I have a queue, consumer1, and consumer2.
My understanding is:
exclusive queue
If queue is exclusive queue and was created by consumer1, only consumer1 can access the queue, and when consumer1 goes down, queue is deleted. It sounds like messages are then lost.
exclusive consumer
If queue is NOT exclusive and was created by consumer1, then both consumer1 and consumer2 can read from it. If consumer1 goes down, consumer2 can pick up where it left off. If consumer1 is exclusive consumer and starts a subscription first, then consumer2 cannot access queue, while it is locked by consumer1.
exclusive queue sounds like a weird use case to me whereas exclusive consumer sounds like the proper solution for "only 1 consumer should process from the queue at a time".
I think the difference between them is that for exclusive queue it can only be consumed by consumers that are on the same connection which is used for declaring the queue. When this connection is closed or disconnected, the exclusive queue is automatically closed by rabbitmq. The use case for it is that you usually want the queue to be destroyed when the exclusive (private) consumer exit or disconnect.
Whereas the use case for exclusive consumer on a normal queue is slightly different. The private consumer may be gone or disconnected but another consumer can be the successor of the next exclusive consumer, and it may not have to be on the same rabbitmq connection as the one used for declaring the queue.
Essentially both use cases are about allowing exclusive access to the queue but the latter is less strict.
Related
Similar to this question, we have FIFO queues and the messages must be processed in order. We want competing consumers from different machines for redundancy and performance reasons, but only one consumer on one machine should handle a message for a given queue at a time.
I tried setting the prefetch count to 1, but I believe this will only work if used with a single machine. Is this possible by default with RabbitMQ or do we need to implement our own lock?
Given a single queue with multiple consumers there is no way to block one of the consumers, all of them receive the messages in round-robin fashion.
EDIT
See https://www.rabbitmq.com/consumers.html#single-active-consumer
/EDIT
You could see this plugin, https://github.com/rabbitmq/rabbitmq-consistent-hash-exchange to distribute the load using different queues.
I tried setting the prefetch count to 1
prefetch=1 means that the consumers take one message at a time.
do we need to implement our own lock
Yes, If you want one single consumer for queue avoiding other consumers.
EDIT
There are also the Exclusive Queues https://www.rabbitmq.com/queues.html#exclusive-queues but note:
Exclusive queues are deleted when their declaring connection is closed or gone (e.g. due to underlying TCP connection loss). They, therefore, are only suitable for client-specific transient state.
I'm trying to setup RabbitMQ in a model where there is only one producer and one consumer, and where messages sent by the producer are delivered to the consumer only if the consumer is connected, but dropped if the consumer is not present.
Basically I want the queue to drop all the messages it receives when no consumer is connected to it.
An additional constraint is that the queue must be declared on the RabbitMQ server side, and must not be explicitly created by the consumer or the producer.
Is that possible?
I've looked at a few things, but I can't seem to make it work:
durable vs non-durable does not work, because it is only useful when the broker restarts. I need the same effect but on a connection.
setting auto_delete to true on the queue means that my client can never connect to this queue again.
x-message-ttl and max-length make it possible to lose message even when there is a consumer connected.
I've looked at topic exchanges, but as far as I can tell, these only affect the routing of messages between the exchange and the queue based on the message content, and can't take into account whether or not a queue has connected consumers.
The effect that I'm looking for would be something like auto_delete on disconnect, and auto_create on connect. Is there a mechanism in rabbitmq that lets me do that?
After a bit more research, I discovered that one of the assumptions in my question regarding x-message-ttl was wrong. I overlooked a single sentence from the RabbitMQ documentation:
Setting the TTL to 0 causes messages to be expired upon reaching a queue unless they can be delivered to a consumer immediately
https://www.rabbitmq.com/ttl.html
It turns out that the simplest solution is to set x-message-ttl to 0 on my queue.
You can not doing it directly, but there is a mechanism not dificult to implement.
You have to enable the Event Exchange Plugin. This is a exchange at which your server app can connect and will receive internal events of RabbitMQ. You would be interested in the consumer.created and consumer.deleted events.
When these events are received you can trigger an action (create or delete the queue you need). More information here: https://www.rabbitmq.com/event-exchange.html
Hope this helps.
If your consumer is allowed to dynamically bind / unbind a queue during start/stop on the broker it should be possible by that way (e.g. queue is pre setup and the consumer binds the queue during startup to an exchange it wants to receive messages from)
I have a RabbitMQ cluster (without HA) setup with the nodes in multiple instances. From the documentation, what I understood is, in cluster mode, the queues are not mirrored and it is owned by the node where it is declared.
So, now the question is, what will happen when the node which owns the queue goes down? Correct me if I'm wrong, since the queues are not mirrored, the client applications will throw up for missing queues.
Should we write our logic to figure out if the node goes down, the queues have to be re declared and in this case, what will happen to the messages?
So, now the question is, what will happen when the node which owns the queue goes down?
From the docs:
When RabbitMQ quits or crashes it will forget the queues and messages unless you tell it not to. Two things are required to make sure that messages aren't lost: we need to mark both the queue and messages as durable.
next question:
Should we write our logic to figure out if the node goes down, the queues have to be re declared and in this case, what will happen to the messages?
Yes, its a good idea to re-declare your queues.
In case when your node is going down, all consumers connected to it will be disconnected. Every time consumer connects, it should assume it's queue does not exist and so, it needs to fire declare queue request as first request when its connected.
If consumer sends declare queue request and queue does exist then:
declare queue won't affect queue's
messages in any way. If messages were persisted, they continue to be
in the queue.
Under normal circumstances (if you don't change the queue's
properties) no errors will be thrown
I have one direct exchange. There is also one queue, bound to this exchange.
I have two consumers for that queue. The consumers are manually ack'ing the messages once they've done the corresponding processing.
The messages are logically ordered/sorted, and should be processed in that order. Is it possible to enforce that all messages are received and processed sequentially accross consumer A and consumer B? In other words, prevent A and B from processing messages at the same time.
Note: the consumers are not sharing the same connection and/or channel. This means I cannot use <channel>.basicQoS(1);.
Rationale of this question: both consumers are identicall. If one goes down, the other queue starts processing messages and everything keeps working without any required intervention.
One approach to handling failover in a case where you want redundant consumers but need to process messages in a specific order is to use the exclusive consumer option when setting up the bind to the queue, and to have two consumers who keep trying to bind even when they can't get the exclusive lock.
The process is something like this:
Consumer A starts first and binds to the queue as an exclusive consumer. Consumer A begins processing messages from the queue.
Consumer B starts next and attempts to bind to the queue as an exclusive consumer, but is rejected because the queue already has an exclusive consumer.
On a recurring basis, consumer B attempts to get an exclusive bind on the queue but is rejected.
Process hosting consumer A crashes.
Consumer B attempts to bind to the queue as an exclusive consumer, and succeeds this time. Consumer B starts processing messages from the queue.
Consumer A is brought back online, and attempts an exclusive bind, but is rejected now.
Consumer B continues to process messages in FIFO order.
While this approach doesn't provide load sharing, it does provide redundancy.
Even though this is already answered. May be this can help others.
RabbitMQ has a feature known as Single Active Consumer, which matches your case.
We can have N consumers attached to a Queue but only 1 (one) of them will be actively consuming messages from the Queue. Fail-over happens only when active consumer fails.
Kindly take a look at the link https://www.rabbitmq.com/consumers.html#single-active-consumer
Thank you
Usually the point of a MQ system is to distribute workload. Of course, there are some situations where processing of message N depends on result of processing the message N-1, or even the N-1 message itself.
If A and B can't process messages at the same time, then why not just have A or just B? As I see it, you are not saving anything with having 2 consumers in a way that one can work only when the other one is not...
In your case, it would be best to have one consumer but to actually do the parallelisation (not a word really) on the processing part.
Just to add that RMQ is distributing messages evenly to all consumers (in round-robin fashion) regardless on any criteria. Of course this is when prefetch is set to 1, which by default it is. More info on that here, look for "fair dispatch".
The "RabbitMQ in Action" book on page 19 gives these descriptions of exclusive and auto-delete:
exclusive - When set to true, your queue becomes private and can only
be consumed by your app. This is useful when you need to limit a queue
to only one consumer.
auto-delete - The queue is automatically deleted when the last
consumer unsubscribes. If you need a temporary queue used only by one
consumer, combine auto-delete with exclusive. When the consumer
disconnects, the queue will be removed.
But as far as I can see when using exclusive, auto-delete is redundant. Only exclusive is needed. The RabbitMQ tutorial seems to say that is the case
...once we disconnect the consumer the queue should be deleted. There's
an exclusive flag for that:
result = channel.queue_declare(exclusive=True)
There is no mention in that tutorial about auto-delete and sudo rabbitmqctl list_bindings seems to indicate that the queue is in fact deleted after the receiver goes away.
Well, it is true that exclusive queues will auto-delete when the consumer disconnects (see the documentation pasted below). However, there are cases when you want queues to be non-exclusive, yet still auto-delete (for example, if I want to add another consumer).
exclusive
Exclusive queues may only be accessed by the current connection, and are deleted when that connection closes. Passive declaration of an exclusive queue by other connections are not allowed.
auto-delete
If set, the queue is deleted when all consumers have finished using
it. The last consumer can be cancelled either explicitly or because
its channel is closed. If there was no consumer ever on the queue, it
won't be deleted. Applications can explicitly delete auto-delete
queues using the Delete method as normal.
Personally, I prefer to use neither of these parameters, instead opting for the RabbitMQ queue expiration parameter, which is better if I have a consumer disconnect and then re-connect immediately (or a short time) later; messages are not lost in this case. But, of course it all depends upon your application and requirements.
In contrast to what theMayer described, my testing showed that there is a difference in behavior when auto-delete is toggled while exclusive is set to true.
If auto-delete is set to false, the queue is indeed tied to the connection and will disappear when the connection is terminated.
If auto-delete is set to true, the queue will be deleted after the last consumer is cancelled.
There is a difference between a connection and a consumer. You can be connected, but not consuming a given queue. If you need the queue's lifecycle to be tied to your connection rather than to whether or not you're actively consuming it, set auto-delete to false in conjunction with exclusive=true.