Does unbinding a queue from an exchange in RabbitMQ affects in any way connections of consumers to that queue?
I want to unbind a queue from an exchange without messages in that queue being lost or connection to the consumer being disconnected.
Binding/Unbinding queues from exchanges doesn't affect connections, channels and consumers in any way (apart that message flow may become different).
Side effect of unbinding queues is that it may become empty when all messages will be processed by consumers, and, depends on client library, consumers may raise exception due to receive timeout, but this is very edge case.
Related
RabbitMQ: version 3.11.2
I wish to configure a fanout exchange for which there will be two consumers and a single producer. Each of the two consumers can go offline for several minutes at a time; in the worst case scenario, one of the consumers could go offline for hours.
QUESTION: how should the fanout exchange and/or consumer queues be configured such that no message is ever lost, even if and when one of the consumers goes offline for several minutes or hours?
It is enough to bind queues to your exchange.
See here for more details https://www.rabbitmq.com/tutorials/tutorial-three-python.html
result = channel.queue_declare(queue='')
At this point result.method.queue contains a random queue name. For example it may look like amq.gen-JzTY20BRgKO-HjmUJj0wLg.
Secondly, once the consumer connection is closed, the queue should be deleted. There's an exclusive flag for that:
result = channel.queue_declare(queue='', exclusive=True)
channel.queue_bind(exchange='logs',
queue=result.method.queue)
Note:
If you need to handle the offline consumers, you need persistent queues and persistent messages instead of exclusive and auto-delete queues
I'm trying to setup RabbitMQ in a model where there is only one producer and one consumer, and where messages sent by the producer are delivered to the consumer only if the consumer is connected, but dropped if the consumer is not present.
Basically I want the queue to drop all the messages it receives when no consumer is connected to it.
An additional constraint is that the queue must be declared on the RabbitMQ server side, and must not be explicitly created by the consumer or the producer.
Is that possible?
I've looked at a few things, but I can't seem to make it work:
durable vs non-durable does not work, because it is only useful when the broker restarts. I need the same effect but on a connection.
setting auto_delete to true on the queue means that my client can never connect to this queue again.
x-message-ttl and max-length make it possible to lose message even when there is a consumer connected.
I've looked at topic exchanges, but as far as I can tell, these only affect the routing of messages between the exchange and the queue based on the message content, and can't take into account whether or not a queue has connected consumers.
The effect that I'm looking for would be something like auto_delete on disconnect, and auto_create on connect. Is there a mechanism in rabbitmq that lets me do that?
After a bit more research, I discovered that one of the assumptions in my question regarding x-message-ttl was wrong. I overlooked a single sentence from the RabbitMQ documentation:
Setting the TTL to 0 causes messages to be expired upon reaching a queue unless they can be delivered to a consumer immediately
https://www.rabbitmq.com/ttl.html
It turns out that the simplest solution is to set x-message-ttl to 0 on my queue.
You can not doing it directly, but there is a mechanism not dificult to implement.
You have to enable the Event Exchange Plugin. This is a exchange at which your server app can connect and will receive internal events of RabbitMQ. You would be interested in the consumer.created and consumer.deleted events.
When these events are received you can trigger an action (create or delete the queue you need). More information here: https://www.rabbitmq.com/event-exchange.html
Hope this helps.
If your consumer is allowed to dynamically bind / unbind a queue during start/stop on the broker it should be possible by that way (e.g. queue is pre setup and the consumer binds the queue during startup to an exchange it wants to receive messages from)
Let's say that a consumer for a queue has been disconnected for some time during which many number of messages are produced.
How long does RabbitMQ keep the messages for the disconnected consumer without durable mode?
(Will it discard the queue right after the consumer is disconnected? or will it keep the queue until the memory allows?)
Does the durable mode will give a functionality for a consumer to consume any message which is published until now? (i.e. random access to the queue, fetching messages out-of-order, or consuming from the beginning of the queue)
There are some TTL extensions.
TTL can be set for a given queue by setting the x-message-ttl argument
to queue.declare, or by setting the message-ttl policy.
No it doesn't. The messages are kept in queue until they are acknowledged, regardless of durability. (unless of course the server dies, then the messages are gone if not previously marked as durable).
Is there any way in RabbitMQ to have multiple consumers get the same message from the same queue?
I need to send the same message to anyone who's listening but also ensure that someone deals with it. Basically, I need the fanout functionality of an exchange combined with the basic.ack functionality of a queue. Is there any way to accomplish this in a scalable way?
If you are trying to ensure that the message is properly processed, acknowledgement already provides this capability. If your consumer is unable to process the message and does not provide an ack it will be requeued and processed again by the next available consumer. Implementing multiple competing consumers on the same queue will give you round-robin delivery, allowing the other consumers a chance for success.
How scalable this will be depends on how long it takes to process each message compared to the incoming rate, queue durability, prefetch and how many competing consumers you have on the queue.
I'm in a phase of learning RabbitMQ/AMQP from the RabbitMQ documentation. Something that is not clear to me that I wanted to ask those who have hands-on experience.
I want to have multiple consumers listening to the same queue in order to balance the work load. What I need is pretty much close to the "Work Queues" example in the RabbitMQ tutorial.
I want the consumer to acknowledge message explicitly after it finishes handling it to preserve the message and delegate it to another consumer in case of crash. Handling a message may take a while.
My question is whether AMQP postpones next message processing until the previous message is ack'ed? If so how do I achieve load balancing between multiple workers and guarantee no messages get lost?
No, the other consumers don't get blocked. Other messages will get delivered even if they have unacknowledged but delivered predecessors. If a channel closes while holding unacknowledged messages, those messages get returned to the queue.
See RabbitMQ Broker Semantics
Messages can be returned to the queue using AMQP methods that feature a requeue parameter (basic.recover, basic.reject and basic.nack), or due to a channel closing while holding unacknowledged messages.
EDIT In response to your comment:
Time to dive a little deeper into the AMQP specification then perhaps:
3.1.4 Message Queues
A message queue is a named FIFO buffer that holds message on behalf of a set of consumer applications.
Applications can freely create, share, use, and destroy message queues, within the limits of their authority.
Note that in the presence of multiple readers from a queue, or client transactions, or use of priority fields,
or use of message selectors, or implementation-specific delivery optimisations the queue MAY NOT
exhibit true FIFO characteristics. The only way to guarantee FIFO is to have just one consumer connected
to a queue. The queue may be described as “weak-FIFO” in these cases. [...]
3.1.8 Acknowledgements
An acknowledgement is a formal signal from the client application to a message queue that it has
successfully processed a message.[...]
So acknowledgement confirms processing, not receipt. The broker will hold on to the message until it's gotten acknowleged, so that it can redeliver them. But it is free to deliver more messages to consumers even if the prededing messages have not yet been acknowledged. The consumers will not be blocked.