I'm having a little trouble figuring if Rabbit MQ can publish a message to a single queue with multiple subscribers, where the message will not get deleted until all subscribers to that queue have gotten the message.
The closest I can find is https://www.rabbitmq.com/tutorials/amqp-concepts.html, where it states:
AMQP 0-9-1 has a built-in feature called message acknowledgements (sometimes referred to as acks) that consumers use to confirm message delivery and/or processing. If an application crashes (the AMQP broker notices this when the connection is closed), if an acknowledgement for a message was expected but not received by the AMQP broker, the message is re-queued (and possibly immediately delivered to another consumer, if any exists).
Does this mean if the queue has more than one subscriber, it will wait until the message is consumed by all subscribers?
You should use multiple queues bound to the same exchange, using the same binding. Then, when a message matches the binding, it will be delivered to all queues, which presumably each have a consumer.
If you have multiple consumers on a single queue, RabbitMQ will round-robin deliveries among those consumers (which is not what you want).
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
Related
If I have and exchange with a single queue bound to it. I have two processes each with their own connection. Each connection has its own consumer which is consuming messages from that single queue.
Is it possible that when a message is placed onto the queue, both consumers could be delivered the same message?
Pre-fetch seems to apply across consumers on a single channel, or connection if the global flag is set. What happens with consumers across multiple connections
No. Messages are always round-robin delivered to multiple consumers on the same queue.
One solution is to have two queues bound to the same exchange using the same routing key if you need messages to go to both consumers.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
Let's suppose we have one producer, one queue and some consumers which are subscribed on queue.
Producer -> Queue -> Consumers
Queues contains messages about life events. These messages should receive all consumers.
When queue will be erased?
When all consumers get message?
Or when one of consumers confirm message with flag ack (true)?
And how to manage priority, who from consumers must to get message first/last (don't confuse with message priority).
As instance I have 10 consumers and I want that the fifth consumer get message first, remaining consumers later after specified time.
Be careful: when there are many consumers on one queue, only one of them will receive a given message, provided that it is consumed and acked properly. You need to bind as many queues as consumers to an exchange to have all consumers receive the message.
For your priority question, there is no built-in mecanism to have consumers receive the same message with a notion of priority: consumer priority exists (see https://www.rabbitmq.com/consumer-priority.html), but it is made to have consumer receive a given message before the others on a given queue, so the other consumers won't receive this message. It you need to orchestrate the delivery of your messages, you have to think of a more complex system (maybe a saga or a resequencer?).
Note that you can delay messages using this pattern. Again, this requires having multiple queues.
Finally, there are many scenarios when a queue is deleted. Take a look at the documentation, these are well explained.
I have a producer and a consumer. Multiple instances of the consumer are running. When producer publishes a message, my intention is to consume the message by all the instances. So, I am using the direct exchange. Producer publishes a message to the direct exchange with a topic. Consumers are listening to that topic with the exclusive queue. This process is working fine when the consumer is up and producer publishes a message. But when consumers are down and producer publishes a message, consumers are not consuming this message when up.
I googled about the issue. A suggestion was to use named queue. But if I use named queue, messages will be consumed following the round-robin algorithm. That does not meet my expectation to consume the same message by all the consumers.
Is there any other solution?
Appreciated your help.
There are two solutions to your issue.
Using named queue is one of them.
Set your exchange in fanout mode and subscribe your named queues to it. Doing so, when a publisher send a message in your exchange, it will be dispatched to all the queues listening.
You can then have one or more consumer for each queue (allowing you to scale). You'll have to define a named queue / consumer. When one consumer disconnect, his queue still receive messages and when he comes back he can consume them.
You should be able to do what you want that way.
The other way is more for your personnal knowledge since you said you want to use RabbitMQ. But in that particular case you could use Kafkha, your consummer could then, after reconnection, resume at the message index he was when he disconnected.
Please update me if it doesn't work :)
I'm in a phase of learning RabbitMQ/AMQP from the RabbitMQ documentation. Something that is not clear to me that I wanted to ask those who have hands-on experience.
I want to have multiple consumers listening to the same queue in order to balance the work load. What I need is pretty much close to the "Work Queues" example in the RabbitMQ tutorial.
I want the consumer to acknowledge message explicitly after it finishes handling it to preserve the message and delegate it to another consumer in case of crash. Handling a message may take a while.
My question is whether AMQP postpones next message processing until the previous message is ack'ed? If so how do I achieve load balancing between multiple workers and guarantee no messages get lost?
No, the other consumers don't get blocked. Other messages will get delivered even if they have unacknowledged but delivered predecessors. If a channel closes while holding unacknowledged messages, those messages get returned to the queue.
See RabbitMQ Broker Semantics
Messages can be returned to the queue using AMQP methods that feature a requeue parameter (basic.recover, basic.reject and basic.nack), or due to a channel closing while holding unacknowledged messages.
EDIT In response to your comment:
Time to dive a little deeper into the AMQP specification then perhaps:
3.1.4 Message Queues
A message queue is a named FIFO buffer that holds message on behalf of a set of consumer applications.
Applications can freely create, share, use, and destroy message queues, within the limits of their authority.
Note that in the presence of multiple readers from a queue, or client transactions, or use of priority fields,
or use of message selectors, or implementation-specific delivery optimisations the queue MAY NOT
exhibit true FIFO characteristics. The only way to guarantee FIFO is to have just one consumer connected
to a queue. The queue may be described as “weak-FIFO” in these cases. [...]
3.1.8 Acknowledgements
An acknowledgement is a formal signal from the client application to a message queue that it has
successfully processed a message.[...]
So acknowledgement confirms processing, not receipt. The broker will hold on to the message until it's gotten acknowleged, so that it can redeliver them. But it is free to deliver more messages to consumers even if the prededing messages have not yet been acknowledged. The consumers will not be blocked.
Does anyone know if the pop operation on a RabbitMQ queue is atomic?
I have several processes reading from the same queue (the queue is marked as durable, running on version 2.0.0) and I am seeing some quite odd behaviour.
If your multiple processes are consuming messages from the same queue then they should never consume the same message.
Here are the caveats, though:
If a message has been delivered by the broker to one of your consumers and it rejects the message (or terminates before getting a chance to acknowledge it) then the broker will put it back on the same queue and it would be delivered to one of your remaining active consumers.
If your consumers are pulling from distinct queues -- each with a matching binding -- then the broker will put copies of the message on each queue and each consumer will get a copy of the same message.