Let's say that a consumer for a queue has been disconnected for some time during which many number of messages are produced.
How long does RabbitMQ keep the messages for the disconnected consumer without durable mode?
(Will it discard the queue right after the consumer is disconnected? or will it keep the queue until the memory allows?)
Does the durable mode will give a functionality for a consumer to consume any message which is published until now? (i.e. random access to the queue, fetching messages out-of-order, or consuming from the beginning of the queue)
There are some TTL extensions.
TTL can be set for a given queue by setting the x-message-ttl argument
to queue.declare, or by setting the message-ttl policy.
No it doesn't. The messages are kept in queue until they are acknowledged, regardless of durability. (unless of course the server dies, then the messages are gone if not previously marked as durable).
Related
I have a RabbitMQ setup in which jobs are sent to an exchange, which passes them to a queue. A consumer carries out the jobs from the queue correctly in turn. However, these jobs are long processes (several minutes at least). For scalability, I need to be able to have multiple consumers picking a job from the top of the queue and executing it.
The consumer is running on a Heroku dyno called 'queue'. When I scale the dyno, it appears to create additional consumers for each dyno (I can see these on the RabbitMQ dashboard). However, the number of tasks in the queue is unchanged - the extra consumers appear to be doing nothing. Please see the picture below to understand my setup.
Am I missing something here?
Why are the consumers showing as 'idle'? I know from my logs that at least one consumer is actively working through a task.
How can my consumer utilisation be 0% when at least one consumer is definitely working hard.
How can I make the other three consumers actually pull some jobs from the queue?
Thanks
EDIT: I've discovered that the round robin dispatching is actually working, but only if the additional consumers are already running when the messages are sent to the queue. This seems like counterintuitive behaviour to me. If I saw a large queue and wanted to add more consumers, the added consumers would do nothing until more items are added to the queue.
To pick out the key point from the other answer, the likely culprit here is pre-fetching, as described under "Consumer Acknowledgements and Publisher Confirms".
Rather than delivering one message at a time and waiting for it to be acknowledged, the server will send batches to the consumer. If the consumer acknowledges some but then crashes, the remaining messages will be sent to a different consumer; but if the consumer is still running, the unacknowledged messages won't be sent to any new consumer.
This explains the behaviour you're seeing:
You create the queue, and deliver some messages to it, with no consumer running.
You run a single consumer, and it pre-fetches all the messages on the queue.
You run a second consumer; although the queue isn't empty, all the messages are marked as sent to the first consumer, awaiting acknowledgement; so the second consumer sits idle.
A new message arrives in the queue; it is distributed in round-robin fashion to the second consumer.
The solution is to specify the basic.qos option in the consumer. If you set this to 1, RabbitMQ won't send a message to a consumer until it has acknowledged the previous message; multiple consumers with that setting will receive messages in strictly round-robin fashion.
I am not familiar to Heroku, so I don't know how Heroku worker build rabbitMQ consumer, I just have a quick view over Heroku document.
Why are the consumers showing as 'idle'?
I think your mean the queue is 'idle'? Because the queue's state is about the queue's traffic, it just means there is not on-doing job for the queue's job thread. And it will become 'running' when a message is published in the queue.
How can my consumer utilisation be 0% when at least one consumer is definitely working hard.
The same as queue state, from official explanation, consumer utilisation too low means:
There were more consumers
The consumers were faster
The consumers had a higher prefetch count
In your situation, prefetch_count = 0 means no limits on prefetch, so it's too large. And Messages.total = Messages.unacked = 78 means your consumer is too slow, there are two many messages have been processed by consumer.
So if your message rate is not large enough, the state and consumer utilisation field of the queue is useless.
If I saw a large queue and wanted to add more consumers, the added consumers would do nothing until more items are added to the queue.
Because these unacked messages have already been prefetched by exist consumers, they will not be consumed by new consumers unless you requeue the unacked messages.
I have a queue producer(NON_PERSISTENT) and a queue consumer connected to my ActiveMQ server. If a producer sends a message to a queue while the consumer is not connected to ActiveMQ, it seems it is stored and delivered when the consumer is up and connected to ActiveMQ.
I want ActiveMQ to ignore the message if the consumer is not connected at the time of delivery. How can I achieve this?
Thanks in advance.
Use a topic instead of a queue - this is the default behaviour for topics (unless a durable subscriber is used).
Otherwise, for queues, you can set a message expiry when sending the message. It will be discarded if not read within that time frame. Make sure to set enough time frame so that clock sync issues between servers won't be a factor. Let's say 2 minutes or so.
I'm in a phase of learning RabbitMQ/AMQP from the RabbitMQ documentation. Something that is not clear to me that I wanted to ask those who have hands-on experience.
I want to have multiple consumers listening to the same queue in order to balance the work load. What I need is pretty much close to the "Work Queues" example in the RabbitMQ tutorial.
I want the consumer to acknowledge message explicitly after it finishes handling it to preserve the message and delegate it to another consumer in case of crash. Handling a message may take a while.
My question is whether AMQP postpones next message processing until the previous message is ack'ed? If so how do I achieve load balancing between multiple workers and guarantee no messages get lost?
No, the other consumers don't get blocked. Other messages will get delivered even if they have unacknowledged but delivered predecessors. If a channel closes while holding unacknowledged messages, those messages get returned to the queue.
See RabbitMQ Broker Semantics
Messages can be returned to the queue using AMQP methods that feature a requeue parameter (basic.recover, basic.reject and basic.nack), or due to a channel closing while holding unacknowledged messages.
EDIT In response to your comment:
Time to dive a little deeper into the AMQP specification then perhaps:
3.1.4 Message Queues
A message queue is a named FIFO buffer that holds message on behalf of a set of consumer applications.
Applications can freely create, share, use, and destroy message queues, within the limits of their authority.
Note that in the presence of multiple readers from a queue, or client transactions, or use of priority fields,
or use of message selectors, or implementation-specific delivery optimisations the queue MAY NOT
exhibit true FIFO characteristics. The only way to guarantee FIFO is to have just one consumer connected
to a queue. The queue may be described as “weak-FIFO” in these cases. [...]
3.1.8 Acknowledgements
An acknowledgement is a formal signal from the client application to a message queue that it has
successfully processed a message.[...]
So acknowledgement confirms processing, not receipt. The broker will hold on to the message until it's gotten acknowleged, so that it can redeliver them. But it is free to deliver more messages to consumers even if the prededing messages have not yet been acknowledged. The consumers will not be blocked.
Does anyone know if the pop operation on a RabbitMQ queue is atomic?
I have several processes reading from the same queue (the queue is marked as durable, running on version 2.0.0) and I am seeing some quite odd behaviour.
If your multiple processes are consuming messages from the same queue then they should never consume the same message.
Here are the caveats, though:
If a message has been delivered by the broker to one of your consumers and it rejects the message (or terminates before getting a chance to acknowledge it) then the broker will put it back on the same queue and it would be delivered to one of your remaining active consumers.
If your consumers are pulling from distinct queues -- each with a matching binding -- then the broker will put copies of the message on each queue and each consumer will get a copy of the same message.
ActiveMQ: 5.10.2 inside ServiceMix's Karaf OSGi
KahaDB persistence.
Default broker settings.
Default settings in connections(tcp://x.x.x.x:61616)
16 queues predefined in activemq.xml.
Two client connections to ActiveMQ. One for producer sessions, one for consumer sessions.
Producers send messages to all queues.
16 consumer sessions consumes messages.
All going ok, but:
If I reduce number of consumers to 1 (or 2 or three, I don't know where is threshold) so that messages from 1 queue are consuming and messages from another queues are storing.
While some time passing, I see this picture:
That 1 consumer stop receiving message. He think that there are no more messages.
From activemqweb I can see that message count on that consuming queue is > 0
From activemqweb I cannot see any messages in Message Browser in that consuming queue.
I can see messages from other queues in Message Browser.
If I start some other consumer(or restart activemq) to consume messages from different queue I see:
I start to see messages in first queue Message Browser(those that were sent before but haven't been seen after "freeze").
First queue continue to consuming
Second queue begin to consuming.
The "freeze" can occur again in some time and start consuming another queue will help again.
If I start all consumers I see no "message freeze".
If just stop and start consumer on "frozen" queue, nothing happens. It need to be done on "unfrozen queue" to "unfroze" "frozen queue".
It also happens if there is no active producer, only consumer.
What can it be?
Thank you.
Oups. I've found what it was.
It's just available memory exceeded.
I didn't set -Xms and -Xmx, so it run with only 512mb of max heap.
And when messages size stored and not consumed is closed to the top, I get these behavior.