I have a RabbitMQ setup where a (java) producer sends messages to a fanout exchange, which are handled by a consumer. It's no problem if messages get lost when the consumer dies, so for performance I set autoAck=true at the consumer side.
Now I'm investigating a situation in which the rate the consumer can handle messages, is lower than the rate at which they are sent.
After a while, a (huge) backlog of messages must queue up somewhere. Is there a way to get visibility on this backlog?
Using the rabbitmqmanagement interface does not work: the queue appears empty
Ready: 0
Unacknowledged: 0
Total: 0
I assume the queue is empty because the messages are (unlimitedly) prefetched by the rabbitmqclient used by the consumer. But limiting the prefetch by e.g.
channel.basicQos(10)
does not help either, probably because this only limits unacknowledged messages, and with autoAck=true, messages are ack'ed from the moment they are prefetched by the client.
Setting autoAck=false (and explicit ack'ing on delivery) is a solution (the Unacknowledged counter keeps on rising), but I was wondering whether this is the only way?
Preferably I'd like to limit the amount of cached messages at the client side irrespective of acknowledgements, such that the backlog eventually becomes visible through the rabbitmqmanagement interface.
Alternatively, is there a way to query the number of messages sitting somewhere in the client's prefetch queue waiting to be delivered?
I suggest using a combination of basicQos and autoAck=false. This will make everything show up in the queues both through the admin website and the REST APIs. Having an unlimited number of messages sent to each consumer seems to defeat the point of a queue.
If your queues are time sensitive you can also add a TTL on the queues so that messages are automatically Nacked after (as an example) 60 minutes.
Related
Within my infrastructure I handle a lot of messages in RabbitMQ queues but recently I have a problem. In particular I'm having trouble with two queues consuming messages too slowly. And I discovered that they have a similar behavior that differentiates them from the rest of the queues that work perfectly well. The amount of consumer ack is always equal to the amount of redelivered message.
I would like to know what exactly this means. A first impression tells me that the messages cannot be processed and are returned by Rabbitmq, but in that case there should be no consumer acks.
I have a RabbitMQ setup in which jobs are sent to an exchange, which passes them to a queue. A consumer carries out the jobs from the queue correctly in turn. However, these jobs are long processes (several minutes at least). For scalability, I need to be able to have multiple consumers picking a job from the top of the queue and executing it.
The consumer is running on a Heroku dyno called 'queue'. When I scale the dyno, it appears to create additional consumers for each dyno (I can see these on the RabbitMQ dashboard). However, the number of tasks in the queue is unchanged - the extra consumers appear to be doing nothing. Please see the picture below to understand my setup.
Am I missing something here?
Why are the consumers showing as 'idle'? I know from my logs that at least one consumer is actively working through a task.
How can my consumer utilisation be 0% when at least one consumer is definitely working hard.
How can I make the other three consumers actually pull some jobs from the queue?
Thanks
EDIT: I've discovered that the round robin dispatching is actually working, but only if the additional consumers are already running when the messages are sent to the queue. This seems like counterintuitive behaviour to me. If I saw a large queue and wanted to add more consumers, the added consumers would do nothing until more items are added to the queue.
To pick out the key point from the other answer, the likely culprit here is pre-fetching, as described under "Consumer Acknowledgements and Publisher Confirms".
Rather than delivering one message at a time and waiting for it to be acknowledged, the server will send batches to the consumer. If the consumer acknowledges some but then crashes, the remaining messages will be sent to a different consumer; but if the consumer is still running, the unacknowledged messages won't be sent to any new consumer.
This explains the behaviour you're seeing:
You create the queue, and deliver some messages to it, with no consumer running.
You run a single consumer, and it pre-fetches all the messages on the queue.
You run a second consumer; although the queue isn't empty, all the messages are marked as sent to the first consumer, awaiting acknowledgement; so the second consumer sits idle.
A new message arrives in the queue; it is distributed in round-robin fashion to the second consumer.
The solution is to specify the basic.qos option in the consumer. If you set this to 1, RabbitMQ won't send a message to a consumer until it has acknowledged the previous message; multiple consumers with that setting will receive messages in strictly round-robin fashion.
I am not familiar to Heroku, so I don't know how Heroku worker build rabbitMQ consumer, I just have a quick view over Heroku document.
Why are the consumers showing as 'idle'?
I think your mean the queue is 'idle'? Because the queue's state is about the queue's traffic, it just means there is not on-doing job for the queue's job thread. And it will become 'running' when a message is published in the queue.
How can my consumer utilisation be 0% when at least one consumer is definitely working hard.
The same as queue state, from official explanation, consumer utilisation too low means:
There were more consumers
The consumers were faster
The consumers had a higher prefetch count
In your situation, prefetch_count = 0 means no limits on prefetch, so it's too large. And Messages.total = Messages.unacked = 78 means your consumer is too slow, there are two many messages have been processed by consumer.
So if your message rate is not large enough, the state and consumer utilisation field of the queue is useless.
If I saw a large queue and wanted to add more consumers, the added consumers would do nothing until more items are added to the queue.
Because these unacked messages have already been prefetched by exist consumers, they will not be consumed by new consumers unless you requeue the unacked messages.
Shovels
consumes messages from the queue,
re-publishes each message to the destination broker (using, by default, the original exchange name and routing_key when
applicable).
I could not find any documentation what's the expected behavior for message TTLs when shovels are involved:
Does the time used for calculating TTL start when message is received
at the source broker or at the destination broker? Or is it just valid for the first publish, that is at the source broker?
What happens if the expiration time elapses before the message reaches the destination broker?
So, I think you answered the question in the documentation you pasted in. All shovel does is move messages from one queue to another, re-publishing them in the process. It's going to preserve all original message properties, which theoretically includes the TTL property.
That being said, I don't believe this is something you need to worry about.
Message TTL starts when the queue receives the message. When the message is re-published, the clock resets on the new queue.
Messages being transported by shovel will ideally spend no more than a few milliseconds in the initial queue, if they even end up there at all (a message queue with a consumer attached doesn't actually enqueue any messages under most conditions). So, the time spent in the first queue should be so small that it doesn't matter.
Message lifetime should have a fair amount of tolerance for network transport, etc., so the activities of shovel are on par with the normal noise.
If you find yourself in the situation where a large number of messages are accumulating in the queue before they can be shovel'd, then you might need to handle expiration in your application. There are other benefits and caveats to doing this, but you get a little finer-grained control overall.
We're seeing an issue where consumers of our message queues are picking up messages from queues at the top of the alphabetical range. We have two applications: a producer, and a subscriber. We're using RabbitMQ 3.6.1.
Let's say that the message queues are setup like so:
Our first application, the producer, puts say 100 messages/second onto each queue:
Our second application, the subscriber, has five unique consumer methods that can deal with messages on each respective queue. Each method binds to it's respective queue. A subscriber has a prefetch of 1 meaning it can only hold one message at a time, regardless of queue. We may run numerous instances of the subscriber like so:
So the situation is thus: each queue is receiving 100 msg/sec, and we have four instances of subscriber consuming these messages, so each queue has four consumers. Let's say that the consumer methods can deal with 25 msg/sec each.
What happens is that instead of all the queues being consumed equally, the alphabetically higher queues instead get priority. It's seems as though when the subscriber becomes ready, RabbitMQ looks down the list of queues that this particular ready channel is bound to, and picks the first queue with pending messages.
In our situation, A_QUEUE will have every message consumed. B_QUEUE may have some consumed in certain race conditions, but C_QUEUE/D_QUEUE and especially E_QUEUE will rarely get touched.
If we turn off the publisher, the queues will eventually drain, top to bottom.
Is it possible to configure either RabbitMQ itself or possibly even the channel to use some sort of round robin distribution policy or maybe even random policy so that when a channel has numerous bound queues, all with messages pending, the distribution is even?
to clarify: you have a single subscriber application with multiple consumers in it, right?
I'm guessing you're using a single RabbitMQ Connection within the subscriber app.
Are you also re-using a single RabbitMQ Channel for all of your consumers? If so, that would be a problem. Be sure to use a new Channel for each consumer you start.
Maybe the picture is wrong, but if it's not then your setup is wrong. You don't need 4 queues if you are going to have subscribers that listen to each and every queue. You'd just need one queue, that has multiple instances of the same subscriber consuming from it.
Now to answer, yes (but no need to configure, as long as prefetch is 1), actually rabbitmq does distribute messages evenly. You can find about about that here, and on the same place actually how your setup should look like. Here is a quote from the link.
RabbitMQ just dispatches a message when the message enters the queue.
It doesn't look at the number of unacknowledged messages for a
consumer. It just blindly dispatches every n-th message to the n-th
consumer.
We are building a solution in which we are publishing message to a time-out queue. After TTL expiry messages are pushed to main queue for re-processing.
We are setting up counter value so that messages will be tried for x no. of times for the redelivery.
Solution is working fine. But the scenario is when the message on the head position is highest TTL is not expired, other messages of lower expiry will not be re-published (to main queue).
Is this understanding correct ? If Yes what is the solution so that each message re-processed just after TTL.
Appreciating answers / viewpoint.
Thanks.
If you use per-queue message TTL, then message expires and get removed from queue from head to tail (in the same order they was published).
When you use per-message TTL, then messages removed from queue only when they reach queue head, so situation when expired messages still reside in the middle of queue is normal. Such messages will not be send to consumer, and will be deadlettered (or dropped), but due to strict FIFO nature or RabbitMQ's queues that will happen as written above, when they reach queue head and delay before removal may be greater than actual message TTL. For example, if there are two message, first with TTL=10sec and the second one with TTL=1sec, second message will be deadlettered also in 10sec while it stay after first one.
To deal with messages that has different TTL, common workaround is to declare few queues, each for messages with same TTL or almost same, say, with precision 10sec. Actual precision may vary while it very application-specific and somehow empirical value.
If you will pick separate per-TTL queues, use per-queue TTL rather than per-message TTL for ease of messages workflow and to prevent disambiguation of understanding what happens with messages. Developers after you will thank you for that.
To re-process messages after their TTL use Dead Letter Exchanges, but beware of cycled messages problem: if RabbitMQ broker detects that your messages workflow cycled (messages get published to same exchange with the same routing key after it was deadlettered from it), it will silently drop message.
the queue ttl is simple enough and working fine.
but set per message ttl is not working expectly: each message publish to online consumer just after ttl.
why rabbitmq provide this feature? for which biz scenario?