I am using Celery COnsumers and Rabbitmq for Publishing.
I am using upto 20 Worker Threads but as the time goes by, the messages_ready Count is very Huge.
Would this be a problem.
There would be hardly, few messages in the UNACKED count.
Have set message-ttl expires for Queue messages which doesnt work because i believe this doesnt work on messages in ready count.
How do i make sure, there are less messages in the "Ready" Count ?
Having high READY Count messages mean, that the Consumers are slow.
I also see the problem where consumers are not consuming after a point and the Ready Count doesnt go down at all.
Any Help would be great. Thanks
Related
I have a publisher that sends messages to a consumer that moves a motor.
The motor has a work queue which I cannot access, and it works slower than the rate of the incoming messages, so I'm trying to control the traffic on the consumer.
To keep updated and relevant data coming to the motor without the queue filling up and creating a traffic jam, I set the RabbitMQ queue size limit to 5 and basicQos to 1.
The idea is that the RabbitMQ queue will drop the old messages when it is filled up, so the newest commands are at the front of the queue.
Also by setting basicQos to 1 I ensure that the consumer doesn't grab all messages from the queue and bombards the motor at once, which is exactly what i'm trying to avoid since I can't do anything once the command was sent to the motor.
This way the consumer takes messages from the queue one by one, while new messages replace the old ones on the queue.
Practically this moves the bottleneck to the RabbitMQ queue instead of the motor's queue.
I also cannot check the motor's work queue, so all traffic control must be done on the consumer.
I added messageId and tested, and found out many messages are still coming and going long after the publisher is being shut down.
I'm expecting around 5 messages after shutdown since that's the size of the queue, but i'm getting hundreds.
I also added a few seconds of sleep inside the callback to make sure this isn't the robot queue that's acting up, but i'm still getting many messages after shutdown, and I can see in the logs the callback is being called every time so it's definitely still getting messages from somewhere.
Please help.
Thanks.
Moving the acknowledgment to the end of the callback solved the problem.
I'm guessing that by setting basicQos to 1 it did execute the callback for each message one after another, but in the background it kept grabbing messages from the queue.
So even when the publisher was shutdown, the consumer still had messages that were taken from the queue in it, and those messages were the ones that I saw being executed.
I have a RabbitMQ setup in which jobs are sent to an exchange, which passes them to a queue. A consumer carries out the jobs from the queue correctly in turn. However, these jobs are long processes (several minutes at least). For scalability, I need to be able to have multiple consumers picking a job from the top of the queue and executing it.
The consumer is running on a Heroku dyno called 'queue'. When I scale the dyno, it appears to create additional consumers for each dyno (I can see these on the RabbitMQ dashboard). However, the number of tasks in the queue is unchanged - the extra consumers appear to be doing nothing. Please see the picture below to understand my setup.
Am I missing something here?
Why are the consumers showing as 'idle'? I know from my logs that at least one consumer is actively working through a task.
How can my consumer utilisation be 0% when at least one consumer is definitely working hard.
How can I make the other three consumers actually pull some jobs from the queue?
Thanks
EDIT: I've discovered that the round robin dispatching is actually working, but only if the additional consumers are already running when the messages are sent to the queue. This seems like counterintuitive behaviour to me. If I saw a large queue and wanted to add more consumers, the added consumers would do nothing until more items are added to the queue.
To pick out the key point from the other answer, the likely culprit here is pre-fetching, as described under "Consumer Acknowledgements and Publisher Confirms".
Rather than delivering one message at a time and waiting for it to be acknowledged, the server will send batches to the consumer. If the consumer acknowledges some but then crashes, the remaining messages will be sent to a different consumer; but if the consumer is still running, the unacknowledged messages won't be sent to any new consumer.
This explains the behaviour you're seeing:
You create the queue, and deliver some messages to it, with no consumer running.
You run a single consumer, and it pre-fetches all the messages on the queue.
You run a second consumer; although the queue isn't empty, all the messages are marked as sent to the first consumer, awaiting acknowledgement; so the second consumer sits idle.
A new message arrives in the queue; it is distributed in round-robin fashion to the second consumer.
The solution is to specify the basic.qos option in the consumer. If you set this to 1, RabbitMQ won't send a message to a consumer until it has acknowledged the previous message; multiple consumers with that setting will receive messages in strictly round-robin fashion.
I am not familiar to Heroku, so I don't know how Heroku worker build rabbitMQ consumer, I just have a quick view over Heroku document.
Why are the consumers showing as 'idle'?
I think your mean the queue is 'idle'? Because the queue's state is about the queue's traffic, it just means there is not on-doing job for the queue's job thread. And it will become 'running' when a message is published in the queue.
How can my consumer utilisation be 0% when at least one consumer is definitely working hard.
The same as queue state, from official explanation, consumer utilisation too low means:
There were more consumers
The consumers were faster
The consumers had a higher prefetch count
In your situation, prefetch_count = 0 means no limits on prefetch, so it's too large. And Messages.total = Messages.unacked = 78 means your consumer is too slow, there are two many messages have been processed by consumer.
So if your message rate is not large enough, the state and consumer utilisation field of the queue is useless.
If I saw a large queue and wanted to add more consumers, the added consumers would do nothing until more items are added to the queue.
Because these unacked messages have already been prefetched by exist consumers, they will not be consumed by new consumers unless you requeue the unacked messages.
I have a rabbitmq queue that each data which publish to this queue is approximately 1 MB size. Every second 4 or 5 data publish to this queue.
Consumer consume each data one by one(Fetch=1). When i stop the consumer service 30000 queued message became ready to consume. When i start the consumer its consume rate 30/s+. It is just fine for now.
Hovewer in day light publisher never stops publishing and consumer can handle the queue. But at night publisher don't send data anymore(it is not error. It is how it suppose to be). At the first light of the next day publisher starts to publish 7 data per second. This time rabbit queue is started to go up continuesly.
First thought was that consumer can't handle the data. But it can consume 30/s+ data every sec.
I know that consume speed depends on consumer.
BUT.
I think that rabbit have some mechanism that after a time it decrease the consumer speed. Maybe it is lock mechanism maybe internal logs. I couldn't find any solution. Please help
This picture shows the limit of consumer speed.
Thanks to Lutz Horn
How is the consumer implemented? Does it use a push or a pull approach? It only consumes 3.8 messages per second. Edit: The screenshot shows a Consumer utilisation of 0%. This means that RabbitMQ always has to wait for the consumer to be able to handle the next message. RabbitMQ can never just push a published message to the consumer. See https://www.rabbitmq.com/blog/2014/04/14/finding-bottlenecks-with-rabbitmq-3-3/– Lutz Horn
I'm confused how rabbitmq works when a new consumer comes online.
I understand when there are currently x number of consumers connected, and then a producer sends a message the consumers will receive these messages.
But say consumerX was down, and now comes online or it is a brand new consumer. Is it possible for it to replay messages in the past 24 hours?
This is a normal behavior for RabbitMQ.
Please read:
https://www.rabbitmq.com/tutorials/tutorial-two-python.html
Is it possible for it to replay messages in the past 24 hours?
It depends on how you set things up.
If you have queues that don't auto-delete, they'll just keep collecting messages and waiting around for a consumer to connect.
I've had instances w/ thousands of messages stuck in a queue because my consumer was crashing. As soon as I fixed my code, the messages started consuming again.
But, if you're letting your queues get deleted when your consumers die, then you're in a bit of trouble.
There is a plugin to read the last ## of messages from an exchange, but it doesn't work in a time-based manner... just the last ## of messages: https://github.com/rabbitmq/rabbitmq-recent-history-exchange
I have made a consumer for RabbitMQ as a console application written in C#.NET. It is programmed to listen to a queue perpetually and whenever it find a message in the queue, it processes it. The consumer processes on an average 35 messages / second. The consumers are scheduled to run at system startup in the task scheduler. The consumers run fine for 3 - 4 days. But then, they keep on running but don't process any messages although the queue has messages in it. When the consumer is stopped and again started, it again starts processing the messages properly. But, by the time you manually restart, millions of messages get queued. Can someone please help me explain this abnormal behavior. I have other queues too which are running since months together without ceasing to stop.
Requesting prompt response. Thanks in advance to the experts.
I suggest you to look at the consumer code, it might be running but stuck in RabbitMQ exceptions. It sounds odd that it runs fine for 3-4 days.
I had similar problem of consumer not consuming messages in the queue because I was using "RabbitMQ.Client.QueueingBasicConsumer" to dequeue the message and when the queue was closed abruptly although consumer was running but it was in System.IO.EndOfStreamException. I am using "RabbitMQ.Client.Events.EventingBasicConsumer" which has helped me to solve the issue.