I have a situation where messages are not getting cleared from Que (please refer to above pic), and I am not sure which que they belong to, so how do I figure that out and how to purge these queued messages.
any additional messages coming in are being cleared but these 21 messages are stuck.
I found how to do it,Under Ques sort by Total or ready then click on that Que and Perge messages.
Related
I have a publisher that sends messages to a consumer that moves a motor.
The motor has a work queue which I cannot access, and it works slower than the rate of the incoming messages, so I'm trying to control the traffic on the consumer.
To keep updated and relevant data coming to the motor without the queue filling up and creating a traffic jam, I set the RabbitMQ queue size limit to 5 and basicQos to 1.
The idea is that the RabbitMQ queue will drop the old messages when it is filled up, so the newest commands are at the front of the queue.
Also by setting basicQos to 1 I ensure that the consumer doesn't grab all messages from the queue and bombards the motor at once, which is exactly what i'm trying to avoid since I can't do anything once the command was sent to the motor.
This way the consumer takes messages from the queue one by one, while new messages replace the old ones on the queue.
Practically this moves the bottleneck to the RabbitMQ queue instead of the motor's queue.
I also cannot check the motor's work queue, so all traffic control must be done on the consumer.
I added messageId and tested, and found out many messages are still coming and going long after the publisher is being shut down.
I'm expecting around 5 messages after shutdown since that's the size of the queue, but i'm getting hundreds.
I also added a few seconds of sleep inside the callback to make sure this isn't the robot queue that's acting up, but i'm still getting many messages after shutdown, and I can see in the logs the callback is being called every time so it's definitely still getting messages from somewhere.
Please help.
Thanks.
Moving the acknowledgment to the end of the callback solved the problem.
I'm guessing that by setting basicQos to 1 it did execute the callback for each message one after another, but in the background it kept grabbing messages from the queue.
So even when the publisher was shutdown, the consumer still had messages that were taken from the queue in it, and those messages were the ones that I saw being executed.
I'm trying to gracefully shutdown a process that is consuming messages off RabbitMQ. I know that I can use Channel.cancel to stop RabbitMQ from sending any new messages to the process, but I need to deal with any pending, unacknowledged messages as well. I could just call Channel.nackAll and re-queue all of them, but it would be better if I could wait till all of the pending messages - messages that were consumed, but not yet acked/nacked - were done.
Any ideas how to achieve this?
Cancel the consumer
Ack the messages you are currently "working" on when the work is done
When the above is done, close channels and connections. Anything that is un-acked will be re-enqueued by RabbitMQ.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
I'm confused how rabbitmq works when a new consumer comes online.
I understand when there are currently x number of consumers connected, and then a producer sends a message the consumers will receive these messages.
But say consumerX was down, and now comes online or it is a brand new consumer. Is it possible for it to replay messages in the past 24 hours?
This is a normal behavior for RabbitMQ.
Please read:
https://www.rabbitmq.com/tutorials/tutorial-two-python.html
Is it possible for it to replay messages in the past 24 hours?
It depends on how you set things up.
If you have queues that don't auto-delete, they'll just keep collecting messages and waiting around for a consumer to connect.
I've had instances w/ thousands of messages stuck in a queue because my consumer was crashing. As soon as I fixed my code, the messages started consuming again.
But, if you're letting your queues get deleted when your consumers die, then you're in a bit of trouble.
There is a plugin to read the last ## of messages from an exchange, but it doesn't work in a time-based manner... just the last ## of messages: https://github.com/rabbitmq/rabbitmq-recent-history-exchange
I am using the alerts feature of IronMQ service provided by IronIO to start workers.
I have things setup so that a message is pushed onto the push queue. The push queue sends an alert that starts a worker. The worker pulls off the message on the push queue, reserving it. Sometimes for whatever reason the job fails, the reservation for a message expires, and the message becomes available again. However, from what I can tell, no alert is sent when the reservation expires on a message. So the message sits in the queue until another message is added to the queue firing an alert and starting a worker. But the new message is not processed.
Are alerts created for messages that have a reservation expire in IronMQ? Is there any documentaion that I missed describing what can happen?
I am working on having workers pull off multiple messages but I am running into issues unrelated to iron io when processing multiple messages in the same worker.
Also is there a way to pull off the top of the queue. To avoid pulling off messages that may be causing errors? Should I just modify my workers to delete messages causing errors?
Currently there are no alerts for when a message times out and goes back on the queue, but that does seem like it would be a good idea. I assume this is a pretty inactive queue? I made a feature request for this here: https://trello.com/c/XcHi0NdN/35-fire-alert-when-a-message-times-out-goes-back-on-queue
And regarding messages that are causing issues, your best bet would be to add them to a different queue (an error queue) and delete them off the original queue. Then you can go through the error queue to figure out why certain messages are causing you problems. This is known as a "dead letter queue" btw and we have a feature request for it here, please give it a vote! https://trello.com/c/bGnJcNa9/26-dead-letter-queue
We are currently evaluating RabbitMQ. Trying to determine how best to implement some of our processes as Messaging apps instead of traditional DB store and grab. Here is the scenario. We have a department of users who perform similar tasks. As they submit work to the server applications we would like the server app to send messages back into a notification window saying what was done - to all the users, not just the one submitting the work. This is all easy to do.
The question is we would like these message to live for say 4 hours in the Queue. If a new user logs in or say a supervisor they would get all the messages from the last 4 hours delivered to their notification window. This gives them a quick way to review what has recently happened and what is going on without having to ask others, "have you talked to John?", "Did you email him is itinerary?", etc.
So, how do we publish messages that have a lifetime of x hours from the time they were published AND any new consumers that connect will get all of these messages delivered in chronological order? And preferably the messages just disappear after they have expired from the queue.
Thanks
There is Per-Queue Message TTL and Per-Message TTL in RabbitMQ. If I am right you can utilize them for your task.
In addition to the above answer, it would be better to have the application/client publish messages to two queues. Consumer would consume from one of the queues while the other queue can be configured using per queue-message TTL or per message TTL to retain the messages.
Queuing messages you do to get a message from one point to the other reliable. So the sender can work independently from the receiver. What you propose is working with a temporary persistent store.
A sql database would fit perfectly, but also a mongodb would work nicely. You drop a document in mongo, give it a ttl and let the database handle the expiration.
http://docs.mongodb.org/master/tutorial/expire-data/