Im trying to create infinite job queue using redis and ruby eventmachine. To achieve that Im using redis BLPOP command with 0 timeout. After successful BLPOP I run it again.
Am I on the right way or there is a better way to create job queue with redis?
If you use BLPOP alone to remove a message from the queue, and your message consumer fails to process it, the message will have to be re-queued, lest it disappear forever along with the failed consumer.
For more durable message processing, a list of messages being processed must be maintained so they can be re-queued in the event of failure.
[B]RPOPLPUSH is perfect for this scenario; it can atomically pop a message from the message queue and pushes it onto a processing queue so that the application can respond in the case of a failure on the consumer's end.
http://redis.io/commands/rpoplpush
Actual re-queueing is left to the application, but this redis command provides the foundations to do so.
There are also some drop-in-place implementations of queues using redis floating around the web, such as RestMQ [ http://www.restmq.com/ ]
Related
I am using celery and rabbitmq , but due to pushing several task in queue my server memory utilization becomes more than 40% , so that rabbit further will not accepting any task . so i want to delete those message which are already executed , but due to durable behavior of rabbitmq those message not automatically delete, so i want to set some configuration like autoAck=True , so that if message is consumed from celery ,it will delete from rabbitmq queues and also from my server memory. please explain how can we do that .
OK, so while I don't fully understand why you have the problem you have, it is clear what is going on.
A publisher puts a message task in the queue
Your worker process pulls the message and processes it
The message is never actually removed from the queue
This behavior happens when a consumer fails to acknowledge the processing of a message. To confirm, if you look at the RabbitMQ management plug-in, you'll see a whole bunch of unacknowledged messages. These will be unavailable for consumption, but will continue to be held on the server and taking up disk space and memory.
Further, if you do a Basic.Recover, all of these messages will then get dumped back into the queue to be processed again.
This problem is due to incorrect configuration of your consumer. There are two ways to address this:
You can configure the consumer to auto-ack (i.e. acknowledge the message automatically upon receipt). This is done when you declare the consumer (using Basic.Consume). Edit: It looks like this may be the default behavior of Celery.
You can configure your worker process to submit an acknowledgement (using Basic.Ack). Edit: this is done via the acks_late property in Celery.
I have got several messages on error queue which has name TestQueue_errors.
One of the messages on error queue is important and should be moved back to service queue TestQueue so it can be processed again. The other messages on error queue are broken and should stay on error queue.
I have tried to do that with shovel plugin but it seems it is able only to move all messages from one queue to another. Is there a way I could achieve that, to move single message from one queue to another?
As far as I know Rabbit Management does not allow to do it. The only thing you can do is to publish this message again.
Maybe there are some tools which give possibility to achieve it but it is not a standard behaviour.
Here are actions which you are able to perform on the queue (from RabbitMQ Management page):
Move all messages from one queue to another
Get all messages without requeue option (they would not be in the queue anymore)
Get first N messages without requeue option and then move the rest of messages to another queue
I have thousands of unacked messages in my dev environment which I can't restart.
Is there a way to remove (purge) all messages even if they are unacknowledged?
Close the channel that the unacked messages reside on, which will nack them back into the queue, then call purge.
You have to make consumer ack them (or nack) and only after that they will be removed. Alternatively you can shutdown consumers and purge the queue completely.
If you are looking for some way to purge all unacked messages - there are no such feature nor in AMQP protocol neither in RabbitMQ.
It looks like your consumer is the cause of the problem, so you have to adjust it (rewrite) to release message immediately after it processed or failed.
Once there are no "ready" messages in the queue, delete it and recreate.
YOU WILL LOSE THE QUEUE CONTENTS with this method.
You need to put messages back into the queue before you can purge them:
close the channel
close the connection (the script doesn't work for me)
As an alternative, this doesn't require to wait:
delete and recreate the queue
restart the server
You need to call basic.recover to force all unacked messages to be re-enqueued to a channel that failed. Be aware of the errata concerning this function specifying that only the requeue mode is supported by RabbitMQ.
For software developer use below code.
channel.purgeQueue(queue-name);
if we use this code the Queue will be clear and same queue will exist.
One way this can happen is if the consumer is stuck recycling the same messages due to a processing error. In this case, the RabbitMQ queue management interface may show the messages as Unacked, but really they are being read from the queue and processed (to the point of the failure) then requeued (to enable a retry) at a rapid pace -- maybe thousands of times per second.
During this loop, the messages exist briefly in the Ready state, but are immediately removed again by you application -- and the cycle begins again. As an example, this auto-requeue behavior is the default for Spring AMQP.
Since the messages are never left in the Ready state, the Management Interface's Get Message(s) button is unlikely work. What can work, if you have queue access, is to run a separate custom consumer instance, perhaps locally, but with the specific intent of removing and not requeuing the messages in question.
By RabbitMQ's Fair Dispatch mechanism, your additional consumer will likely receive the messages in question and have the opportunity to perform your custom handling.
You might even write a custom utility to do this, with logic to filter, analyze, or deadletter the messages of interest.
If you want to clear the contents of the queue, then you can use the AMQP method queue.purge: There is queue purge in AMQP: http://www.rabbitmq.com/amqp-0-9-1-reference.html#queue.purge
You could do similar using the management plugin.
I want a consumer to perform some actions every time that a message is received. Must the consumer be running 24/7 "listening" to the queue or it can be run only when an appropiate message is received?
I am not sure your question makes sense. The message can only be received from a queue by the consumer of a queue. To know if a message is in the queue one must look at the queue. The only way to do that is to be a consumer.
If you really want you could have a script that ran the commandline interface for the management plugin. That could poll the queue and when it had a size of more than one could start a program that would run a consumer to consume from the queue.
None of this makes any sense. If it is just sitting waiting for the queue and doing nothing else it is hardly consuming any resources so I do not see what the problem would be running a consumer 24/7.
Of course the consumer doesn't have to run 24/7, thats part of the point of MQ. It is asynchronous. The consumer does not have to be running when the producer writes to the queue. You could therefore have a scheduled task that runs your consumer periodically to check and process messages from the queue. But I do not think that is what you want.
The whole point of listening is: do nothing until a message comes, process the message, do nothing until the next message. This is what you want the first sentence of your question. Why the problem with listening?
We've been using Rabbit successfully for about a year. Recently have upgraded to v2.6.1, because we want to use clusters with replicated message queues.
My testing has hit a puzzling behavior that smells like a Rabbit bug to me. The test that uncovers this is working with a two-node cluster. Both nodes are running v2.6.1. Both nodes have disk. Both nodes are running on Mac OS, though I doubt this is pertinent.
I'm also running Alice on the node that runs the test. The test uses it to programmatically do a stop_app on one of the nodes, because the test is trying to validate that if the cluster master fails, and a slave is elevated to take its place, that we don't lose messages.
So, the test has a small thread pool, which is given tasks that periodically 1) publish messages, and 2) toggle the state of the Rabbit master node (stopped if running; started if stopped). Other threads are consuming messages from queues.
I'm using publisher confirms, and I'm also acknowledging the messages in the consumers (using autoAck=false for channel.basicConsume()).
When the master node is stopped, I see both the producers and consumers catching ShutdownSignalException. They handle this by attempting to reconnect to the cluster. This works fine. When reconnected, they continue with their business.
Sometimes, what I see is that a consumer has successfully fetched a message from the broker, and is calling channel.basicAck() when it gets that ShutdownSignalException.
Later, when the consumer has reconnected, it again pulls down the same message. (The message bodies are tagged with a UUID, so I know it is the same one.) This time, when the consumer attempts to basicAck() the message, it again gets ShutdownSignalException, but this one has the following text in it: "reply-text=PRECONDITION_FAILED - unknown delivery tag 7".
In fact, that is the same delivery tag that was offered to the consumer by the broker before the master went down and the consumer reconnected.
Googling suggests that this event means that the consumer is attempting to ack the same message more than once.
But, how can this be so? If the first ack succeeded, then the message should have been removed from the broker's queues, and the consumer shouldn't see the same message again.
Yet, if the first ack did not succeed, then the consumer shouldn't be dinged for attempting to re-ack the message.
Anyone seen this before? It smells like a bug in Rabbit's replicated queues to me, but I've still new to Rabbit, and so am willing to believe there's a subtlety here in consuming from a clustered broker that I haven't yet grokked!
Thanks, --Steve
I'm not sure if my case matching yours, but I have seen similar "unknown delivery tag" on attempts to ack after reconnect and then the same message arrived again. Initially it looked like a bug to me, but in fact this is expected behavior. Consumer with QOS>1 may have in it's local buffer some messages and delivery tag will be invalid for all o them after reconnect. From another hand, attempt to ack even the current message after reconnect doesn't make any sense, because that message already nacked automatically on connection lost and this is why I got it again.