I'm currently using ActiveMQ for my queueing system, and I'm wanting to make the transition to RabbitMQ. One feature I've been using that belongs to ActiveMQ is a redelivery policy, as sometimes our consumer rejects a message because it cannot handle it at this time, but may want to try again later, so it requeues it.
Right now in AMQP, when I reject a message, it's instantly pulled off the queue again immediately and tried again.
Is there a way, in RabbitMQ, to specify a redelivery policy for a queue, consumer, or message?
I also had problems with that behaviour. According to documentation (as far as I remember, maybe in newer version something changed) after requeue it is not stated where a message will be placed (it was described as undetermined). In my testcases (with version 2.8.2) some of messages were put to the end of a queue and one message (precisely first from clients prefetch) land on beggining (and being consumed immediately). In our application this caused livelock.
You could walkaround this by publishing copy of message to a queue and acking already delivered one in one transaction (but I recommend to carefully read section about transactions in docs) or use deadlettering to deal with temporaly unprocessable messages.
Related
I have an queue and messages in it. Also i have two consumer in separate processes. I take message by one, and decide that this message is not mine, and reject it with requeue flag. In documentation I found the next phrase "The server MUST NOT deliver the message to the same client within the context of the current channel". Is that mean that the rejected message should be deliver to another consumer or not?
So, there are a couple of things going on here that I'd like to touch on.
First, your question as to the behavior of RabbitMQ. The rule referenced above comes from the AMQP-0-9-1 specification. As with most implementation of open specs, RabbitMQ is not fully-conforming. This page describes in precise detail exactly which portions of the specification are implemented, and where any deviations occur.
On that page, it stipulates that "No attempt is made to prevent redelivery to the same client." RabbitMQ lists this as a planned addition in a future release, but it has been planned for quite a few years now.
Should Consumers Be Picky?
The more important question is the one you haven't directly asked, but that is "should my consumer be picky about which messages from the queue it processes?
The answer to this is a definitive "no." One of the key design assumptions about message queues is that any consumer subscribed to the queue should be able to process any message in the queue. Thus, it should be considered proper design that all consumers attached to the queue are running identical code (same code base, same version). If not, you're going to have some serious problems with your application sooner or later.
Reject should only be used to tell the broker that there is a problem with a particular message. If there is a problem with a particular consumer (e.g. loses connection to a database), it should not reject the message, but instead should close the connection, triggering redelivery to another, working consumer. By design, messages that need to be processed by a specialized or different consumer should be deposited in a different queue.
I'm going through a few examples using NServiceBus and I've stumbled across a feature I'm hoping ships with MassTransit (As it is a free service).
The feature is based around 'poisoned' messages.
If, due to a bug in your system, these messages cant ever be handled, and end up permanently in the error queue.
NServiceBus has a cool feature whereby, once you have corrected the bugs in your code, allows those messages in the error queue to be 'redirected' to the original working queue, to be redelivered.
This is done by using a NServiceBus specific tool :- ReturnToSourceQueue.exe.
Does MassTransit have a similar tool for this kind of issue?
Or is there another workaround availble, preferbly to work with RabbitMQ.
With RabbitMQ, it's easy to move messages between queues. You can use the management console to do it manually, by installing the shovel plug-in.
You can also create shovels in RabbitMQ that are scheduled, and perform the message movement in response to that schedule. The visibility of having the shovels configured in RabbitMQ has been invaluable to our operations staff, since they rarely think that a Windows Scheduled Task (or other random scheduler) is going to be doing something as risky as moving previously failed messages back into the production queues.
I would suggest reading this blog post on how MassTransit deals with poison messages: Error Handling in MassTransit with RabbitMQ
The tooling around RabbitMQ is so much better than anything MSMQ provides, which is one of the reasons we have completely abandoned MSMQ for production queuing.
This functionality is easily recreated with nothing more than RabbitMQ and a bit of code. While it's nice that NServicebus includes it, building it with MassTransit should be easy enough.
(note: i haven't used .NET in a few years, so my knowledge of NSB and MT are a bit rusty... this will be high level answer only, no code)
The thing to start with, is a proper configuration of a dead letter exchange and a poison message queue. https://www.rabbitmq.com/dlx.html
Once you have knowledge that a message is causing errors and is a bad message, you can reject or nack (with no requeue) the message in order to send it through the dead letter exchange (DLX).
Once a message has gone through the DLX, you will have some additional properties on the message, including:
queue - the name of the queue the message was in before it was dead-lettered,
exchange - the exchange the message was published to (note that this will be a dead letter exchange if the message is dead lettered multiple times),
routing-keys - the routing keys (including CC keys but excluding BCC ones) the message was published with,
there will be more, but these are the things you want to pay attention to. by examining these properties on the message, you can re-send the original message back through the original exchange, with the original routing-keys. alternatively, you can re-send straight to the original destination queue... i think sending through the exchange would be better, personally, as the original queue might not exist anymore (depending on system configuration, consumers creating exclusive queues, etc).
with this information, recreating the feature set should not be too difficult. rabbitmq provides all of the features that you need, you just have to write a bit of code to take advantage of it.
I have thousands of unacked messages in my dev environment which I can't restart.
Is there a way to remove (purge) all messages even if they are unacknowledged?
Close the channel that the unacked messages reside on, which will nack them back into the queue, then call purge.
You have to make consumer ack them (or nack) and only after that they will be removed. Alternatively you can shutdown consumers and purge the queue completely.
If you are looking for some way to purge all unacked messages - there are no such feature nor in AMQP protocol neither in RabbitMQ.
It looks like your consumer is the cause of the problem, so you have to adjust it (rewrite) to release message immediately after it processed or failed.
Once there are no "ready" messages in the queue, delete it and recreate.
YOU WILL LOSE THE QUEUE CONTENTS with this method.
You need to put messages back into the queue before you can purge them:
close the channel
close the connection (the script doesn't work for me)
As an alternative, this doesn't require to wait:
delete and recreate the queue
restart the server
You need to call basic.recover to force all unacked messages to be re-enqueued to a channel that failed. Be aware of the errata concerning this function specifying that only the requeue mode is supported by RabbitMQ.
For software developer use below code.
channel.purgeQueue(queue-name);
if we use this code the Queue will be clear and same queue will exist.
One way this can happen is if the consumer is stuck recycling the same messages due to a processing error. In this case, the RabbitMQ queue management interface may show the messages as Unacked, but really they are being read from the queue and processed (to the point of the failure) then requeued (to enable a retry) at a rapid pace -- maybe thousands of times per second.
During this loop, the messages exist briefly in the Ready state, but are immediately removed again by you application -- and the cycle begins again. As an example, this auto-requeue behavior is the default for Spring AMQP.
Since the messages are never left in the Ready state, the Management Interface's Get Message(s) button is unlikely work. What can work, if you have queue access, is to run a separate custom consumer instance, perhaps locally, but with the specific intent of removing and not requeuing the messages in question.
By RabbitMQ's Fair Dispatch mechanism, your additional consumer will likely receive the messages in question and have the opportunity to perform your custom handling.
You might even write a custom utility to do this, with logic to filter, analyze, or deadletter the messages of interest.
If you want to clear the contents of the queue, then you can use the AMQP method queue.purge: There is queue purge in AMQP: http://www.rabbitmq.com/amqp-0-9-1-reference.html#queue.purge
You could do similar using the management plugin.
I'm trying to implement kind of "Exclusive Consumer" or "Exclusive Queues" which avoids the end user having to pin anything: the broker will pick a single Message-Consumer to get all the messages for a queue to ensure ordering, and if that consumer fails, the broker will auto failover and choose another consumer.
I wondered if a combination of Celery and RabbitMQ can make it.
Despite realising that this question is more than 7 years old, someone may stumble across this page as I just did. So, in 2019, RabbitMQ team has released a version 3.8 which supports new feature called "Single Active Consumer". It offers exactly that type of consumer-exclusivity combined with automatic fallback behavior, where the next (randomly chosen) consumer listening to the queue will be made the "single active consumer" upon the first consumer falling off or disconnecting for any reason. And this feature does not require any complicated setup of the message queue itself; it works with almost any queue.
Read more at https://www.cloudamqp.com/blog/2019-04-23-rabbitmq-3-8-feature-focus-single-active-consumer.html and https://www.rabbitmq.com/consumers.html#single-active-consumer
Though RabbitMQ has the concept of "exclusive" queues, I do not think you can get the fallback behavior that you need.
The broker will not "choose" a single consumer. Failure modes can be addressed by not using auto-ACK, rejecting on error and letting your worker die/restart.
This preserves message order if there is only one consumer. With load-balancing your mileage might vary.
I am using RabbitMQ server.
For publishing messages, I set the immediate field to true and tried sending 50,000 messages. Using rabbitmqctl list_queues, I saw that the number of messages in the queue was zero.
Then, I changed the immediate flag to false and again tried sending 50,000 messages. Using rabbitmqctl list_queues, I saw that a total of 100,000 messages were in queues (till now, no consumer was present).
After that, I started a consumer and it consumed all the 100,000 messages.
Can anybody please help me in understanding about the immediate bit field and this behavior too? Also, I could not understand the concept of the mandatory bit field.
The immediate and mandatory fields are part of the AMQP specification, and are also covered in the RabbitMQ FAQ to clarify how its implementers interpreted their meaning:
Mandatory
This flag tells the server how to
react if a message cannot be routed to
a queue. Specifically, if mandatory is
set and after running the bindings the
message was placed on zero queues then
the message is returned to the sender
(with a basic.return). If mandatory
had not been set under the same
circumstances the server would
silently drop the message.
Or in my words, "Put this message on at least one queue. If you can't, send it back to me."
Immediate
For a message published with immediate
set, if a matching queue has ready
consumers then one of them will have
the message routed to it. If the lucky
consumer crashes before ack'ing
receipt the message will be requeued
and/or delivered to other consumers on
that queue (if there's no crash the
messaged is ack'ed and it's all done
as per normal). If, however, a
matching queue has zero ready
consumers the message will not be
enqueued for subsequent redelivery on
from that queue. Only if all of the
matching queues have no ready
consumers that the message is returned
to the sender (via basic.return).
Or in my words, "If there is at least one consumer connected to my queue that can take delivery of a message right this moment, deliver this message to them immediately. If there are no consumers connected then there's no point in having my message consumed later and they'll never see it. They snooze, they lose."
http://www.rabbitmq.com/blog/2012/11/19/breaking-things-with-rabbitmq-3-0/
Removal of "immediate" flag
What changed? We removed support for the
rarely-used "immediate" flag on AMQP's basic.publish.
Why on earth did you do that? Support for "immediate" made many parts
of the codebase more complex, particularly around mirrored queues. It
also stood in the way of our being able to deliver substantial
performance improvements in mirrored queues.
What do I need to do? If you just want to be able to publish messages
that will be dropped if they are not consumed immediately, you can
publish to a queue with a TTL of 0.
If you also need your publisher to be able to determine that this has
happened, you can also use the DLX feature to route such messages to
another queue, from which the publisher can consume them.
Just copied the announcement here for a quick reference.