I have an queue and messages in it. Also i have two consumer in separate processes. I take message by one, and decide that this message is not mine, and reject it with requeue flag. In documentation I found the next phrase "The server MUST NOT deliver the message to the same client within the context of the current channel". Is that mean that the rejected message should be deliver to another consumer or not?
So, there are a couple of things going on here that I'd like to touch on.
First, your question as to the behavior of RabbitMQ. The rule referenced above comes from the AMQP-0-9-1 specification. As with most implementation of open specs, RabbitMQ is not fully-conforming. This page describes in precise detail exactly which portions of the specification are implemented, and where any deviations occur.
On that page, it stipulates that "No attempt is made to prevent redelivery to the same client." RabbitMQ lists this as a planned addition in a future release, but it has been planned for quite a few years now.
Should Consumers Be Picky?
The more important question is the one you haven't directly asked, but that is "should my consumer be picky about which messages from the queue it processes?
The answer to this is a definitive "no." One of the key design assumptions about message queues is that any consumer subscribed to the queue should be able to process any message in the queue. Thus, it should be considered proper design that all consumers attached to the queue are running identical code (same code base, same version). If not, you're going to have some serious problems with your application sooner or later.
Reject should only be used to tell the broker that there is a problem with a particular message. If there is a problem with a particular consumer (e.g. loses connection to a database), it should not reject the message, but instead should close the connection, triggering redelivery to another, working consumer. By design, messages that need to be processed by a specialized or different consumer should be deposited in a different queue.
Related
Saw similar questions but different expected answers. My question is can I can create a consumer to focus on a single queue until it empties then switch to the other queue, until new work is sent to their main queue?
For example: 1 queue contains large amount of work to be processed in longer time frame and its own dedicated consumers (3 for instance). The 2nd queue receives much less work that requires less processing. If the consumers for the 2nd queue complete their work can I make it so they switch to the first queue until their queue receives more work?
I think for this question, it's important to keep in mind that there is a difference between a "consumer" in the canonical sense vs. a "consumer" in the RabbitMQ sense.
A RabbitMQ Consumer is a contrivance of the protocol - basically, it is a designation that the channel/connection would like to have messages pushed to it, under a designated consumer tag. In this sense, it is merely a notification to the broker to immediately send messages.
In the canonical sense, a message consumer is any piece of code that processes messages.
So, the answer to your question is "yes, go ahead and write your program to do that." You have control over the canonical consumer code. It is up to your software to determine what to do with a message that arrives from a queue.
Now, if you're wondering if RabbitMQ can re-subscribe a consumer to a different queue, the answer is "that's not how it works." In RabbitMQ, a consumer is simply a response to a request to subscribe to a queue - it is a "consumer tag" object. The ongoing nature of the subscription is tied to the channel/connection pair.
What should you do? While your code doesn't specify any particular coding language, in my opinion, you're off-track by even asking this question. Subscribe to both queues. If there is nothing for the worker to do, I think the computer would be perfectly happy with that. If you're worried about a particularly busy queue issuing too much work, you can use a number of techniques to throttle messages coming into that consumer. One popular technique is prefetch.
One of the main characteristics of a message queue service, RabbitMQ included, is preserving message publication order. This is confirmed in the RabbitMQ documentation:
[QUOTE 1] Section 4.7 of the AMQP 0-9-1 core specification explains the
conditions under which ordering is guaranteed: messages published in
one channel, passing through one exchange and one queue and one
outgoing channel will be received in the same order that they were
sent. RabbitMQ offers stronger guarantees since release 2.7.0.
Let's assume in the following that there are no consumers active, to simplify things. We are publishing over one single channel.
So far, so good.
RabbitMQ also provides possibility to inform the publisher that a certain publication has been completely and correctly processed [*]. This is explained here. Basically, the broker will either send a basic.ack or basic.nack message. The documentation also says this:
[QUOTE 2] basic.ack for a persistent message routed to a durable queue will be
sent after persisting the message to disk.
In most cases, RabbitMQ will acknowledge messages to publishers in the
same order they were published (this applies for messages published on
a single channel). However, publisher acknowledgements are emitted
asynchronously and can confirm a single message or a group of
messages. The exact moment when a confirm is emitted depends on the
delivery mode of a message (persistent vs. transient) and the
properties of the queue(s) the message was routed to (see above).
Which is to say that different messages can be considered ready for
acknowledgement at different times. This means that acknowledgements
can arrive in a different order compared to their respective messages.
Applications should not depend on the order of acknowledgements when
possible.
At first glance, this makes sense: persisting a message takes much more time than just storing it in memory, so it's perfectly possibly that the acknowledgment of a later transient message will arrive before the acknowledgement of an earlier persistent message.
But, if we re-read the first quote regarding message order [QUOTE 1] here above, it gets confusing. I'll explain. Assume we are sending two messages to the same exchange: first a persistent and then a transient message. Since RabbitMQ claims to preserve message order, how can it send an acknowledgment of the second/transient message before it knows that the first/persistent message is indeed completely written to disk?
In other words, does the remark regarding illogical acknowledgement order [QUOTE 2] here above only apply in case the two messages are each routed to completely different target queue(s) (which might happen if they have different routing keys, for example)? In that case, we don't have to guarantee anything as done in [QUOTE 1].
[*] In most cases, this means 'queued'. However, if there are no routing rules applicable, it cannot be enqueued in a target queue. However, this is still a positive outcome regarding publication confirmation.
update
I read this answer on a similar question. This basically says that there are no guarantees whatsoever. Even the most naive implementation, where we delay the publication of message 2 to the point after we got an acknowledgment of message 1, might not result in the desired message order. Basically, [QUOTE 1] is not met.
Is this correct?
From this response on rabbitmq-users:
RabbitMQ knows message position in a queue regardless of whether it is transient or not.
My guess (I did not write that part of the docs) the ack ordering section primarily tries to communicate that if two messages are routed to two different queues, those queues will handle/replicate/persist them concurrently. Reasoning about ordering in more than one queue is pretty hard. A message can go into more than one queue as well.
Nonetheless, RabbitMQ queues know what position a message has in what queues. Once all routing/delivery acknowledgements are received by a channel that handled the publish, it is added to the list of acknowledgements to send out. Note that that
list may or may not be ordered the same way as the original publishes and worrying about that is not practical for many reasons, most importantly: the user typically primarily cares about the ordering in the queues.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
I`ve been reading about the principles of AMQP messaging confirms. (https://www.rabbitmq.com/confirms.html). Really helpful and wel written article but one particular thing about consumer aknowledgments is really confusing, here is the quote:
Another things that's important to consider when using automatic acknowledgement mode is that of consumer overload.
Consumer overload? Message queue is processed and kept in RAM by broker (if I understand it correctly). What overload is it about? Does consumer have some kind of second queue?
Another part of that article is even more confusing:
Consumers therefore can be overwhelmed by the rate of deliveries, potentially accumulating a backlog in memory and running out of heap or getting their process terminated by the OS.
What backlog? How is this all works together? What part of job is done by consumer (besides consuming message and processing it of course)? I thought that broker is keeping queues alive and forwards the messages but now I am reading about some mysterious backlogs and consumer overloads. This is really confusing, can someone explain it a bit or at least point me to the good source?
I believe the documentation you're referring to deals with what, in my opinion, is sort of a design flaw in either AMQP 0-9-1 or RabbitMQ's implementation of it.
Consider the following scenario:
A queue has thousands of messages sitting in it
A single consumer subscribes to the queue with AutoAck=true and no pre-fetch count set
What is going to happen?
RabbitMQ's implementation is to deliver an arbitrary number of messages to a client who has not pre-fetch count. Further, with Auto-Ack, prefetch count is irrelevant, because messages are acknowledged upon delivery to the consumer.
In-memory buffers:
The default client API implementations of the consumer have an in-memory buffer (in .NET it is some type of blocking collection (if I remember correctly). So, before the message is processed, but after the message is received from the broker, it goes into this in-memory holding area. Now, the design flaw is this holding area. A consumer has no choice but to accept the message coming from the broker, as it is published to the client asynchronously. This is a flaw with the AMQP protocol specification (see page 53).
Thus, every message in the queue at that point will be delivered to the consumer immediately and the consumer will be inundated with messages. Assuming each message is small, but takes 5 minutes to process, it is entirely possible that this one consumer will be able to drain the entire queue before any other consumers can attach to it. And since AutoAck is turned on, the broker will forget about these messages immediately after delivery.
Obviously this is not a good scenario if you'd like to get those messages processed, because they've left the relative safety of the broker and are now sitting in RAM at the consuming endpoint. Let's say an exception is encountered that crashes the consuming endpoint - poof, all the messages are gone.
How to work around this?
You must turn Auto-Ack off, and generally it is also a good idea to set reasonable pre-fetch count (usually 2-3 is sufficient).
Being able to signal back pressure a basic problem in distributed systems. Without explicit acknowledgements, the consumer does not have any way to say "Slow down" to broker. With auto-ack on, as soon as the TCP acknowledgement is received by broker, it deletes the message from its memory/disk.
However, it does not mean that the consuming application has processed the message or ave enough memory to store incoming messages. The backlog in the article is simply a data structure used to store unprocessed messages (in the consumer application)
I'm currently using ActiveMQ for my queueing system, and I'm wanting to make the transition to RabbitMQ. One feature I've been using that belongs to ActiveMQ is a redelivery policy, as sometimes our consumer rejects a message because it cannot handle it at this time, but may want to try again later, so it requeues it.
Right now in AMQP, when I reject a message, it's instantly pulled off the queue again immediately and tried again.
Is there a way, in RabbitMQ, to specify a redelivery policy for a queue, consumer, or message?
I also had problems with that behaviour. According to documentation (as far as I remember, maybe in newer version something changed) after requeue it is not stated where a message will be placed (it was described as undetermined). In my testcases (with version 2.8.2) some of messages were put to the end of a queue and one message (precisely first from clients prefetch) land on beggining (and being consumed immediately). In our application this caused livelock.
You could walkaround this by publishing copy of message to a queue and acking already delivered one in one transaction (but I recommend to carefully read section about transactions in docs) or use deadlettering to deal with temporaly unprocessable messages.
In our application the publisher creates a message and sends it to a topic.
It then needs to wait, when all of the topic's subscribers ack the message.
It does not appear, the message bus implementations can do this automatically. So we are leaning towards making each subscriber send their own new message for the client, when they are done.
Now, the client can receive all such messages and, when it got one from each destination, do whatever clean-ups it has to do. But what if the client (sender) crashes part way through the stream of acknowledgments? To handle such a misfortune, I need to (re)implement, what the buses already implement, on the client -- save the incoming acknowledgments until I get enough of them.
I don't believe, our needs are that esoteric -- how would you handle the situation, where the sender (publisher) must wait for confirmations from multiple recipients (subscribers)? Sort of like requesting (and awaiting) Return-Receipts from each subscriber to a mailing list...
We are using RabbitMQ, if it matters. Thanks!
The functionality that you are looking for sounds like a messaging solution that can perform transactions across publishers and subscribers of a message. In The Java world, JMS specifies such transactions. One example of a JMS implementation is HornetQ.
RabbitMQ does not provide such functionality and it does for good reasons. RabbitMQ is built for being extremely robust and to perform like hell at the same time. The transactional behavior that you describe is only achievable with the cost of reasonable performance loss (especially if you want to keep outstanding robustness).
With RabbitMQ, one way to assure that a message was consumed successfully, is indeed to publish an answer message on the consumer side that is then consumed by the original publisher. This can be achieved through RabbitMQ's RPC procedure calls which might help you to get a clean solution for your problem setting.
If the (original) publisher crashes before all answers could be received, you can assume that all outstanding answers are still queued on the broker. So you would have to build your publisher in a way that it is capable to resume with processing those left messages. This might turn out to be none-trivial.
Finally, I recommend the following solution: Design your producing component in a way that you can consume the answers with one or more dedicated answer consumers that are separated from the origin publisher.
Benefits of this solution are:
the origin publisher can finish its task independent of consumer success
the origin publisher is independent of consumer availability and speed
the origin publisher implementation is far less complex
in a crash scenario, the answer consumer can resume with processing answers
Now to a more general point: One of the major benefits of messaging is the decoupling of application components by the broker. In AMQP, this is achieved with exchanges and bindings that allow you to move message distribution logic from your application to a central point of configuration.
If you add RPC-style calls to your clients, then your components are most likely closely coupled again, meaning that the publishing component fails if one of the consuming components fails / is not available / too slow. This is exactly what you will want to avoid. Otherwise, why would you have split the components then?
My recommendation is that you design your application in a way that publishers can complete their tasks independent of the success of consumers wherever possible. Back-channels should be an exceptional case and be implemented in the described not-so coupled way.