I am new to RabbitMQ and I have a question on get messages option in management plugin. It says "Clicking 'Get Message(s)' will consume messages from the queue. If requeue is set, the message will be re-added to the queue, but ordering will not be preserved and 'redelivered' will be set."
This article says
In the next version of rabbit (after v2.6.1) the order will not be
destroyed when requeueing messages. Messages will be requeued to the
front of the queue, meaning that you can "peek" at the head of the
queue. You can make use of the management interface or use an AMQP
client to do this.
So why management plugin says the ordering will not be preserved? I am just trying to understand which is one correct?
Thanks
I got a reply from RabbitMQ team. Here is what they say...
On 25/04/13 01:05, Prazzy wrote:
So why management plugin says the ordering will not be preserved? I am
just trying to understand which is one correct?
The management plugin did not get updated it seems. I'll fix that.
Thanks for pointing it out.
Cheers, Simon
--
Simon MacMullen
RabbitMQ, VMware
The management plugin is wrong.
As the response from the RabbitMQ team said, the answer to your question is that the management plug-in is mistaken. The behavior was changed in 2.6.1 to re-queue messages to as close to the original order as possible - which in this case would almost certainly be at the head of the queue.
Beware of side-effects.
As this answer points out, however, "peeking" is a rather mild-sounding term for what is actually happening: a message is (however briefly) being removed from the queue, then put back. During this brief span of time, it is possible that another message will be delivered to a consumer, in which case your "peeked" message will still go back to the head, but messages may be processed out of order.
Related
I'm going through a few examples using NServiceBus and I've stumbled across a feature I'm hoping ships with MassTransit (As it is a free service).
The feature is based around 'poisoned' messages.
If, due to a bug in your system, these messages cant ever be handled, and end up permanently in the error queue.
NServiceBus has a cool feature whereby, once you have corrected the bugs in your code, allows those messages in the error queue to be 'redirected' to the original working queue, to be redelivered.
This is done by using a NServiceBus specific tool :- ReturnToSourceQueue.exe.
Does MassTransit have a similar tool for this kind of issue?
Or is there another workaround availble, preferbly to work with RabbitMQ.
With RabbitMQ, it's easy to move messages between queues. You can use the management console to do it manually, by installing the shovel plug-in.
You can also create shovels in RabbitMQ that are scheduled, and perform the message movement in response to that schedule. The visibility of having the shovels configured in RabbitMQ has been invaluable to our operations staff, since they rarely think that a Windows Scheduled Task (or other random scheduler) is going to be doing something as risky as moving previously failed messages back into the production queues.
I would suggest reading this blog post on how MassTransit deals with poison messages: Error Handling in MassTransit with RabbitMQ
The tooling around RabbitMQ is so much better than anything MSMQ provides, which is one of the reasons we have completely abandoned MSMQ for production queuing.
This functionality is easily recreated with nothing more than RabbitMQ and a bit of code. While it's nice that NServicebus includes it, building it with MassTransit should be easy enough.
(note: i haven't used .NET in a few years, so my knowledge of NSB and MT are a bit rusty... this will be high level answer only, no code)
The thing to start with, is a proper configuration of a dead letter exchange and a poison message queue. https://www.rabbitmq.com/dlx.html
Once you have knowledge that a message is causing errors and is a bad message, you can reject or nack (with no requeue) the message in order to send it through the dead letter exchange (DLX).
Once a message has gone through the DLX, you will have some additional properties on the message, including:
queue - the name of the queue the message was in before it was dead-lettered,
exchange - the exchange the message was published to (note that this will be a dead letter exchange if the message is dead lettered multiple times),
routing-keys - the routing keys (including CC keys but excluding BCC ones) the message was published with,
there will be more, but these are the things you want to pay attention to. by examining these properties on the message, you can re-send the original message back through the original exchange, with the original routing-keys. alternatively, you can re-send straight to the original destination queue... i think sending through the exchange would be better, personally, as the original queue might not exist anymore (depending on system configuration, consumers creating exclusive queues, etc).
with this information, recreating the feature set should not be too difficult. rabbitmq provides all of the features that you need, you just have to write a bit of code to take advantage of it.
I'm trying to implement kind of "Exclusive Consumer" or "Exclusive Queues" which avoids the end user having to pin anything: the broker will pick a single Message-Consumer to get all the messages for a queue to ensure ordering, and if that consumer fails, the broker will auto failover and choose another consumer.
I wondered if a combination of Celery and RabbitMQ can make it.
Despite realising that this question is more than 7 years old, someone may stumble across this page as I just did. So, in 2019, RabbitMQ team has released a version 3.8 which supports new feature called "Single Active Consumer". It offers exactly that type of consumer-exclusivity combined with automatic fallback behavior, where the next (randomly chosen) consumer listening to the queue will be made the "single active consumer" upon the first consumer falling off or disconnecting for any reason. And this feature does not require any complicated setup of the message queue itself; it works with almost any queue.
Read more at https://www.cloudamqp.com/blog/2019-04-23-rabbitmq-3-8-feature-focus-single-active-consumer.html and https://www.rabbitmq.com/consumers.html#single-active-consumer
Though RabbitMQ has the concept of "exclusive" queues, I do not think you can get the fallback behavior that you need.
The broker will not "choose" a single consumer. Failure modes can be addressed by not using auto-ACK, rejecting on error and letting your worker die/restart.
This preserves message order if there is only one consumer. With load-balancing your mileage might vary.
I'm using ActiveMQ and I would like to know how to solve this specific case.
When the consumer is down, the producer sends a message to the queue. The message will remain in the queue until the consumer is running to consume it.
Now imagine I shutdown the producer, the message will STILL remain in the queue. Now i run the consumer and it will try to consume that message, but won't be able to reply back to the producer since its down.
I would like to solve this problem by cleaning the messages if the producer is out.
The ActiveMQ Broker cleans the Queue after stopping. I would like to do the same for the messages of a respective producer.
Thanks.
Based on what I understand now from your question and additional comments I propose to add a Message Property to your messages to identify the Producer, and write a small utility that uses a Message Selector to read all messages matching the Producer from the queue. You can run that utility straight after the Producer is stopped (or crashes), and that should quite accurately do what you want to achieve.
EDIT: although primarily focused on EE, the Sun/Oracle JavaEE Tutorial contains a very good chapter on general JMS programming that starts off with standalone producers and consumers. The accompanying source code bundle can be downloaded here, the ready to comoile samples in that bundle should get you started very quickly.
You can solve it a couple of ways. One is to set a TTL on the message so it goes away. The other is to connect via JMX and purge the Queue or remove the specific message using a selector statement or with the Message's specific MessageId value.
See this article for some hints.
I'm currently using ActiveMQ for my queueing system, and I'm wanting to make the transition to RabbitMQ. One feature I've been using that belongs to ActiveMQ is a redelivery policy, as sometimes our consumer rejects a message because it cannot handle it at this time, but may want to try again later, so it requeues it.
Right now in AMQP, when I reject a message, it's instantly pulled off the queue again immediately and tried again.
Is there a way, in RabbitMQ, to specify a redelivery policy for a queue, consumer, or message?
I also had problems with that behaviour. According to documentation (as far as I remember, maybe in newer version something changed) after requeue it is not stated where a message will be placed (it was described as undetermined). In my testcases (with version 2.8.2) some of messages were put to the end of a queue and one message (precisely first from clients prefetch) land on beggining (and being consumed immediately). In our application this caused livelock.
You could walkaround this by publishing copy of message to a queue and acking already delivered one in one transaction (but I recommend to carefully read section about transactions in docs) or use deadlettering to deal with temporaly unprocessable messages.
Well I've been doing this NServiceBus project for a while and once I got it working for PubSub I then spent the rest of the time on the actual workflow logic. However, I can see a serious issue which I want to get around (or rather learn how to handle correctly).
A publisher publishes a message to the storage queues of any subscribers as far as I understand. Great. But what happens when the subscriber isn't running (I've read other posts about this and they don't seem to be asking the same question).
Scenario - I get the publisher to Publish a message when no subscribers are running (attached/requested messages to be relayed to them).. I then find that.. the message is "gone" just simply isn't there! where did it go? Did the publisher say "hey, no one's subscribing to this, so I wont bother publishing it?", shouldn't it NOT do that and require at least one subscriber?
Can anyone shed any light on this? (nservicenewbie)
You should publish an event that has happened - a statement of fact, that other handler may or may not be interested in. It's perfectly valid to have zero subscribers! If this is not the case then maybe you should be Send()ing a command instead of Publish()ing an event.
If you are using a persistent subscription storage, start the subscriber up once and it will always be subscribed. If the subscriber is offline, messages for it will pile up in its Input Queue, ready to be processed when the subscriber comes back online.
If you're just testing with NServiceBus, the NServiceBus.Host.exe is running in the Lite profile, which uses in-memory (non-persistant) subscription storage, which would result in what you are seeing.
Ah ha! Well though it's not always an error to have no subscriber for a message type, there is a way to handle it.
In your publisher simply modify the:
IBus Bus
To use (you will need NServiceBus.Core.dll and the NS NServiceBus.Unicast):
IUnicastBus Bus
Then you can attach an handler to:
Bus.NoSubscribersForMessage += .......
This can then put the message in an error queue.. or perhaps retry forever.. or publish something else etc.. etc.. what ever you want. Thus ensuring there is nothing lost where your particular system (from a business perspective) requires an outcome