This morning I found that my rabbitmq instances does not have several queues that are usually there.
What I noticed is a pattern, that the remaining queues are the ones that had consumers attached to those.
The queues that are gone are mostly retry and DLQs. How does one investigate what happened? What do I look for and where?
Update:
This is my queue details :
The expires is your problem there, if the queue has not been used for some time, it will get dropped.
Also I would get rid of the message-ttl unless you want your messages to be dropped after certain amount of time.
More info here: http://www.rabbitmq.com/ttl.html
These are rabbitmq configuration settings you must change, here some more info https://www.rabbitmq.com/parameters.html
This is due to a feature of TTL or Time to Live for a queue .
As shown in this example below
The value against expires correspond to 28 days. So If a queue hasn't had any consumer for 28 days it gets deleted.
It could also be possible that the auto-delete property is set to true in which case the queue will automatically get deleted the moment the last consumer gets disconnected.
You should be able to get both these parameters in the rabbitmq console where you check the queue properties .
Related
We have a queue which takes incoming data and pushes the data into another queue in another machine through a shovel.
For some reason, we found that the source queue had backed up with around 2M messages. We couldn't figure out the cause for that as it seemed that the destination queue and the consumer of that queue was working fine.
We also realized that the shovel was setup with the default pre-fetch count of 1000.
We are not able to modify the shovel to set a higher pre-fetch count, the only option is to delete the shovel and setup a new one with higher pre-fetch count.
What will happen if we delete the shovel ?
Will it delete the messages in the queues ?
Thanks
Based on your latest comment, if I understand correctly, you have:
RabbitMQ1 - exchange RMQ1EXCA -> queueA1
RabbitMQ2 - exchange RMQ2EXCB -> queueB1
A shovel has been configured, exchange [RMQ1EXCA] to exchange [RMQ2EXCB]
And you found out that the queueA1 is filled up with millions of message.
If indeed this is an accurate depiction of your setup:
it's quite normal, as queueA1 is not part of the shovel process
if you check the queues bound to RMQ1EXCA, you should see one queue with a name starting with amq.gen-......
deleting the shovel will not impact the queueA1, as it's not related to the process (but will delete the queue amq.gen-...... that is)
If the description provided doesn't match your setup, please provide additional information to clarify your situation so that I can adapt my answer accordingly
My problem:
I pull off a message from a RabbitMQ-Queue. I try to process this message and realize that it can't be processed yet. So i would like to add it back to the queue and let it return only on a specific time + 5000ms. Unfortunately that is more challenging than i thought.
What i've tried:
RabbitMQ Dead Letter Attributes -> My issue here is, even though the manual says that the default exchange is binded to every queue it doesnt forward it according to the routing criteria. I've tried to add expires = "5000" and x-dead-letter-routing-key = "queuename" also "x-dead-letter-exchange = "" as the default exchange should work. The only part which works is the expires part. The message will disappear and go into the dark. This also occurs with the dead-letter-exchange beeing amq.direct including the binding on the targeted queue.
Open gaps for me:
Where i'm a bit left in the dark is if the receivers have to be dead letter queues and if i the dead letter queue is a basic queue with extended functionality. It is also not clear if those parameters (x-dead-letter..) are only for DLX Queues. I would like to do this delayed delivery persistent and purely via. the message attributes and not via. queue configurations (only if required).
I've searched on the web and checked many different dead-letter infos. Im trying to build a micro-service like architecture while using RabbitMQ as the delivery mechanism (i use processes which take their work from the queue and forward it). I would believe other people here have the same running already but i couldn't find any blogs about this.
I had to come to the conclusion that on the message level it is not possible.
I've created now for each queue which is in use a separate queue ("name.delayed") , where i can add the message with the argument "expiration" = 5000
The queue settings itself has to be a dead letter queue routing it to the queue "name"
In the console pane rabbitmq one day I had accumulated 8000 posts, but I am embarrassed that their status is idle at the counter ready and total equal to 1. What status should be completed at the job, idle? In what format is registered x-pires? It seems to me that I had something wrong =(
While it's difficult to fully understand what you are asking, it seems that you simply don't have anything pulling messages off of the queue in question.
In general, RabbitMQ will hold on to a message in a queue until a listener pulls it off and successfully ACKs, indicating that the message was successfully processed. You can configure queues to behave differently by setting a Time-To-Live (TTL) on messages or having different queue durabilities (eg. destroyed when there are no more listeners), but the default is to play it safe.
I am using RabbitMQ server.
For publishing messages, I set the immediate field to true and tried sending 50,000 messages. Using rabbitmqctl list_queues, I saw that the number of messages in the queue was zero.
Then, I changed the immediate flag to false and again tried sending 50,000 messages. Using rabbitmqctl list_queues, I saw that a total of 100,000 messages were in queues (till now, no consumer was present).
After that, I started a consumer and it consumed all the 100,000 messages.
Can anybody please help me in understanding about the immediate bit field and this behavior too? Also, I could not understand the concept of the mandatory bit field.
The immediate and mandatory fields are part of the AMQP specification, and are also covered in the RabbitMQ FAQ to clarify how its implementers interpreted their meaning:
Mandatory
This flag tells the server how to
react if a message cannot be routed to
a queue. Specifically, if mandatory is
set and after running the bindings the
message was placed on zero queues then
the message is returned to the sender
(with a basic.return). If mandatory
had not been set under the same
circumstances the server would
silently drop the message.
Or in my words, "Put this message on at least one queue. If you can't, send it back to me."
Immediate
For a message published with immediate
set, if a matching queue has ready
consumers then one of them will have
the message routed to it. If the lucky
consumer crashes before ack'ing
receipt the message will be requeued
and/or delivered to other consumers on
that queue (if there's no crash the
messaged is ack'ed and it's all done
as per normal). If, however, a
matching queue has zero ready
consumers the message will not be
enqueued for subsequent redelivery on
from that queue. Only if all of the
matching queues have no ready
consumers that the message is returned
to the sender (via basic.return).
Or in my words, "If there is at least one consumer connected to my queue that can take delivery of a message right this moment, deliver this message to them immediately. If there are no consumers connected then there's no point in having my message consumed later and they'll never see it. They snooze, they lose."
http://www.rabbitmq.com/blog/2012/11/19/breaking-things-with-rabbitmq-3-0/
Removal of "immediate" flag
What changed? We removed support for the
rarely-used "immediate" flag on AMQP's basic.publish.
Why on earth did you do that? Support for "immediate" made many parts
of the codebase more complex, particularly around mirrored queues. It
also stood in the way of our being able to deliver substantial
performance improvements in mirrored queues.
What do I need to do? If you just want to be able to publish messages
that will be dropped if they are not consumed immediately, you can
publish to a queue with a TTL of 0.
If you also need your publisher to be able to determine that this has
happened, you can also use the DLX feature to route such messages to
another queue, from which the publisher can consume them.
Just copied the announcement here for a quick reference.
I am sending messages to an Active MQ queue through Mule.
I want that, that only the latest message remains in the queue and not any previous one.
How can this be achieved?
Thank you very much.
set the queue size to be 1, the default eviction policy, which is oldestMessageEvictionStrategy, should kick out the previous message and put in the newest one. You can also set other policies for priorities, such as OldestMessageWithLowestPriorityEvictionStrategy or mess around with the TTL settings on your messages.
You may also get some warnings about full queue though, so be prepared to handle that.