I want to have messages that end up in a Dead Letter Queue automatically deleted after a certain period of time.
My understanding of the documentation is that messages that are in the head of the queue will be automatically deleted when they expire ( TTL).
I have setup a message-ttl policy with value of 10 seconds on a dead letter queue, but messages that end up here aren't deleted. The policy is visible on the queue, so it is applied. There are no consumers added to this queue.
This is the policy:
Why don't the messages get deleted after 10 seconds?
Related
I am facing a scenario where every message in the queue should be consumed after a delay that is achievable using TTL and Dead Letter Exchange in RabbitMQ but when a max TTL will be at the head than no other messages will be consumed.
I want to achieve that every message in the queue gets consumed after it's delay irrespective of its position in the queue.
Instead of relying on TTL and the dead-letter exchange you can use the delayed-message-exchange. With the delayed-message-exchange, the message is published to the queue only when its TTL expires. Consuming every message in the queue will require to add more consumers to the queue.
i have set TTL of persistent messages in a queue as 5 seconds, the messages did expire and landed in the DLQ, however, i notice that the expired messages will only appear in the DLQ onlyafter 10+ to 20+ seconds at random after it was sent even though the TTL is set as 5 seconds. Is there a way to configure such that expired messages are moved to DLQ queue immediately after it expires ?
In the absence of some consumer pulling messages off the Queue and the broker seeing prior to dispatch that the message has expired there is a periodic task that is run to scan for expired messages that are in memory (those paged to disk will be expired when paged back in).
You can configure the scan to run more often but it will have an impact on broker performance. The option is documented in the ActiveMQ Destination Policy options for Queue only values.
I have an a queue that has x-expires set. The issue I am having is that I need to do further processing on the messages that are in the queue IF the queue expires. My initial idea was to set x-dead-letter-exchange on the queue. But, when the queue expires, the messages just vanish without making it to the dead-letter exchange.
How can I dead-letter, or otherwise process, messages that are in a queue that expires?
As suggested in the comments, you cannot do this by relying only on the x-expire feature. But a solution that worked in a similar case I had was to:
Use x-message-ttl to make sure messages die if not consumed in a timely manner,
Assign a dead letter exchange to the queue where all those messages will be routed,
Use x-expires to set the queue expiration to a value higher than the TTL of the messages,
(and this is the tricky part) Assuming you have control over your consumers, before the last consumer goes offline, delete the binding to your "dying" queue, potentially through a REST API call - this will prevent new messages from being routed to the queue.
This way the messages that were published before the last consumer died were already processed, existing messages will be dead-lettered before the queue expires, and new messages cannot come into the queue.
You need to add a new dead letter queue that is bound to your dead letter exchange with the binding routing key set as the original queue name. In this way all expired messages sent to the dead letter exchange are routed to the dead letter queue.
The "RabbitMQ in Action" book on page 19 gives these descriptions of exclusive and auto-delete:
auto-delete - The queue is automatically deleted when the last
consumer unsubscribes. If you need a temporary queue used only by one
consumer, combine auto-delete with exclusive. When the consumer
disconnects, the queue will be removed.
Then, in https://www.rabbitmq.com/ttl.html, gives expeire description:
expires policy controls for how long a queue can
be unused before it is automatically deleted. Unused means the queue
has no consumers, the queue has not been recently redeclared
(redeclaring renews the lease),and basic.get has not been invoked for a duration of at least the expiration period
expalation for auto-delete: the queue is deleted when all consumers have finished using it. The last consumer can be cancelled either explicitly or because its channel is closed. If there was no consumer ever on the queue, it won't be deleted. Applications can explicitly delete auto-delete queues using the Delete method.
explanation for expires: Expiry time can be set for a given queue by setting the x-expires argument to queue.declare, or by setting the expires policy. This controls for how long a queue can be unused before it is automatically deleted. Unused means the queue has no consumers, the queue has not been recently redeclared (redeclaring renews the lease), and basic.get has not been invoked for a duration of at least the expiration period. This can be used, for example, for RPC-style reply queues, where many queues can be created which may never be drained.
I think it would be better to use expires instead of auto-delete because if somehting happens and the subscriber went down for a short time then the msg will not be lost that's a big advantage of using expires and that explains simply the difference between the two.
The plain and simple answer here is that exclusive/auto-delete will delete the queue immediately after the first consumer has disconnected, while expires will delete the queue after a period of inactivity regardless of consumer(s) having connected to it in the past.
I have 2 RabbitMQ queues:
incoming_message => where I push all messages that I want to process later
incoming_message_dlx => where I push the message whose the processing failed
As you can supposed with its name, the incoming_message_dlx queue use the Dead Letter Exchange feature, that means when the message expires, it will be requeue to my incoming_message.
What I try to achieve is to increase the expiration of messages each time the processing failed and that they are push to the DLX queue.
The problem is that even if a message expired, it will not be requeue to my incoming_message while it's not at the bottom (head) of the queue. So if there is a message with an expiration time of 7 days in the DLX queue and that we enqueue a new message with the expiration time of 5 seconds, this message will only be requeue to the incoming_message after 7 days + 5 seconds...
I've found on the documentation that I can use my DLX queue as a priority queue and put a priority on my messages according to the expiration time, but it doesn't work as expected, the priority seems to be ignored.
However, when I use the RabbitMQ admin (management plugin) and that I get the first message of the queue, it's always the one with the higher priority, but the "internal consumer" of the DLX queue seems to ignore this priority.
Do you know what could be the problem?
Thanks a lot in advance.
PS: I'm using RabbitMQ server version 3.6.10.
as a queue structure(fifo),rabbitmq do expire from the head of the queue.
queue ttl contains 3 type:
Per-Queue Message TTL： x-message-ttl
Per-Message TTL： expiration
Queue TTL：x-expires
when you want the message just deliver on the ttl value ,try use multi level ttl queue.
you can predefined some dlx queue as you need.
eg: you want error message do retry in (5s,15s,60s), you can define 3 dlx queue by set different x-message-ttl value, and this 3 incoming_message_dlx queue binding the dlx router to the incoming_message;
but if you have a message ttl=30s ,you just prefdefind 3 queue with ttl(5s,15s,60s) , so where to diliver ? try priority queue.
offical doc
Messages which should expire will still only expire from the head of the queue. This means that unlike with normal queues,
even per-queue TTL can lead to expired lower-priority messages getting stuck behind
non-expired higher priority ones.
These messages will never be delivered, but they will appear in queue statistics.
expired lower-priority messages getting stuck behind non-expired higher priority ones
queue like [60s(p=1),30s(p=0)] will not happen!
we defined 3 queue ttl(5s,15s,60s),in order to prevent lower ttl message getting stucked , we push the message to the queue with flor ttl not ceil ttl；
so ttl=30s will deliver to queue which ttl=60s,and set priority=1
ttl=30s is between the predefined queue (15s,60s);
set ttl=60s queue's max-priority=1, default is 0;
deliver ttl=30s message with priority=1;
so the message in a queue just like [30,60,60,60,60].
ttl=30s will not be blocked by the ttl=60s.