It is possible to publish messages into a RabbitMQ queue with an expiration TTL: such messages will expire once the TTL is done and (if a dead-letter queue is setup,) removed to the dead-letter queue.
But is it possible to specify such per-message TTL using Celery?
Note that I'm not looking for a way to specify task-expiration but rather message expiration: I want my messages to spend (a configurable) amount of time in the queue before finally getting picked up # the dead-letter queue.
TIA.
Short introduction: Expiration vs Expires
RabbitMQ does support per-message TTL (as well as TTL for the queue), the behavior is documented here: https://www.rabbitmq.com/ttl.html#per-message-ttl-in-publishers. The trick is to set the expiration Message Property (https://www.rabbitmq.com/publishers.html#message-properties) when the message is published (in milliseconds).
Celery on the other hand allows you to set the expires parameter (https://docs.celeryproject.org/en/stable/reference/celery.app.task.html) in seconds or as a datetime. The difference from the native RabbitMQ functionality is that the message remains in the queue after expiration. The expired message is delivered to the worker, which then reads the expires header to determine that the message has expired and rejects the message.
tl;dr: expiration != expires
How to pass a message property in Celery
This method is not documented in Celery. I figured it out by trial and error because I wanted a native TTL myself.
The send_task method (celery.app.base.Celery.send_task), which is called for example by apply_async, accepts the **options parameter. All **options unknown to Celery are then passed in the celery.app.amqp.Queues->send_task_message( ... ) method as **kwargs and then as message properties.
So if we can set the message property, there is nothing easier than setting the native expiration:
my_awesome_task.apply_async(args=(11,), expiration=42)
Note that Celery automatically converts 42 seconds to 42000 milliseconds (which is correct).
Expiration (in properties) and Expires (in headers) can be combined, the two functionalities are not affected in any way.
Related
Is there a way how to get notification from RabbitMQ if a message is waiting in queue for a long time?
A common and standard solution to ensure that a delivery latency requirement is met is to set a Time To Live (TTL) on the message, and if this expires, this will be moved to the Dead Letter Queue where you can perform compensating actions.
TTL can be defined
On a per-message basis by the message publisher
As a default on the queue itself
If both the queue and the message have the TTL set, then the lower latency will be policed.
The "RabbitMQ in Action" book on page 19 gives these descriptions of exclusive and auto-delete:
auto-delete - The queue is automatically deleted when the last
consumer unsubscribes. If you need a temporary queue used only by one
consumer, combine auto-delete with exclusive. When the consumer
disconnects, the queue will be removed.
Then, in https://www.rabbitmq.com/ttl.html, gives expeire description:
expires policy controls for how long a queue can
be unused before it is automatically deleted. Unused means the queue
has no consumers, the queue has not been recently redeclared
(redeclaring renews the lease),and basic.get has not been invoked for a duration of at least the expiration period
expalation for auto-delete: the queue is deleted when all consumers have finished using it. The last consumer can be cancelled either explicitly or because its channel is closed. If there was no consumer ever on the queue, it won't be deleted. Applications can explicitly delete auto-delete queues using the Delete method.
explanation for expires: Expiry time can be set for a given queue by setting the x-expires argument to queue.declare, or by setting the expires policy. This controls for how long a queue can be unused before it is automatically deleted. Unused means the queue has no consumers, the queue has not been recently redeclared (redeclaring renews the lease), and basic.get has not been invoked for a duration of at least the expiration period. This can be used, for example, for RPC-style reply queues, where many queues can be created which may never be drained.
I think it would be better to use expires instead of auto-delete because if somehting happens and the subscriber went down for a short time then the msg will not be lost that's a big advantage of using expires and that explains simply the difference between the two.
The plain and simple answer here is that exclusive/auto-delete will delete the queue immediately after the first consumer has disconnected, while expires will delete the queue after a period of inactivity regardless of consumer(s) having connected to it in the past.
I have planned to delay the processing of messages in queue by following these two links link1 link2. So, as suggested in the link. I have declared the original queue with the x-dead-letter-exchange and x-dead-letter-routing-key args. Which published the messages to the so called dead-letter-queue when message either failed to get processed by consumer or ttl happen or queue length exceed. Now in the dead-letter-queue similar args have been set along with the ttl parameter. Which is suppose to republish the messages to the original queue after ttl exceed. But the problem is it is dropping all the messages.
Moreover, there is a catch here. If i explicitly publish the failed messages from original queue to dead-letter-queue. Then after ttl it republish the messages to the original queue. Why is it so and how do i make it work. So that dead-letter-queue republishes the messages to the original queue instead of dropping. I am using RabbitMQ 3.0.0.
FYI, I have created both the exchanges of direct type along with the routing key
When a queue has a TTL setup that means that messages in that queue will be sent to the dead-letter-exchange (DLX) associated with that queue after the TTL has expired. If the queue has no DLX assigned then the messages go into the bit bucket.
If you want to send messages back into the queue from which they came to be re-processed then you need to have the setup that I described in this post.
Dead-lettering dead-lettered messages in RabbitMQ
Hopefully that is helpful for you.
Suppose your original exchange is x.notification and is bind to the queue q.A with routing queue A. And your dead-letter-exchange namae is dlx.notification. Now in the queue q.A set ttl the time interval you want to wait and dead-lleter-exchange as dlx.notification. Now create another queue dlq.A to route the expired message from dlx.notification into dlq.A with routing key "A". I think thats all you need to do to achive your goal.
I am sending a normal message through a producer to RabbitMQ and then I send a second message with the expiration attribute assigned to a value. Then using the rabbitmqctl list_queues command I monitor the status of the messages.
I found that if I send a normal message first and then a message with expiration, the rabbitmqctl list_queues is always showing me 2 messages pending on the queue. When I consume them, I get only one.
On the other hand if I send just 1 message with expiration, in the beginning I see the message and then after the correct expiration time, I find it deleted.
My question is, on the first situation is actually the message taking space? Or it is an interface bug?
My rabbitMQ version is:
rabbitmq-server.noarch -> 3.1.5-1.el6
Looks like you missed some of the documentation on this feature. If you read the RabbitMQ documentation on per-message TTL (expiration), you will notice the following warning for exactly the behavior you are seeing (emphasis added):
Caveats
While consumers never see expired messages, only when expired messages reach the head of a queue will they actually be discarded (or dead-lettered). When setting a per-queue TTL this is not a problem, since expired messages are always at the head of the queue. When setting per-message TTL however, expired messages can queue up behind non-expired ones until the latter are consumed or expired. Hence resources used by such expired messages will not be freed, and they will be counted in queue statistics (e.g. the number of messages in the queue).
We are building a solution in which we are publishing message to a time-out queue. After TTL expiry messages are pushed to main queue for re-processing.
We are setting up counter value so that messages will be tried for x no. of times for the redelivery.
Solution is working fine. But the scenario is when the message on the head position is highest TTL is not expired, other messages of lower expiry will not be re-published (to main queue).
Is this understanding correct ? If Yes what is the solution so that each message re-processed just after TTL.
Appreciating answers / viewpoint.
Thanks.
If you use per-queue message TTL, then message expires and get removed from queue from head to tail (in the same order they was published).
When you use per-message TTL, then messages removed from queue only when they reach queue head, so situation when expired messages still reside in the middle of queue is normal. Such messages will not be send to consumer, and will be deadlettered (or dropped), but due to strict FIFO nature or RabbitMQ's queues that will happen as written above, when they reach queue head and delay before removal may be greater than actual message TTL. For example, if there are two message, first with TTL=10sec and the second one with TTL=1sec, second message will be deadlettered also in 10sec while it stay after first one.
To deal with messages that has different TTL, common workaround is to declare few queues, each for messages with same TTL or almost same, say, with precision 10sec. Actual precision may vary while it very application-specific and somehow empirical value.
If you will pick separate per-TTL queues, use per-queue TTL rather than per-message TTL for ease of messages workflow and to prevent disambiguation of understanding what happens with messages. Developers after you will thank you for that.
To re-process messages after their TTL use Dead Letter Exchanges, but beware of cycled messages problem: if RabbitMQ broker detects that your messages workflow cycled (messages get published to same exchange with the same routing key after it was deadlettered from it), it will silently drop message.
the queue ttl is simple enough and working fine.
but set per message ttl is not working expectly: each message publish to online consumer just after ttl.
why rabbitmq provide this feature? for which biz scenario?