As Per RabbitMQ docs, it is defined what happens if there is a conflict between TTL on queue level and on message level.
It is not clear however what will take precedence in case of value conflict between a queue policy, queue declaration and per message TTL?
Related
What is the difference between delivery-limit and x-delivery-limit?
When I set the x-delivery-limit as RabbitMQ queue argument I can see it is limiting my message requeue attempts, as I expected, but in the official RabbitMQ documentation I see the usage of delivery-limit.
Both are valid settings.
The difference is that delivery-limit is a policy vs x-delivery-limit is a queue argument.
The same difference applies to other RabbitMQ settings, for example
dead-letter-exchange is a policy vs x-dead-letter-exchange is a queue argument
queue-length is a policy vs x-queue-length is a queue argument
A queue argument is prefixed by x- and is also referred to as an x-argument. The x stands for "extra" or "extended" because these arguments extend the mandatory queue settings. Mandatory queue settings are for example the durable and exclusive properties. x-arguments are optional queue settings. x-arguments are set by clients when they declare a queue.
That is, to change an x-argument, you would need to re-deploy the client, and re-declare the queue. For an existing queue, changing an x-argument is not allowed and will result in an inequivalent arg error closing the channel.
This is where policies come in handy. They have the following benefits:
Policies are applied dynamically. Once a queue is declared, policies can change queue settings at run time. Note that not all queue settings can be changed by a policy. For example changing the x-queue-type (for example from classic queue to quorum queue) is not allowed since a queue process and how it stores messages cannot just be changed dynamically once it has been created. However, most queue settings (including delivery-limit) can be changed dynamically via a policy.
Policies can be applied to groups of queues (and groups of exchanges). A queue argument can only be applied to a single queue.
In general, it's good practice to use a policy instead of queue argument where possible because policies are more flexible.
More in the official docs: https://www.rabbitmq.com/parameters.html#why-policies-exist
I believe that delivery-limit is just name for header value x-delivery-limit. You can find in RabbitMQ UI for queue arguments.
There is a blog post from RabbitMQ with screenshots (Fig 9. Quorum queue arguments) where they are using x-delivery-limit header which works only with quorum queues (feature matrix)
UPD: in according with this screenshot x-delivery-limit is a part of queue features however delivery-limit is a part of policy definition applied to this queue. Check this article for more details.
The "RabbitMQ in Action" book on page 19 gives these descriptions of exclusive and auto-delete:
auto-delete - The queue is automatically deleted when the last
consumer unsubscribes. If you need a temporary queue used only by one
consumer, combine auto-delete with exclusive. When the consumer
disconnects, the queue will be removed.
Then, in https://www.rabbitmq.com/ttl.html, gives expeire description:
expires policy controls for how long a queue can
be unused before it is automatically deleted. Unused means the queue
has no consumers, the queue has not been recently redeclared
(redeclaring renews the lease),and basic.get has not been invoked for a duration of at least the expiration period
expalation for auto-delete: the queue is deleted when all consumers have finished using it. The last consumer can be cancelled either explicitly or because its channel is closed. If there was no consumer ever on the queue, it won't be deleted. Applications can explicitly delete auto-delete queues using the Delete method.
explanation for expires: Expiry time can be set for a given queue by setting the x-expires argument to queue.declare, or by setting the expires policy. This controls for how long a queue can be unused before it is automatically deleted. Unused means the queue has no consumers, the queue has not been recently redeclared (redeclaring renews the lease), and basic.get has not been invoked for a duration of at least the expiration period. This can be used, for example, for RPC-style reply queues, where many queues can be created which may never be drained.
I think it would be better to use expires instead of auto-delete because if somehting happens and the subscriber went down for a short time then the msg will not be lost that's a big advantage of using expires and that explains simply the difference between the two.
The plain and simple answer here is that exclusive/auto-delete will delete the queue immediately after the first consumer has disconnected, while expires will delete the queue after a period of inactivity regardless of consumer(s) having connected to it in the past.
I am sending a normal message through a producer to RabbitMQ and then I send a second message with the expiration attribute assigned to a value. Then using the rabbitmqctl list_queues command I monitor the status of the messages.
I found that if I send a normal message first and then a message with expiration, the rabbitmqctl list_queues is always showing me 2 messages pending on the queue. When I consume them, I get only one.
On the other hand if I send just 1 message with expiration, in the beginning I see the message and then after the correct expiration time, I find it deleted.
My question is, on the first situation is actually the message taking space? Or it is an interface bug?
My rabbitMQ version is:
rabbitmq-server.noarch -> 3.1.5-1.el6
Looks like you missed some of the documentation on this feature. If you read the RabbitMQ documentation on per-message TTL (expiration), you will notice the following warning for exactly the behavior you are seeing (emphasis added):
Caveats
While consumers never see expired messages, only when expired messages reach the head of a queue will they actually be discarded (or dead-lettered). When setting a per-queue TTL this is not a problem, since expired messages are always at the head of the queue. When setting per-message TTL however, expired messages can queue up behind non-expired ones until the latter are consumed or expired. Hence resources used by such expired messages will not be freed, and they will be counted in queue statistics (e.g. the number of messages in the queue).
We are building a solution in which we are publishing message to a time-out queue. After TTL expiry messages are pushed to main queue for re-processing.
We are setting up counter value so that messages will be tried for x no. of times for the redelivery.
Solution is working fine. But the scenario is when the message on the head position is highest TTL is not expired, other messages of lower expiry will not be re-published (to main queue).
Is this understanding correct ? If Yes what is the solution so that each message re-processed just after TTL.
Appreciating answers / viewpoint.
Thanks.
If you use per-queue message TTL, then message expires and get removed from queue from head to tail (in the same order they was published).
When you use per-message TTL, then messages removed from queue only when they reach queue head, so situation when expired messages still reside in the middle of queue is normal. Such messages will not be send to consumer, and will be deadlettered (or dropped), but due to strict FIFO nature or RabbitMQ's queues that will happen as written above, when they reach queue head and delay before removal may be greater than actual message TTL. For example, if there are two message, first with TTL=10sec and the second one with TTL=1sec, second message will be deadlettered also in 10sec while it stay after first one.
To deal with messages that has different TTL, common workaround is to declare few queues, each for messages with same TTL or almost same, say, with precision 10sec. Actual precision may vary while it very application-specific and somehow empirical value.
If you will pick separate per-TTL queues, use per-queue TTL rather than per-message TTL for ease of messages workflow and to prevent disambiguation of understanding what happens with messages. Developers after you will thank you for that.
To re-process messages after their TTL use Dead Letter Exchanges, but beware of cycled messages problem: if RabbitMQ broker detects that your messages workflow cycled (messages get published to same exchange with the same routing key after it was deadlettered from it), it will silently drop message.
the queue ttl is simple enough and working fine.
but set per message ttl is not working expectly: each message publish to online consumer just after ttl.
why rabbitmq provide this feature? for which biz scenario?
Can you tell me the meaning of this function, what they do? I have tried to find it on Google but i can not be understand.
I am setting the redelivery policy.
policy.setBackOffMultiplier(2);
policy.setUseExponentialBackOff(true);
policy.setMaximumRedeliveries(2);
policy.setRedeliveryDelay(5000)
Messages are redelivered to a client when any of the following occurs:
A transacted session is used and rollback() is called.
A transacted session is closed before commit is called.
A session is using CLIENT_ACKNOWLEDGE and Session.recover() is called.
Once a message's redelivery attempts exceeds the maximumRedeliveries configured for the Redelivery Policy, a "Poison ack" is sent back to the broker letting him know that the message was considered a poison pill. The Broker then takes the message and sends it to a Dead Letter Queue so that it can be analyzed later on.
You can read more about message redelivery handling in Active MQ here:
http://activemq.apache.org/message-redelivery-and-dlq-handling.html
The the redelivery policy properties you are referring to are:
maximumRedeliveries: Sets the maximum number of times a message will be redelivered before it is considered a poisoned pill and returned to the broker so it can go to a Dead Letter Queue (use value -1 to define infinite number of redeliveries)
maximumRedeliveryDelay: Sets the maximum delivery delay that will be applied if the useExponentialBackOff option is set. (use value -1 to define that no maximum be applied) (v5.5)
useExponentialBackOff: Should exponential back-off be used (i.e. to exponentially increase the timeout)
backOffMultiplier: The back-off multiplier
All properties for the redelivery policy are described here:
http://activemq.apache.org/redelivery-policy.html
Few points to add to those mentioned above
If you are using CLIENT_ACK as the acknowledgement mode and closing the session without acknowledging the latest message all the pending messages will be redelivered.
If you are using asynchronous message transfer and a RuntimeException is thrown inside onMessage() method the corresponding message will be redelivered.
All of the above is subjected to maximum redelivery tries of-course. After maximum redelivery tries the message will go to dead message queue(if you have configured) or will be simply dropped.
What you have asked is a type of redelivery policy. So in case a message has to be redelivered how much importance(delay) should be given to it's redelivery. If setUseExponentialBackOff is set for each redelivery of the same message delivery delay will increase exponentially.
All the properties available for redelivery policy are
For complete details look here.