The "RabbitMQ in Action" book on page 19 gives these descriptions of exclusive and auto-delete:
auto-delete - The queue is automatically deleted when the last
consumer unsubscribes. If you need a temporary queue used only by one
consumer, combine auto-delete with exclusive. When the consumer
disconnects, the queue will be removed.
Then, in https://www.rabbitmq.com/ttl.html, gives expeire description:
expires policy controls for how long a queue can
be unused before it is automatically deleted. Unused means the queue
has no consumers, the queue has not been recently redeclared
(redeclaring renews the lease),and basic.get has not been invoked for a duration of at least the expiration period
expalation for auto-delete: the queue is deleted when all consumers have finished using it. The last consumer can be cancelled either explicitly or because its channel is closed. If there was no consumer ever on the queue, it won't be deleted. Applications can explicitly delete auto-delete queues using the Delete method.
explanation for expires: Expiry time can be set for a given queue by setting the x-expires argument to queue.declare, or by setting the expires policy. This controls for how long a queue can be unused before it is automatically deleted. Unused means the queue has no consumers, the queue has not been recently redeclared (redeclaring renews the lease), and basic.get has not been invoked for a duration of at least the expiration period. This can be used, for example, for RPC-style reply queues, where many queues can be created which may never be drained.
I think it would be better to use expires instead of auto-delete because if somehting happens and the subscriber went down for a short time then the msg will not be lost that's a big advantage of using expires and that explains simply the difference between the two.
The plain and simple answer here is that exclusive/auto-delete will delete the queue immediately after the first consumer has disconnected, while expires will delete the queue after a period of inactivity regardless of consumer(s) having connected to it in the past.
Related
We are working in a microservice architecture and we are using RabbitMQ as a message broker. We want to avoid the scenarios where the following happens:
An entity begins its creation but it takes a while for it to finish.
The system decides that the creation time has taken too long and that the entity should be deleted due to a timeout, so it sends out a message to delete the entity which is currently still being created
Delete message gets consumed and the system checks whether the entity exists and does not find it due to the entity still being in the process of being created.
Delete entity message consumer returns an error due to not finding the entity.
How can we ensure that the delete message is consumed after the create message is finished in such a way that we do not block the consumption of other messages?
How can we ensure that the delete message is consumed after the create message is finished in such a way that we do not block the consumption of other messages?
Let's say your entity creation timeout is N. The worker(s) responsible for creating entities should know about this timeout, and should be able to cancel entity creation should N be reached. This isn't strictly necessary but it sounds like your entity creation may be resource intensive so cancellation should be a feature you have.
If your workers know to cancel entity creation when timeout N is reached, then perhaps you don't even need the deletion message?
If you keep the delete message, the workers processing that could do the following:
First, ensure your queue has a Dead Letter Exchange configured
Consume the message, and try to delete the entity
If deletion succeeds, great, ack the message with RabbitMQ and you're done
If deletion fails, nack (reject) the message with RabbitMQ and do set requeue to be false. This will cause the message to be routed to the dead-letter exchange
A worker should consume from a queue bound to this dead-letter exchange. You could have a queue dedicated to re-trying entity deletions. When a worker consumes a message from this queue, it can re-try the deletion. If it fails, you can reject it again (after a delay, of course) and, if this queue has the same dead-letter settings, the same process will happen
Finally, ensure that your deletion workers respect the count property and only try a certain number of times to delete an entity. If a limit is exceeded, this should create an exception in your system
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
This morning I found that my rabbitmq instances does not have several queues that are usually there.
What I noticed is a pattern, that the remaining queues are the ones that had consumers attached to those.
The queues that are gone are mostly retry and DLQs. How does one investigate what happened? What do I look for and where?
Update:
This is my queue details :
The expires is your problem there, if the queue has not been used for some time, it will get dropped.
Also I would get rid of the message-ttl unless you want your messages to be dropped after certain amount of time.
More info here: http://www.rabbitmq.com/ttl.html
These are rabbitmq configuration settings you must change, here some more info https://www.rabbitmq.com/parameters.html
This is due to a feature of TTL or Time to Live for a queue .
As shown in this example below
The value against expires correspond to 28 days. So If a queue hasn't had any consumer for 28 days it gets deleted.
It could also be possible that the auto-delete property is set to true in which case the queue will automatically get deleted the moment the last consumer gets disconnected.
You should be able to get both these parameters in the rabbitmq console where you check the queue properties .
I have an a queue that has x-expires set. The issue I am having is that I need to do further processing on the messages that are in the queue IF the queue expires. My initial idea was to set x-dead-letter-exchange on the queue. But, when the queue expires, the messages just vanish without making it to the dead-letter exchange.
How can I dead-letter, or otherwise process, messages that are in a queue that expires?
As suggested in the comments, you cannot do this by relying only on the x-expire feature. But a solution that worked in a similar case I had was to:
Use x-message-ttl to make sure messages die if not consumed in a timely manner,
Assign a dead letter exchange to the queue where all those messages will be routed,
Use x-expires to set the queue expiration to a value higher than the TTL of the messages,
(and this is the tricky part) Assuming you have control over your consumers, before the last consumer goes offline, delete the binding to your "dying" queue, potentially through a REST API call - this will prevent new messages from being routed to the queue.
This way the messages that were published before the last consumer died were already processed, existing messages will be dead-lettered before the queue expires, and new messages cannot come into the queue.
You need to add a new dead letter queue that is bound to your dead letter exchange with the binding routing key set as the original queue name. In this way all expired messages sent to the dead letter exchange are routed to the dead letter queue.
We are building a solution in which we are publishing message to a time-out queue. After TTL expiry messages are pushed to main queue for re-processing.
We are setting up counter value so that messages will be tried for x no. of times for the redelivery.
Solution is working fine. But the scenario is when the message on the head position is highest TTL is not expired, other messages of lower expiry will not be re-published (to main queue).
Is this understanding correct ? If Yes what is the solution so that each message re-processed just after TTL.
Appreciating answers / viewpoint.
Thanks.
If you use per-queue message TTL, then message expires and get removed from queue from head to tail (in the same order they was published).
When you use per-message TTL, then messages removed from queue only when they reach queue head, so situation when expired messages still reside in the middle of queue is normal. Such messages will not be send to consumer, and will be deadlettered (or dropped), but due to strict FIFO nature or RabbitMQ's queues that will happen as written above, when they reach queue head and delay before removal may be greater than actual message TTL. For example, if there are two message, first with TTL=10sec and the second one with TTL=1sec, second message will be deadlettered also in 10sec while it stay after first one.
To deal with messages that has different TTL, common workaround is to declare few queues, each for messages with same TTL or almost same, say, with precision 10sec. Actual precision may vary while it very application-specific and somehow empirical value.
If you will pick separate per-TTL queues, use per-queue TTL rather than per-message TTL for ease of messages workflow and to prevent disambiguation of understanding what happens with messages. Developers after you will thank you for that.
To re-process messages after their TTL use Dead Letter Exchanges, but beware of cycled messages problem: if RabbitMQ broker detects that your messages workflow cycled (messages get published to same exchange with the same routing key after it was deadlettered from it), it will silently drop message.
the queue ttl is simple enough and working fine.
but set per message ttl is not working expectly: each message publish to online consumer just after ttl.
why rabbitmq provide this feature? for which biz scenario?
The "RabbitMQ in Action" book on page 19 gives these descriptions of exclusive and auto-delete:
exclusive - When set to true, your queue becomes private and can only
be consumed by your app. This is useful when you need to limit a queue
to only one consumer.
auto-delete - The queue is automatically deleted when the last
consumer unsubscribes. If you need a temporary queue used only by one
consumer, combine auto-delete with exclusive. When the consumer
disconnects, the queue will be removed.
But as far as I can see when using exclusive, auto-delete is redundant. Only exclusive is needed. The RabbitMQ tutorial seems to say that is the case
...once we disconnect the consumer the queue should be deleted. There's
an exclusive flag for that:
result = channel.queue_declare(exclusive=True)
There is no mention in that tutorial about auto-delete and sudo rabbitmqctl list_bindings seems to indicate that the queue is in fact deleted after the receiver goes away.
Well, it is true that exclusive queues will auto-delete when the consumer disconnects (see the documentation pasted below). However, there are cases when you want queues to be non-exclusive, yet still auto-delete (for example, if I want to add another consumer).
exclusive
Exclusive queues may only be accessed by the current connection, and are deleted when that connection closes. Passive declaration of an exclusive queue by other connections are not allowed.
auto-delete
If set, the queue is deleted when all consumers have finished using
it. The last consumer can be cancelled either explicitly or because
its channel is closed. If there was no consumer ever on the queue, it
won't be deleted. Applications can explicitly delete auto-delete
queues using the Delete method as normal.
Personally, I prefer to use neither of these parameters, instead opting for the RabbitMQ queue expiration parameter, which is better if I have a consumer disconnect and then re-connect immediately (or a short time) later; messages are not lost in this case. But, of course it all depends upon your application and requirements.
In contrast to what theMayer described, my testing showed that there is a difference in behavior when auto-delete is toggled while exclusive is set to true.
If auto-delete is set to false, the queue is indeed tied to the connection and will disappear when the connection is terminated.
If auto-delete is set to true, the queue will be deleted after the last consumer is cancelled.
There is a difference between a connection and a consumer. You can be connected, but not consuming a given queue. If you need the queue's lifecycle to be tied to your connection rather than to whether or not you're actively consuming it, set auto-delete to false in conjunction with exclusive=true.