How can I delete each non processed messages from my queue automatically after 60 seconds?
Use the option x-message-ttl in queue properties(not message properties).
set a value of milliseconds to auto expire messages
x-message-ttl: 60000 means expire and remove each message after 60 seconds
Related
If my queue queue1 has property: x-message-ttl: 60000, but for part of the messages in this queue I want TTL 120000 or no TTL at all.
TLDR: All messages by default in this queue lives 1 minute, but part of them 2 minutes or has no time to live at all
RabbitMQ has separate mechanisms for setting per-queue and per-message TTLs, as described in the online manual. Using both at once is explicitly mentioned on that page:
When both a per-queue and a per-message TTL are specified, the lower value between the two will be chosen.
As such, neither of your examples will work:
Queue-level TTL of 1 minute + message-level TTL of 2 minutes => message will be deleted after 1 minute
Queue-level TTL of 1 minute + no message-level TTL => message will be deleted after 1 minute
But other combinations would work:
Queue-level TTL of 1 minute + message-level TTL of 30 seconds => message will be deleted after 30 seconds
Queue-level TTL of 2 minutes + message-level TTL of 1 minute => message will be deleted after 1 minute
This kind of makes sense: a message can end up in multiple queues, so if it specifies a TTL of 30 seconds, it shouldn't last longer than that in any of them; and a queue that's told to delete messages after 30 seconds doesn't care about individual message settings.
I'm trying to implement exponential backoff with a RabbitMQ headers exchange, and I had each queue be bound with x-match: "all" and x-retry-count: [RETRY COUNT FOR THIS LEVEL]. However, what I found was that if I try to retry a task and I have backoff queues for 100, 200, 400, and 800 millisecond wait time, each task I send to the retry exchange somehow matches every queue.
As you can see in the picture below, for the 200ms backoff queue, I'm binding the header x-retry-count: 2, but a task with the header x-retry-count: 1 is matching it (and the x-retry-count values for all other queues in the backoff exchange too). Why would that be?
Found what was going on. x-retry-count doesn't count as a header that can be matched on because it starts with x-; naming the header retry-count does work
How can I implement a queue with configurable x-message-ttl?
I have a queue with x-message-ttl set to 1 minute and I want to change it to 2 minute at runtime. How can this be achieved?
I already tried declaring queue again with x-message-ttl = 2 minutes but neither ttl is changing by this nor message is being published.
if you create a queue with arguments x-message-ttl you can't change it, you have to remove and recreate the queue.
but you can use the policies:
Create queues without ttl arguments
create the policy, for example:rabbitmqctl set_policy expiry ".*" "{""expires"":1800000}" --apply-to queues
In this way you can change the queue TTL value
we have setup some workflow environment with Rabbit.
It solves our needs but I like to know if it is also good practise to do it like we do for scheduled tasks.
Scheduling means no mission critical 100% adjusted time. So if a job should be retried after 60 seconds, it does mean 60+ seconds, depends on when the queue is handled.
I have created one Q_WAIT and made some headers to transport settings.
Lets do it like:
Worker is running subscribed on Q_ACTION
If the action missed (e.g. smtp server not reachable)
-> (Re-)Publish the message to Q_WAIT and set properties.headers["scheduled"] = time + 60seconds
Another process loops every 15 seconds through all messages in Q_WAIT by method pop() and NOT by subscribed
q_WAIT.pop(:ack => true) do |delivery_info,properties,body|...
if (properties.headers["scheduled"] has reached its time)
-> (Re-)Publish the message back to Q_ACTION
ack(message)
after each loop, the connection is closed so that the NOT (Re-)Published are left in Q_WAIT because they were not acknowledged.
Can someone confirm this as a working (good) practise.
Sure you can use looping process like described in original question.
Also, you can utilize Time-To-Live Extension with Dead Letter Exchanges extension.
First, specify x-dead-letter-exchange Q_WAIT queue argument equal to current exchange and x-dead-letter-routing-key equal to routing key that Q_ACTION bound.
Then set x-message-ttl queue argument set or set message expires property during publishing if you need custom per-message ttl (which is not best practice though while there are some well-known caveats, but it works too).
In this case your messages will be dead-lettered from Q_WAIT to Q_ACTION right after their ttl expires without any additional consumers, which is more reliable and stable.
Note, if you need advanced re-publish logic (change message body, properties) you need additional queue (say Q_PRE_ACTION) to consume messages from, change them and then publish to target queue (say Q_ACTION).
As mentioned here in comments I tried that feature of x-dead-letter-exchange and it worked for most requirements. One question / missunderstandig is TTL-PER-MESSAGE option.
Please look on the example here. From my understanding:
the DLQ has a timeout of 10 seconds
so first message will be available on subscriber 10 seconds after publishing.
the second message is posted 1 second after the first with a message-ttl (expiration) of 3 seconds
I would expect the second message should be prounounced after 3 seconds from publishing and before first message.
But it did not work like that, both are available after 10 seconds.
Q: Shouldn't the message expiration overrule the DLQ ttl?
#!/usr/bin/env ruby
# encoding: utf-8
require 'bunny'
B = Bunny.new ENV['CLOUDAMQP_URL']
B.start
DELAYED_QUEUE='work.later'
DESTINATION_QUEUE='work.now'
def publish
ch = B.create_channel
# declare a queue with the DELAYED_QUEUE name
q = ch.queue(DELAYED_QUEUE, :durable => true, arguments: {
# set the dead-letter exchange to the default queue
'x-dead-letter-exchange' => '',
# when the message expires, set change the routing key into the destination queue name
'x-dead-letter-routing-key' => DESTINATION_QUEUE,
# the time in milliseconds to keep the message in the queue
'x-message-ttl' => 10000,
})
# publish to the default exchange with the the delayed queue name as routing key,
# so that the message ends up in the newly declared delayed queue
ch.basic_publish('message content 1 ' + Time.now.strftime("%H-%M-%S"), "", DELAYED_QUEUE, :persistent => true)
puts "#{Time.now}: Published the message 1"
# wait moment before next publish
sleep 1.0
# puts this with a shorter ttl
ch.basic_publish('message content 2 ' + Time.now.strftime("%H-%M-%S"), "", DELAYED_QUEUE, :persistent => true, :expiration => "3000")
puts "#{Time.now}: Published the message 2"
ch.close
end
def subscribe
ch = B.create_channel
# declare the destination queue
q = ch.queue DESTINATION_QUEUE, durable: true
q.subscribe do |delivery, headers, body|
puts "#{Time.now}: Got the message: #{body}"
end
end
subscribe()
publish()
sleep
In the Active MQ Admin console of what is the difference between "Number Of Pending Messages" and "Messages Enqueued"? When a Message is placed on to the queue, should both these values should match?
pending messages = number of messages CURRENTLY waiting for delivery in the destination (the current size of the queue)
enqueued messages = number of messages that where enqueued in the destination since the last statistic reset. This number can only rise.
dequeued messages = messages delivered from the destination to consumers. this number can be higher that the number of enqueued messages if a message was delivered to multiple consumers (topics).
Messages Enqueued = Number of messages sent to the queue since the server start
Messages Dequeued = Number of messages received+deleted since the server start