Can I retry a message N times and then send it to a dead queue without making ack and republishing the message from the consumer?
The only way I can think of is to use multiple queues with dlx setup which fills the next retry queue like this:
test ---> test.retries.1 ---> ... ---> test.retries.N ---> test.dead
Is this ok? I am not sure want I mean by ok. I've started playing with rabbitmq recently. Let's say is this a common setup? Are there any disadvantages?
Is there another way? Maybe a plugin that adds a counter to basic.reject and does the same thing?
Side note: I want to known this because I distrust the idea of having a consumer that will acknowledge a message (even though he cannot process it) and then publish it again. At the end you will end up with multiple liers that will publish a message and from time to time fetch it immediately before everyone else "just to be sure" and.. you'll make them remember.. (and they won't) [this also happens in the scenario with the multiple retry queues but at least the broker will control where the message is going not the consumer]
basic.reject with requeue + TTL
You have one queue and you requeue the message multiple times on failure and when the ttl expires you can setup a dlx.
basic.reject with multiple queues
On failure you always do basic.reject without requeue and use the dlx to send the message to the next retry queue:
test ---> test.retries.1 ---> ... ---> test.retries.N ---> test.z_dead
At the moment I am using this approach with only 1 retry queue and I have a special queue that receives certain messages from the dlx and sends me an email. (In my case a message is acknowledged in few hours)
basic.reject with counting of the number of retries
When you do basic.reject without requeue and use a dlx you can check x-death header added by the dlx to determine the number of retries.
Here is how it is done in sneakers - a ruby gem:
---> test (queue)
|
| test.retry (exchange)
|
---> test.retry (queue - wait for some time with ttl)
|
| test.retry.requeue (exchange)
|
---> test (queue)
At the end you count how many times you have passed through the test queue and when you exceed your retry count, you'll have to acknowledge the message (maybe after publishing it somewhere so you could be notified for the error).
Related
I'm trying to understand how to set the redelivery time for basic.nacked messages. Some important info is that I'm using quorum queues with a redelivery amount of 5 times. Consider the following scenario:
What is happening Now:
Producer sends message: Message X
Consumer handles Message X and runs into an error, in the error handler I use basic.nack()
Message is resent to original queue. Consumer immediately handles that task again.
This process repeats until the redelivery amount has been reached and then it's dead-lettered.
What I actually want:
I want the message that is requeued to wait a few seconds (3-5 sec or something) before it's once again handled by a consumer. I want to do this due to the fact that I'm using OCC & there are instances where delaying the message redelivery solves consistency issues. (for the people wondering why in god's name I need this).
I was able to do this with NATS streaming server, but I don't know how to implement it with rabbitMQ.
Additional info: I'm using amqplib (typescript) as the client and prefetch is set to 10 globally. I'm using AWS MQ for Rabbit as my rabbitMQ host
As far as i know, there isn't a way to add RabbitMQ Delayed Message Plugin to AWS MQ.
You can:
create a new dead_letter_queue with x-message-ttl option, with value you need (3-5 secs)
for this dead_letter_queue dead letter exchange will be your original exchange
create dead_letter_exchange connected with dead_letter_queue
Workflow:
Consumer nack message
Message goes to dead_letter_exchange
From dead_letter_exchange it goes to dead_letter_queue
In dead_letter_queue message waits x-message-ttl time
Message marks as dead and goes to your original exchange (and then to your original queue)
I've a service A which is publishing message to Queue(Q-A).
I've a dead letter queue(DLQ) bounded to DLX with DLRK.
Queue A is bounded to an exchange(E-A) with a routing key(RA).
I've also set x-letter-exchange(DLX) and x-dead-letter-routing-key(DLRK) on Q-A with ttl-per-message on this queue to 60 seconds
The DLQ is also set with x-letter-exchange(E-A) and x-dead-letter-routing-key(DLRK) with ttl-per-message on this queue to 60 seconds.
With above configuration I'm trying to route the message to DLQ from Q-A after ttl expires and vice versa.
On the consumer side which is another service, I throw AMQPRejectAndDontRequeueException with defaultRequeueRejected set to fals.
The above configuration works fine when the consumer is up and throws the
exception.
But I'm trying to limit my queue size to 1 and then publish 3 messages to the Q-A and also shutting down the consumer. I see all the three messages placed in both Q-A and DLQ and eventually all the messages are dropped.
But if I don't set the queue limit to 1 or start the consumer, everything works fine.
I've also set the x-overflow to reject-publish and when there is overflow, I get a nack at the publisher and then I've a scheduler which publish it again to Q-A.
Note: Both exchanges are Direct and I'm using routing keys to bind it to respective queue.
Kindly, let me know if I'm missing something here and let me know need to share my config
After digging through, I think i finally found the answer from the link Dead-lettering dead-lettered messages in RabbitMQ
answer by pinepain
It is possible to form a cycle of dead-letter queues. For instance, this can happen when a queue dead-letters messages to the default exchange without specifiying a dead-letter routing key. Messages in such cycles (i.e. messages that reach the same queue twice) will be dropped if the entire cycle is due to message expiry.
So I think to solve the problem I need to create another consumer to consume from dead letter queue and publish it back to original queue from the consumer and not directly ttl from the dead letter queue. Please correct me if my understanding is right.
I may have arrived at this too late, But I think I can help you with this.
Story:
You want a retry queue to send dead messages to and retrieve and re-queue them in the main queue after a certain amount of time.
Solution:
Declare your main queue and bind it to an exchange. We call them main_queue and main_exchange and add this feature to the main_queue: x-dead-letter-exchange: retry_exchange
Create your retry queue and bind it to another exchange. We call these retry_queue and retry_exchange and add these features to the retry queue: x-dead-letter-exchange: main_exchange and x-message-ttl: 10000
With this combination, dead messages from main_queue will be sent to retry_queue and after 10 seconds they will be sent again to the main_queue which will they last indefinitely until a consumer declares them dead.
Note: This method works only if you publish your messages to the exchange and not directly in the queue.
I am using java client of
https://www.rabbitmq.com/tutorials/tutorial-six-java.html
. My setup is RPC. My server is creating queue and client is also creating same queue and sending the message. After receiving message server is performing some operation and sending result back to client.
Now if server created the queue and connect with it while queue get's deleted for some reason. The server is not throwing any exception and when the client is creating the same queue and putting messages server is not getting those messages either as it's not connected.
How do server knows that the queue get deleted?
Thanks so much
It sounds like the following situation is happening:
Queue A is created.
Consumer 1 subscribes to Queue A
Queue A is deleted while Consumer 1 is still active
Queue A is re-created (call it A')
Now, you're wondering why Consumer 1 is not getting any messages? You would have to re-subscribe your consumer. I don't usually delete queues, because there is no need to do so under any reasonable scenario (instead, use the queue.expires property to handle auto-deletion of queues).
According to the AMQP 0-9-1 Specification,
When a queue is deleted any pending messages are sent to a dead-letter
queue if this is defined in the server configuration, and all
consumers on the queue are cancelled.
So, based on the description of the behavior, this is a bug with the consumer. It should throw an exception or otherwise exit the consuming loop in this case. In any case, you'll have to re-subscribe to A' before you'll get any more messages.
I have 2 RabbitMQ queues:
incoming_message => where I push all messages that I want to process later
incoming_message_dlx => where I push the message whose the processing failed
As you can supposed with its name, the incoming_message_dlx queue use the Dead Letter Exchange feature, that means when the message expires, it will be requeue to my incoming_message.
What I try to achieve is to increase the expiration of messages each time the processing failed and that they are push to the DLX queue.
The problem is that even if a message expired, it will not be requeue to my incoming_message while it's not at the bottom (head) of the queue. So if there is a message with an expiration time of 7 days in the DLX queue and that we enqueue a new message with the expiration time of 5 seconds, this message will only be requeue to the incoming_message after 7 days + 5 seconds...
I've found on the documentation that I can use my DLX queue as a priority queue and put a priority on my messages according to the expiration time, but it doesn't work as expected, the priority seems to be ignored.
However, when I use the RabbitMQ admin (management plugin) and that I get the first message of the queue, it's always the one with the higher priority, but the "internal consumer" of the DLX queue seems to ignore this priority.
Do you know what could be the problem?
Thanks a lot in advance.
PS: I'm using RabbitMQ server version 3.6.10.
as a queue structure(fifo),rabbitmq do expire from the head of the queue.
queue ttl contains 3 type:
Per-Queue Message TTL： x-message-ttl
Per-Message TTL： expiration
Queue TTL：x-expires
when you want the message just deliver on the ttl value ,try use multi level ttl queue.
you can predefined some dlx queue as you need.
eg: you want error message do retry in (5s,15s,60s), you can define 3 dlx queue by set different x-message-ttl value, and this 3 incoming_message_dlx queue binding the dlx router to the incoming_message;
but if you have a message ttl=30s ,you just prefdefind 3 queue with ttl(5s,15s,60s) , so where to diliver ? try priority queue.
offical doc
Messages which should expire will still only expire from the head of the queue. This means that unlike with normal queues,
even per-queue TTL can lead to expired lower-priority messages getting stuck behind
non-expired higher priority ones.
These messages will never be delivered, but they will appear in queue statistics.
expired lower-priority messages getting stuck behind non-expired higher priority ones
queue like [60s(p=1),30s(p=0)] will not happen!
we defined 3 queue ttl(5s,15s,60s),in order to prevent lower ttl message getting stucked , we push the message to the queue with flor ttl not ceil ttl；
so ttl=30s will deliver to queue which ttl=60s,and set priority=1
ttl=30s is between the predefined queue (15s,60s);
set ttl=60s queue's max-priority=1, default is 0;
deliver ttl=30s message with priority=1;
so the message in a queue just like [30,60,60,60,60].
ttl=30s will not be blocked by the ttl=60s.
It seems the longer I keep my rabbitmq server running, the more trouble I have with unacknowledged messages. I would love to requeue them. In fact there seems to be an amqp command to do this, but it only applies to the channel that your connection is using. I built a little pika script to at least try it out, but I am either missing something or it cannot be done this way (how about with rabbitmqctl?)
import pika
credentials = pika.PlainCredentials('***', '***')
parameters = pika.ConnectionParameters(host='localhost',port=5672,\
credentials=credentials, virtual_host='***')
def handle_delivery(body):
"""Called when we receive a message from RabbitMQ"""
print body
def on_connected(connection):
"""Called when we are fully connected to RabbitMQ"""
connection.channel(on_channel_open)
def on_channel_open(new_channel):
"""Called when our channel has opened"""
global channel
channel = new_channel
channel.basic_recover(callback=handle_delivery,requeue=True)
try:
connection = pika.SelectConnection(parameters=parameters,\
on_open_callback=on_connected)
# Loop so we can communicate with RabbitMQ
connection.ioloop.start()
except KeyboardInterrupt:
# Gracefully close the connection
connection.close()
# Loop until we're fully closed, will stop on its own
connection.ioloop.start()
Unacknowledged messages are those which have been delivered across the network to a consumer but have not yet been ack'ed or rejected -- but that consumer hasn't yet closed the channel or connection over which it originally received them. Therefore the broker can't figure out if the consumer is just taking a long time to process those messages or if it has forgotten about them. So, it leaves them in an unacknowledged state until either the consumer dies or they get ack'ed or rejected.
Since those messages could still be validly processed in the future by the still-alive consumer that originally consumed them, you can't (to my knowledge) insert another consumer into the mix and try to make external decisions about them. You need to fix your consumers to make decisions about each message as they get processed rather than leaving old messages unacknowledged.
If messages are unacked there are only two ways to get them back into the queue:
basic.nack
This command will cause the message to be placed back into the queue and redelivered.
Disconnect from the broker
This action will force all unacked messages from this channel to be put back into the queue.
NOTE: basic.recover will try to republish unacked messages on the same channel (to the same consumer), which is sometimes the desired behaviour.
RabbitMQ spec for basic.recover and basic.nack
The real question is: Why are the messages unacknowledged?
Possible scenarios to cause unacked messages:
Consumer fetching too many messages, then not processing and acking them quickly enough.
Solution: Prefetch as few messages as appropriate.
Buggy client library (I have this issue currently with pika 0.9.13. If the queue has a lot of messages, a certain number of messages will get stuck unacked, even hours later.
Solution: I have to restart the consumer several times until all unacked messages are gone from the queue.
All the unacknowledged messages will go to ready state once all the workers/consumers are stopped.
Ensure all workers are stopped by confirming with a grep on ps aux output, and stopping/killing them if found.
If you are managing workers using supervisor, which shows as worker is stopped, you may want to check for zombies. Supervisor reports the worker to be stopped but still you will find zombie processes running when grepped on ps aux output. Killing the zombie processes will bring messages back to ready state.