I have a basic setup where a consumer is listening to a "/queue/work" for items. The consumer is meant to only consume one item at a time.
to do so i NACK any other item received concurrently and also UNSUBSCRIBE from the queue while the item is getting processed.
if i now SUBSCRIBE to the same queue again, the NACKed messages are not redelivered to the client - unless i drop the whole connection and reconnect the session - which is not exactly what i want to do since it impacts other subscriptions too :(
is there another way to implement this "take one item - ignore while busy" pattern ?
According to the STOMP Protocol Spec, the server does not re-deliver the NACKed message to the client that sent the NACK.
When you re-subscribe, try using a different id header value that is not used by any other subscriber.
To answer my own question - the way to implement it without ever having to NACK a message which produces the redelivery problem is to use a combination of transactions and prefetch settings:
if setting the "prefetch-count": 1 in the STOMP header - the server will only allow one message "in flight" on the channel before sending the next. This means that the client has to ACK/NACK the message before a new one is sent.
So instead of sending an ACK only if the "job" is completed - we instead start a transaction when we receive the message - ACK the message immediately - and COMMIT the transaction when the job is done. This way "failed" jobs are redelivered correctly.
Related
I'm trying to understand how to set the redelivery time for basic.nacked messages. Some important info is that I'm using quorum queues with a redelivery amount of 5 times. Consider the following scenario:
What is happening Now:
Producer sends message: Message X
Consumer handles Message X and runs into an error, in the error handler I use basic.nack()
Message is resent to original queue. Consumer immediately handles that task again.
This process repeats until the redelivery amount has been reached and then it's dead-lettered.
What I actually want:
I want the message that is requeued to wait a few seconds (3-5 sec or something) before it's once again handled by a consumer. I want to do this due to the fact that I'm using OCC & there are instances where delaying the message redelivery solves consistency issues. (for the people wondering why in god's name I need this).
I was able to do this with NATS streaming server, but I don't know how to implement it with rabbitMQ.
Additional info: I'm using amqplib (typescript) as the client and prefetch is set to 10 globally. I'm using AWS MQ for Rabbit as my rabbitMQ host
As far as i know, there isn't a way to add RabbitMQ Delayed Message Plugin to AWS MQ.
You can:
create a new dead_letter_queue with x-message-ttl option, with value you need (3-5 secs)
for this dead_letter_queue dead letter exchange will be your original exchange
create dead_letter_exchange connected with dead_letter_queue
Workflow:
Consumer nack message
Message goes to dead_letter_exchange
From dead_letter_exchange it goes to dead_letter_queue
In dead_letter_queue message waits x-message-ttl time
Message marks as dead and goes to your original exchange (and then to your original queue)
I am using a RabbitMQ Server (v3.8.9) with Java clients.
Use case is:
Our Backend creates messages for different clients. We send them out to their respective Endpoints.
1 Producer -> Outbound Queue -> 1 Consumer
The producer creates messages for n clients
Which the consumer should send out to the clients' endpoints
Messages must be kept in the correct order regarding each client
Works fine, unless all clients are up and running. Problem: If one client becomes unavailable, we need to have a bulletproof retry mechanism for that.
Say:
Wait 1 Minute and try again
All following messages must NOT be delivered before the first failed one and kept in the correct order
If a retry works, then ALL other messages should be send to the client immediately
As you can see, it is not a solution to just "supsend" the consumer, because it should still deliver msg to the other (alive) clients. Due to application limitations and a dynamic number of clients, we cannot spawn one consumer per client queue.
My best approach right now is to dynamically create one queue per client, which are then routed to a single outbound queue. If one msg to one client cannot be delivered by the consumer, I would like to "pause" the clients queue for x minutes. An API call like "queue_pause('client_q1', '5 Minutes')" would help. But even then I have to deal with the other, already routed messages to that particular client and keep them in the correct order...
Any better ideas?
I think the key here is that a single consumer script can consume from multiple queues. So if I'm understanding correctly, you could model this as:
Each client has its own queue. These could be created by the consumer script when it starts up, or by a back-end process when a new client is created.
The consumer script subscribes to each queue separately
When a message is received, the consumer tries to send it immediately to the client; if it succeeds, it is manually acknowledged with basic.ack, and the consumer is ready to send the next message to that client.
When a message cannot be delivered to the client, it is requeued (basic.nack or basic.reject with requeue=1), retaining its position in the client's queue.
The consumer then needs to pause consuming from that particular queue. Depending on how its written, that could be as simple as a sleep in that particular thread, but if that's not practical, you can effectively "pause" the subscription to the queue:
Cancel the subscription to that queue, leaving other subscriptions in tact
Store the queue name and the retry time in an appropriate variable
If the consumer script is implemented with an event/polling loop, check the list of "paused" subscriptions each time around that loop; if the retry time has been reached, re-subscribe.
Alternatively, if the library / framework supports it, register a delayed event that will fire at the appropriate time and re-subscribe the queue. The exact mechanics of this depend on the technologies you're using.
All the other subscriptions will continue, so messages to other clients will be delivered. The queue with no subscribers will retain the messages for the offline client in order until the consumer script starts consuming them again.
There are some of the messages got stuck in UnAck state in RabbitMQ. Is there any way to move them to ready state without restarting the consumer application or without restarting the RabbitMQ server?
Unacked state literally means messages are being consumed and awaiting for Acknowledgement i.e. status update. If your messages are stuck in this state, it mostly likely means your consumers have not provided appropriate acknowledgements for those message.
You can provide acknowledgements in the following ways.
ack the message. This signals to RabbitMQ that the message has been successfully processed/consumed and can be pop from the queue. See https://www.rabbitmq.com/amqp-0-9-1-quickref.html#basic.ack
reject or nack the message. This signals that the message was not processed correctly and should be either "dead-lettered" or "re-queue", depending on the message/queue configuration. See https://www.rabbitmq.com/amqp-0-9-1-quickref.html#basic.reject
Alternatively, you can also set a TTL for your messages, in which case, they will be automatically reject if their time in the queue exceed their TTL. See here https://www.rabbitmq.com/ttl.html.
Is it possible to use Dead Letter Exchanges to handle both retrying a task and archiving tasks that failed more than retry_max number of times?
Most articles I've read set it up so that when a task fails, it is nacked so the task is sent to a retry-queue via a DLX. retry-queue has a TTL and is configured to send the dead message back to worker-queue.
What I would like to do is to send the message to a different queue, e.g.: dead-queue, when the maximum number of retries fails for manual intervention later. Is this possible using nack?
The diagram in the second answer here suggests that I can somehow nack the message when the max retries is exceeded and send it to a "Fail Exchange" but I'm not sure how I can nack the message and have it delivered to a different exchange.
I'm trying to implement a work queue architecture using RabbitMQ. I have a single sender application and multiple consumers.
I use manual ack on the consumers, so in case of failure in handling a request, it will be re-queued for another consumer to handle.
I was wondering what would happen if all the consumers return nack on a specific request. Is there a way to recognize this behavior and mark the request as 'dead' so it's rerouted to the dead letter exchange? In such a case, I'd like to have a separate consumer open on the queue bound to the dead letter exchange and receive all the messages that failed to be handled by any consumer (for logging purposes or executing this request's task locally, without distributed consumers).
Another question I had. When requeueing the request upon receiving NACK from a consumer, will it try to send this request to other consumers or will it try to send to the first available, even if it's the one that already nacked the request?
Thanks
no there is no such a feature in RabbitMQ. You may handle exceptions, and for specific exception send message to the dead queue or if know maximum time that message must live, configure TTL on queue.
if you nack message, it will go to the next AVAILABLE consumer