Question one: Can I subscribe to the event of a message being sent to the _skipped queue?
I am using masstransit together with rabbit mq. Some messages sometimes are sent to the _skipped queue for unclear reasons. The message type has a consumer, the ttl (time to life) is not small. It should not happen, and I am getting a log entry from masstransit, but I want to do more at the moment. Maybe log an error, in test maybe pop-up a window. Is there a way to achieve this? I am only getting these log messages below.
MassTransit.ReceiveTransport|SKIP rabbitmq://localhost/services_admin db270000-1fd6-00ff-3b83-08d9000ef97c
MassTransit.ReceiveTransport|Declare queue: name: services_admin_skipped, durable, consumer-count: 0 message-count: 3
Question two: What exactly happens to messages in the _skipped queue? Can they be resent?
Skipped messages either don't match the type (namespace included), don't have a consumer on the endpoint, or were a response to a request client that is no longer waiting for it. Since it's a receive endpoint queue, it's likely one of the first two reasons. Look at the message body/details in the RabbitMQ Management Console, that should give you some ideas.
You can use a shovel in RabbitMQ to move the messages back into the queue once you've resolved the issue.
Related
I'm trying to understand how to set the redelivery time for basic.nacked messages. Some important info is that I'm using quorum queues with a redelivery amount of 5 times. Consider the following scenario:
What is happening Now:
Producer sends message: Message X
Consumer handles Message X and runs into an error, in the error handler I use basic.nack()
Message is resent to original queue. Consumer immediately handles that task again.
This process repeats until the redelivery amount has been reached and then it's dead-lettered.
What I actually want:
I want the message that is requeued to wait a few seconds (3-5 sec or something) before it's once again handled by a consumer. I want to do this due to the fact that I'm using OCC & there are instances where delaying the message redelivery solves consistency issues. (for the people wondering why in god's name I need this).
I was able to do this with NATS streaming server, but I don't know how to implement it with rabbitMQ.
Additional info: I'm using amqplib (typescript) as the client and prefetch is set to 10 globally. I'm using AWS MQ for Rabbit as my rabbitMQ host
As far as i know, there isn't a way to add RabbitMQ Delayed Message Plugin to AWS MQ.
You can:
create a new dead_letter_queue with x-message-ttl option, with value you need (3-5 secs)
for this dead_letter_queue dead letter exchange will be your original exchange
create dead_letter_exchange connected with dead_letter_queue
Workflow:
Consumer nack message
Message goes to dead_letter_exchange
From dead_letter_exchange it goes to dead_letter_queue
In dead_letter_queue message waits x-message-ttl time
Message marks as dead and goes to your original exchange (and then to your original queue)
I have a producer and a consumer. Multiple instances of the consumer are running. When producer publishes a message, my intention is to consume the message by all the instances. So, I am using the direct exchange. Producer publishes a message to the direct exchange with a topic. Consumers are listening to that topic with the exclusive queue. This process is working fine when the consumer is up and producer publishes a message. But when consumers are down and producer publishes a message, consumers are not consuming this message when up.
I googled about the issue. A suggestion was to use named queue. But if I use named queue, messages will be consumed following the round-robin algorithm. That does not meet my expectation to consume the same message by all the consumers.
Is there any other solution?
Appreciated your help.
There are two solutions to your issue.
Using named queue is one of them.
Set your exchange in fanout mode and subscribe your named queues to it. Doing so, when a publisher send a message in your exchange, it will be dispatched to all the queues listening.
You can then have one or more consumer for each queue (allowing you to scale). You'll have to define a named queue / consumer. When one consumer disconnect, his queue still receive messages and when he comes back he can consume them.
You should be able to do what you want that way.
The other way is more for your personnal knowledge since you said you want to use RabbitMQ. But in that particular case you could use Kafkha, your consummer could then, after reconnection, resume at the message index he was when he disconnected.
Please update me if it doesn't work :)
I've a service A which is publishing message to Queue(Q-A).
I've a dead letter queue(DLQ) bounded to DLX with DLRK.
Queue A is bounded to an exchange(E-A) with a routing key(RA).
I've also set x-letter-exchange(DLX) and x-dead-letter-routing-key(DLRK) on Q-A with ttl-per-message on this queue to 60 seconds
The DLQ is also set with x-letter-exchange(E-A) and x-dead-letter-routing-key(DLRK) with ttl-per-message on this queue to 60 seconds.
With above configuration I'm trying to route the message to DLQ from Q-A after ttl expires and vice versa.
On the consumer side which is another service, I throw AMQPRejectAndDontRequeueException with defaultRequeueRejected set to fals.
The above configuration works fine when the consumer is up and throws the
exception.
But I'm trying to limit my queue size to 1 and then publish 3 messages to the Q-A and also shutting down the consumer. I see all the three messages placed in both Q-A and DLQ and eventually all the messages are dropped.
But if I don't set the queue limit to 1 or start the consumer, everything works fine.
I've also set the x-overflow to reject-publish and when there is overflow, I get a nack at the publisher and then I've a scheduler which publish it again to Q-A.
Note: Both exchanges are Direct and I'm using routing keys to bind it to respective queue.
Kindly, let me know if I'm missing something here and let me know need to share my config
After digging through, I think i finally found the answer from the link Dead-lettering dead-lettered messages in RabbitMQ
answer by pinepain
It is possible to form a cycle of dead-letter queues. For instance, this can happen when a queue dead-letters messages to the default exchange without specifiying a dead-letter routing key. Messages in such cycles (i.e. messages that reach the same queue twice) will be dropped if the entire cycle is due to message expiry.
So I think to solve the problem I need to create another consumer to consume from dead letter queue and publish it back to original queue from the consumer and not directly ttl from the dead letter queue. Please correct me if my understanding is right.
I may have arrived at this too late, But I think I can help you with this.
Story:
You want a retry queue to send dead messages to and retrieve and re-queue them in the main queue after a certain amount of time.
Solution:
Declare your main queue and bind it to an exchange. We call them main_queue and main_exchange and add this feature to the main_queue: x-dead-letter-exchange: retry_exchange
Create your retry queue and bind it to another exchange. We call these retry_queue and retry_exchange and add these features to the retry queue: x-dead-letter-exchange: main_exchange and x-message-ttl: 10000
With this combination, dead messages from main_queue will be sent to retry_queue and after 10 seconds they will be sent again to the main_queue which will they last indefinitely until a consumer declares them dead.
Note: This method works only if you publish your messages to the exchange and not directly in the queue.
I'm in a phase of learning RabbitMQ/AMQP from the RabbitMQ documentation. Something that is not clear to me that I wanted to ask those who have hands-on experience.
I want to have multiple consumers listening to the same queue in order to balance the work load. What I need is pretty much close to the "Work Queues" example in the RabbitMQ tutorial.
I want the consumer to acknowledge message explicitly after it finishes handling it to preserve the message and delegate it to another consumer in case of crash. Handling a message may take a while.
My question is whether AMQP postpones next message processing until the previous message is ack'ed? If so how do I achieve load balancing between multiple workers and guarantee no messages get lost?
No, the other consumers don't get blocked. Other messages will get delivered even if they have unacknowledged but delivered predecessors. If a channel closes while holding unacknowledged messages, those messages get returned to the queue.
See RabbitMQ Broker Semantics
Messages can be returned to the queue using AMQP methods that feature a requeue parameter (basic.recover, basic.reject and basic.nack), or due to a channel closing while holding unacknowledged messages.
EDIT In response to your comment:
Time to dive a little deeper into the AMQP specification then perhaps:
3.1.4 Message Queues
A message queue is a named FIFO buffer that holds message on behalf of a set of consumer applications.
Applications can freely create, share, use, and destroy message queues, within the limits of their authority.
Note that in the presence of multiple readers from a queue, or client transactions, or use of priority fields,
or use of message selectors, or implementation-specific delivery optimisations the queue MAY NOT
exhibit true FIFO characteristics. The only way to guarantee FIFO is to have just one consumer connected
to a queue. The queue may be described as “weak-FIFO” in these cases. [...]
3.1.8 Acknowledgements
An acknowledgement is a formal signal from the client application to a message queue that it has
successfully processed a message.[...]
So acknowledgement confirms processing, not receipt. The broker will hold on to the message until it's gotten acknowleged, so that it can redeliver them. But it is free to deliver more messages to consumers even if the prededing messages have not yet been acknowledged. The consumers will not be blocked.
Assuming that we only have one consumer and our redelivery policy will allow the message to be redelivered for a quite long time.
I've tried a scenario where I sent two messages(different type), one is designed to be redelivered and the other can be consumed normally.
It seems the normal message will be blocked if it is delivered later than the redelivered one.
It will not be consumed until the redelivered message has tried many times reaching the maximum redeliver times. That would lead to a situation where a easy-to-consumed message must wait a long time to be consumed..
I'm wondering how the AMQ redeliver work. When a message is redelivered in a consumer, the other message can be sent to this consumer until current message has been consumed or timeout(to DLQ).
Can someone help ? Thanks,
ActiveMQ's overriding concern when redelivering messages is to honour message ordering on a queue.
Given two messages A and B, which get sent to a queue with a defined redelivery policy as you describe: if a client fails processing A, that message will get placed back on the queue and no other messages will be consumed until A is consumed successfully.
Check out the ActiveMQ Message Redelivery and DLQ Handling section for further details.
Please remember to vote this response up if it answers your question.
For this case it is possible to set the ActiveConnectionFactory to onBlockingRedelivery.
Find details in de ActiveMq Api documentation: