My setup is as follows:
fanout dl exchange (DLX) => dl quorum queue (DLQ)
direct exchange (MainX) => quorum queue (MainQ) & routing_key =>
policy => {x-dead-letter-exchange: DLX, x-overflow: reject-publish-dlx, x-max-length: 10}
1 Consumer client bound to MainX/MainQ
1 Consumer client bound to DLX/DLQ
1 publisher pushing thousands of messages within seconds to DLQ
Based on the policy, the queue shouldn't hold more than 10 messages and any messages above the length limit should be dropped and dead-lettered into DLX. However the policy seems to be ignored because not a single message gets into the DLQ.
The same thing happens when I use message-ttl or max-length-bytes
I've tried to manually publish a message to the DLQ and it was successfully received and consumed.
I'm hoping someone has run into a similar issue or can tell if I'm doing something wrong based on the above.
I've figured out the issue and it was due to how I had created MainQ originally. I Recreated it with the correct configuration and started receiving messages.
Related
Question one: Can I subscribe to the event of a message being sent to the _skipped queue?
I am using masstransit together with rabbit mq. Some messages sometimes are sent to the _skipped queue for unclear reasons. The message type has a consumer, the ttl (time to life) is not small. It should not happen, and I am getting a log entry from masstransit, but I want to do more at the moment. Maybe log an error, in test maybe pop-up a window. Is there a way to achieve this? I am only getting these log messages below.
MassTransit.ReceiveTransport|SKIP rabbitmq://localhost/services_admin db270000-1fd6-00ff-3b83-08d9000ef97c
MassTransit.ReceiveTransport|Declare queue: name: services_admin_skipped, durable, consumer-count: 0 message-count: 3
Question two: What exactly happens to messages in the _skipped queue? Can they be resent?
Skipped messages either don't match the type (namespace included), don't have a consumer on the endpoint, or were a response to a request client that is no longer waiting for it. Since it's a receive endpoint queue, it's likely one of the first two reasons. Look at the message body/details in the RabbitMQ Management Console, that should give you some ideas.
You can use a shovel in RabbitMQ to move the messages back into the queue once you've resolved the issue.
I have configured a topic in ActiveMQ with lastImageSubscriptionRecoveryPolicy. It works as expected in that the broker persists the last message sent to the topic and when a new consumer subscribes to that topic it receives that last message:
producer publishes N messages
consumer A subscribes to the topic; 1 message is received
consumer B subscribes to the topic; same 1 message is received
However, the message is lost when I restart the broker. This is the sequence of events:
producer publishes N messages
broker is restarted
consumer subscribes to the topic; nothing is received
Is it possible to enable persistence of that last message (Last Image) so that it is persisted even across broker restarts?
No this is not possible. If you need message durability then you either need to use a Queue, or use a Durable Topic subscription in order to keep messages around after restart. The broker has some convenience features for Topics such as recovery policies but they are no real substitute for the stronger guarantees that exists for Queue based messaging, so if you need that then you must use those mechanisms.
I've a service A which is publishing message to Queue(Q-A).
I've a dead letter queue(DLQ) bounded to DLX with DLRK.
Queue A is bounded to an exchange(E-A) with a routing key(RA).
I've also set x-letter-exchange(DLX) and x-dead-letter-routing-key(DLRK) on Q-A with ttl-per-message on this queue to 60 seconds
The DLQ is also set with x-letter-exchange(E-A) and x-dead-letter-routing-key(DLRK) with ttl-per-message on this queue to 60 seconds.
With above configuration I'm trying to route the message to DLQ from Q-A after ttl expires and vice versa.
On the consumer side which is another service, I throw AMQPRejectAndDontRequeueException with defaultRequeueRejected set to fals.
The above configuration works fine when the consumer is up and throws the
exception.
But I'm trying to limit my queue size to 1 and then publish 3 messages to the Q-A and also shutting down the consumer. I see all the three messages placed in both Q-A and DLQ and eventually all the messages are dropped.
But if I don't set the queue limit to 1 or start the consumer, everything works fine.
I've also set the x-overflow to reject-publish and when there is overflow, I get a nack at the publisher and then I've a scheduler which publish it again to Q-A.
Note: Both exchanges are Direct and I'm using routing keys to bind it to respective queue.
Kindly, let me know if I'm missing something here and let me know need to share my config
After digging through, I think i finally found the answer from the link Dead-lettering dead-lettered messages in RabbitMQ
answer by pinepain
It is possible to form a cycle of dead-letter queues. For instance, this can happen when a queue dead-letters messages to the default exchange without specifiying a dead-letter routing key. Messages in such cycles (i.e. messages that reach the same queue twice) will be dropped if the entire cycle is due to message expiry.
So I think to solve the problem I need to create another consumer to consume from dead letter queue and publish it back to original queue from the consumer and not directly ttl from the dead letter queue. Please correct me if my understanding is right.
I may have arrived at this too late, But I think I can help you with this.
Story:
You want a retry queue to send dead messages to and retrieve and re-queue them in the main queue after a certain amount of time.
Solution:
Declare your main queue and bind it to an exchange. We call them main_queue and main_exchange and add this feature to the main_queue: x-dead-letter-exchange: retry_exchange
Create your retry queue and bind it to another exchange. We call these retry_queue and retry_exchange and add these features to the retry queue: x-dead-letter-exchange: main_exchange and x-message-ttl: 10000
With this combination, dead messages from main_queue will be sent to retry_queue and after 10 seconds they will be sent again to the main_queue which will they last indefinitely until a consumer declares them dead.
Note: This method works only if you publish your messages to the exchange and not directly in the queue.
I have 2 RabbitMQ queues:
incoming_message => where I push all messages that I want to process later
incoming_message_dlx => where I push the message whose the processing failed
As you can supposed with its name, the incoming_message_dlx queue use the Dead Letter Exchange feature, that means when the message expires, it will be requeue to my incoming_message.
What I try to achieve is to increase the expiration of messages each time the processing failed and that they are push to the DLX queue.
The problem is that even if a message expired, it will not be requeue to my incoming_message while it's not at the bottom (head) of the queue. So if there is a message with an expiration time of 7 days in the DLX queue and that we enqueue a new message with the expiration time of 5 seconds, this message will only be requeue to the incoming_message after 7 days + 5 seconds...
I've found on the documentation that I can use my DLX queue as a priority queue and put a priority on my messages according to the expiration time, but it doesn't work as expected, the priority seems to be ignored.
However, when I use the RabbitMQ admin (management plugin) and that I get the first message of the queue, it's always the one with the higher priority, but the "internal consumer" of the DLX queue seems to ignore this priority.
Do you know what could be the problem?
Thanks a lot in advance.
PS: I'm using RabbitMQ server version 3.6.10.
as a queue structure(fifo),rabbitmq do expire from the head of the queue.
queue ttl contains 3 type:
Per-Queue Message TTL： x-message-ttl
Per-Message TTL： expiration
Queue TTL：x-expires
when you want the message just deliver on the ttl value ,try use multi level ttl queue.
you can predefined some dlx queue as you need.
eg: you want error message do retry in (5s,15s,60s), you can define 3 dlx queue by set different x-message-ttl value, and this 3 incoming_message_dlx queue binding the dlx router to the incoming_message;
but if you have a message ttl=30s ,you just prefdefind 3 queue with ttl(5s,15s,60s) , so where to diliver ? try priority queue.
offical doc
Messages which should expire will still only expire from the head of the queue. This means that unlike with normal queues,
even per-queue TTL can lead to expired lower-priority messages getting stuck behind
non-expired higher priority ones.
These messages will never be delivered, but they will appear in queue statistics.
expired lower-priority messages getting stuck behind non-expired higher priority ones
queue like [60s(p=1),30s(p=0)] will not happen!
we defined 3 queue ttl(5s,15s,60s),in order to prevent lower ttl message getting stucked , we push the message to the queue with flor ttl not ceil ttl；
so ttl=30s will deliver to queue which ttl=60s,and set priority=1
ttl=30s is between the predefined queue (15s,60s);
set ttl=60s queue's max-priority=1, default is 0;
deliver ttl=30s message with priority=1;
so the message in a queue just like [30,60,60,60,60].
ttl=30s will not be blocked by the ttl=60s.
I have planned to delay the processing of messages in queue by following these two links link1 link2. So, as suggested in the link. I have declared the original queue with the x-dead-letter-exchange and x-dead-letter-routing-key args. Which published the messages to the so called dead-letter-queue when message either failed to get processed by consumer or ttl happen or queue length exceed. Now in the dead-letter-queue similar args have been set along with the ttl parameter. Which is suppose to republish the messages to the original queue after ttl exceed. But the problem is it is dropping all the messages.
Moreover, there is a catch here. If i explicitly publish the failed messages from original queue to dead-letter-queue. Then after ttl it republish the messages to the original queue. Why is it so and how do i make it work. So that dead-letter-queue republishes the messages to the original queue instead of dropping. I am using RabbitMQ 3.0.0.
FYI, I have created both the exchanges of direct type along with the routing key
When a queue has a TTL setup that means that messages in that queue will be sent to the dead-letter-exchange (DLX) associated with that queue after the TTL has expired. If the queue has no DLX assigned then the messages go into the bit bucket.
If you want to send messages back into the queue from which they came to be re-processed then you need to have the setup that I described in this post.
Dead-lettering dead-lettered messages in RabbitMQ
Hopefully that is helpful for you.
Suppose your original exchange is x.notification and is bind to the queue q.A with routing queue A. And your dead-letter-exchange namae is dlx.notification. Now in the queue q.A set ttl the time interval you want to wait and dead-lleter-exchange as dlx.notification. Now create another queue dlq.A to route the expired message from dlx.notification into dlq.A with routing key "A". I think thats all you need to do to achive your goal.