The newly released ActiveMQ (5.12) now supports pausing and resuming message delivery to the consumers for queues which is very useful.
The problem is that, when you resume the message delivery,
If there is a message entering the queue: the broker will immediately send the pending messages to the consumer which is totally OK.
But if no message enters the queue: the pending messages in the queue will not be sent to the consumers until the expiration checking is performed on the queue (which by default is 30 seconds and can be controlled by the expireMessagesPeriod attribute) and non-expired messages will be sent to the consumers afterwards.
Obviously we can change the expireMessagesPeriod to limit this delay, but when you need a milisec precision, performing the expiration check every milisec will not make sense.
Is there anyway to force the queue to start sending messages immediately after resumption?
Bug fixed by ActiveMQ team (details here) and will appear in the 5.12.2 release.
Related
I'm trying to understand how to set the redelivery time for basic.nacked messages. Some important info is that I'm using quorum queues with a redelivery amount of 5 times. Consider the following scenario:
What is happening Now:
Producer sends message: Message X
Consumer handles Message X and runs into an error, in the error handler I use basic.nack()
Message is resent to original queue. Consumer immediately handles that task again.
This process repeats until the redelivery amount has been reached and then it's dead-lettered.
What I actually want:
I want the message that is requeued to wait a few seconds (3-5 sec or something) before it's once again handled by a consumer. I want to do this due to the fact that I'm using OCC & there are instances where delaying the message redelivery solves consistency issues. (for the people wondering why in god's name I need this).
I was able to do this with NATS streaming server, but I don't know how to implement it with rabbitMQ.
Additional info: I'm using amqplib (typescript) as the client and prefetch is set to 10 globally. I'm using AWS MQ for Rabbit as my rabbitMQ host
As far as i know, there isn't a way to add RabbitMQ Delayed Message Plugin to AWS MQ.
You can:
create a new dead_letter_queue with x-message-ttl option, with value you need (3-5 secs)
for this dead_letter_queue dead letter exchange will be your original exchange
create dead_letter_exchange connected with dead_letter_queue
Workflow:
Consumer nack message
Message goes to dead_letter_exchange
From dead_letter_exchange it goes to dead_letter_queue
In dead_letter_queue message waits x-message-ttl time
Message marks as dead and goes to your original exchange (and then to your original queue)
There are some of the messages got stuck in UnAck state in RabbitMQ. Is there any way to move them to ready state without restarting the consumer application or without restarting the RabbitMQ server?
Unacked state literally means messages are being consumed and awaiting for Acknowledgement i.e. status update. If your messages are stuck in this state, it mostly likely means your consumers have not provided appropriate acknowledgements for those message.
You can provide acknowledgements in the following ways.
ack the message. This signals to RabbitMQ that the message has been successfully processed/consumed and can be pop from the queue. See https://www.rabbitmq.com/amqp-0-9-1-quickref.html#basic.ack
reject or nack the message. This signals that the message was not processed correctly and should be either "dead-lettered" or "re-queue", depending on the message/queue configuration. See https://www.rabbitmq.com/amqp-0-9-1-quickref.html#basic.reject
Alternatively, you can also set a TTL for your messages, in which case, they will be automatically reject if their time in the queue exceed their TTL. See here https://www.rabbitmq.com/ttl.html.
I'm not sure how to resiliently handle RabbitMQ messages in the event of an intermittent outage.
I subscribe in a windows service, read the message, then store it my database. If I can't process the record because of the data I publish it to a dead letter queue for a human to address and reprocess.
I am not sure what to do if I have some intermittent technical issue that will fix itself (database reboot, network outage, drive space, etc). I don't want hundreds of messages showing up on dead letter that just needed to wait for a for a glitch but now would be waiting on a human.
Currently, I re-queue the event and retry it once, but it retries so fast the issue is not usually resolved. I thought of retrying forever but I don't want a real issue to get stuck in an infinite loop.
Is a broad topic but from the server side you could persist your messages and make your queues durable, this means that in the eventuality the server gets restarted they won't be lost, check more here How to persist messages during RabbitMQ broker restart?
For the consumer (client) it will depend on how you configure your client, from the docs:
In the event of network failure (or a node crashing), messages can be duplicated, and consumers must be prepared to handle them. If possible, the simplest way to handle this is to ensure that your consumers handle messages in an idempotent way rather than explicitly deal with deduplication.
If a message is delivered to a consumer and then requeued (because it was not acknowledged before the consumer connection dropped, for example) then RabbitMQ will set the redelivered flag on it when it is delivered again (whether to the same consumer or a different one). This is a hint that a consumer may have seen this message before (although that's not guaranteed, the message may have made it out of the broker but not into a consumer before the connection dropped). Conversely if the redelivered flag is not set then it is guaranteed that the message has not been seen before. Therefore if a consumer finds it more expensive to deduplicate messages or process them in an idempotent manner, it can do this only for messages with the redelivered flag set.
Check more here: https://www.rabbitmq.com/reliability.html#consumer
Let's say that a consumer for a queue has been disconnected for some time during which many number of messages are produced.
How long does RabbitMQ keep the messages for the disconnected consumer without durable mode?
(Will it discard the queue right after the consumer is disconnected? or will it keep the queue until the memory allows?)
Does the durable mode will give a functionality for a consumer to consume any message which is published until now? (i.e. random access to the queue, fetching messages out-of-order, or consuming from the beginning of the queue)
There are some TTL extensions.
TTL can be set for a given queue by setting the x-message-ttl argument
to queue.declare, or by setting the message-ttl policy.
No it doesn't. The messages are kept in queue until they are acknowledged, regardless of durability. (unless of course the server dies, then the messages are gone if not previously marked as durable).
ActiveMQ: 5.10.2 inside ServiceMix's Karaf OSGi
KahaDB persistence.
Default broker settings.
Default settings in connections(tcp://x.x.x.x:61616)
16 queues predefined in activemq.xml.
Two client connections to ActiveMQ. One for producer sessions, one for consumer sessions.
Producers send messages to all queues.
16 consumer sessions consumes messages.
All going ok, but:
If I reduce number of consumers to 1 (or 2 or three, I don't know where is threshold) so that messages from 1 queue are consuming and messages from another queues are storing.
While some time passing, I see this picture:
That 1 consumer stop receiving message. He think that there are no more messages.
From activemqweb I can see that message count on that consuming queue is > 0
From activemqweb I cannot see any messages in Message Browser in that consuming queue.
I can see messages from other queues in Message Browser.
If I start some other consumer(or restart activemq) to consume messages from different queue I see:
I start to see messages in first queue Message Browser(those that were sent before but haven't been seen after "freeze").
First queue continue to consuming
Second queue begin to consuming.
The "freeze" can occur again in some time and start consuming another queue will help again.
If I start all consumers I see no "message freeze".
If just stop and start consumer on "frozen" queue, nothing happens. It need to be done on "unfrozen queue" to "unfroze" "frozen queue".
It also happens if there is no active producer, only consumer.
What can it be?
Thank you.
Oups. I've found what it was.
It's just available memory exceeded.
I didn't set -Xms and -Xmx, so it run with only 512mb of max heap.
And when messages size stored and not consumed is closed to the top, I get these behavior.