What's the difference between Pending Queue Size, Dispatched Queue Size, Dispatched Counter, Enqueue Counter, Dequeue Counter in ActiveMQ Admin Console? How to detect which consumer is slow?
Enqueue Count: Number of messages sent to the destination
Dequeue Count: Number of messages that have permanently been removed from the destination
Dispatch Count: Number of messages that have been sent to a consumer
(this may be higher than total enqueue count if consumers reject
messages and the broker has to redeliver messages)
Queue Size: Number of messages currently in the queue (aka Enqueue Count minus
Dequeue Count)
Related
There is a single consumer that receives all messages from a particular queue. The source of the messages is a single publisher. In my case it is a one-way communication and it is important to deliver all messages in order without losses. How can the consumer know that some messages have been dropped by RabbitMQ, for example, because the queue size limit has been reached?
I know that a message sequence number (defined by RabbitMQ) is used to confirm delivery to the consumer, so the question is: Is the sequence number a signal that messages have been dropped? For example, if the sequence number is not linear (difference between the received message and the previous message is greater than 1).
I am facing a scenario where every message in the queue should be consumed after a delay that is achievable using TTL and Dead Letter Exchange in RabbitMQ but when a max TTL will be at the head than no other messages will be consumed.
I want to achieve that every message in the queue gets consumed after it's delay irrespective of its position in the queue.
Instead of relying on TTL and the dead-letter exchange you can use the delayed-message-exchange. With the delayed-message-exchange, the message is published to the queue only when its TTL expires. Consuming every message in the queue will require to add more consumers to the queue.
I have 2 RabbitMQ queues:
incoming_message => where I push all messages that I want to process later
incoming_message_dlx => where I push the message whose the processing failed
As you can supposed with its name, the incoming_message_dlx queue use the Dead Letter Exchange feature, that means when the message expires, it will be requeue to my incoming_message.
What I try to achieve is to increase the expiration of messages each time the processing failed and that they are push to the DLX queue.
The problem is that even if a message expired, it will not be requeue to my incoming_message while it's not at the bottom (head) of the queue. So if there is a message with an expiration time of 7 days in the DLX queue and that we enqueue a new message with the expiration time of 5 seconds, this message will only be requeue to the incoming_message after 7 days + 5 seconds...
I've found on the documentation that I can use my DLX queue as a priority queue and put a priority on my messages according to the expiration time, but it doesn't work as expected, the priority seems to be ignored.
However, when I use the RabbitMQ admin (management plugin) and that I get the first message of the queue, it's always the one with the higher priority, but the "internal consumer" of the DLX queue seems to ignore this priority.
Do you know what could be the problem?
Thanks a lot in advance.
PS: I'm using RabbitMQ server version 3.6.10.
as a queue structure(fifo),rabbitmq do expire from the head of the queue.
queue ttl contains 3 type:
Per-Queue Message TTL： x-message-ttl
Per-Message TTL： expiration
Queue TTL：x-expires
when you want the message just deliver on the ttl value ,try use multi level ttl queue.
you can predefined some dlx queue as you need.
eg: you want error message do retry in (5s,15s,60s), you can define 3 dlx queue by set different x-message-ttl value, and this 3 incoming_message_dlx queue binding the dlx router to the incoming_message;
but if you have a message ttl=30s ,you just prefdefind 3 queue with ttl(5s,15s,60s) , so where to diliver ? try priority queue.
offical doc
Messages which should expire will still only expire from the head of the queue. This means that unlike with normal queues,
even per-queue TTL can lead to expired lower-priority messages getting stuck behind
non-expired higher priority ones.
These messages will never be delivered, but they will appear in queue statistics.
expired lower-priority messages getting stuck behind non-expired higher priority ones
queue like [60s(p=1),30s(p=0)] will not happen!
we defined 3 queue ttl(5s,15s,60s),in order to prevent lower ttl message getting stucked , we push the message to the queue with flor ttl not ceil ttl；
so ttl=30s will deliver to queue which ttl=60s,and set priority=1
ttl=30s is between the predefined queue (15s,60s);
set ttl=60s queue's max-priority=1, default is 0;
deliver ttl=30s message with priority=1;
so the message in a queue just like [30,60,60,60,60].
ttl=30s will not be blocked by the ttl=60s.
I would like to have this constraint on a queue in RabbitMQ:
Next message in the queue can't be dequeued before previous message (the one being processed) is acked.
Through this I will achieve ordered processing of events and parallel processing across multiple queues. How do I/can I configure RabbitMQ for this?
Edit (clarification): There will be many consumers all trying to get work from all the queues and since they can't get work from a queue that has an event being processed that isn't acked - ordered processing is maintained.
Next message in the queue can't be dequeued before previous message (the one being processed) is acked.
you can do this through the consumer prefetch limit for a single consumer.
Through this I will achieve ordered processing of events and parallel processing across multiple queues.
unfortunately, this won't have the effect that you want.
you can set for an individual consumer. that consumer will wait for a message to be acknowledged before getting the next one.
However, this applies to the individual consumer, not the queue.
if you have 2 consumers, each of them will process a message in parallel. if you have 10 consumers, 10 messages will be processed in parallel.
the only way to process every message in order, is to have a single consumer with a prefetch of 1.
I'm trying to do some kind of administration on our activemq process, and I'd like to know what some of the fields mean that I get back.
Specifically,
What is InFlightCount? Is that the number of messages that the producer has sent but not
What is AverageEnqueueTime measured in? Seconds?
What are EnqueueCount and DequeueCount?
What is DispatchCount?
Quoting from here:
InFlightCount: The number of messages that have been dispatched and are
currently awaiting acknowledgment from the consumer. So as this number
decreases, the DequeueCount increases.
AverageEnqueueTime: On average, the amount of time (ms) that messages
remained enqueued. Or average time it is taking the consumers to
successfully process messages.
EnqueueCount: The number of messages that have been written to the queue
over the lifetime of the queue.
DequeueCount: The number of messages that have been successfully (i.e.,
they’ve been acknowledged from the consumer) read off the queue over the
lifetime of the queue.
DispatchCount: The number of messages that have been dispatched (sent) to
the consumer over the lifetime of the queue. Note that dispatched messages
may not have all been acknowledged.
Some points are not matching my observation. One thing is dequeue count; I noticed that
dequeue = expired + dispatched
So dequeue = removed from queue by any reason(expired or dispatched); it does not necessarily mean acked; it's just "number of messages removed from the queue by any reason"; if expired, there's should not be ack, ofc.
Even doc of ActiveMQ does not match what I saw.