I mean dequeue count in the web console, and acknowledge() method in activemq broker's code.
When one message run acknowledge(), the dequeue count will always add 1 ,is that correct?
When you acknowledge a message that has been queued then it is dequeued and so the dequeued count for the destination increases by one yes.
Related
I have a publisher that sends messages to a consumer that moves a motor.
The motor has a work queue which I cannot access, and it works slower than the rate of the incoming messages, so I'm trying to control the traffic on the consumer.
To keep updated and relevant data coming to the motor without the queue filling up and creating a traffic jam, I set the RabbitMQ queue size limit to 5 and basicQos to 1.
The idea is that the RabbitMQ queue will drop the old messages when it is filled up, so the newest commands are at the front of the queue.
Also by setting basicQos to 1 I ensure that the consumer doesn't grab all messages from the queue and bombards the motor at once, which is exactly what i'm trying to avoid since I can't do anything once the command was sent to the motor.
This way the consumer takes messages from the queue one by one, while new messages replace the old ones on the queue.
Practically this moves the bottleneck to the RabbitMQ queue instead of the motor's queue.
I also cannot check the motor's work queue, so all traffic control must be done on the consumer.
I added messageId and tested, and found out many messages are still coming and going long after the publisher is being shut down.
I'm expecting around 5 messages after shutdown since that's the size of the queue, but i'm getting hundreds.
I also added a few seconds of sleep inside the callback to make sure this isn't the robot queue that's acting up, but i'm still getting many messages after shutdown, and I can see in the logs the callback is being called every time so it's definitely still getting messages from somewhere.
Please help.
Thanks.
Moving the acknowledgment to the end of the callback solved the problem.
I'm guessing that by setting basicQos to 1 it did execute the callback for each message one after another, but in the background it kept grabbing messages from the queue.
So even when the publisher was shutdown, the consumer still had messages that were taken from the queue in it, and those messages were the ones that I saw being executed.
Consider the following scenario:
A message is put on a queue. The queue is declared to be durable.
A single consumer gets the message from the queue but do not acknowledge it. A basic consume was performed on the message by the consumer. At this moment, I can see in rabbitmq management website that the ready_message count goes down by 1 and messages_unacknowledged count goes up by 1.
After that, a basic reject with requeue was performed on the message's delivery tag/info. message_ready and message_unacknowledged count still same here on the queue.
Now, I performed a basic recover on the same consumer on the same channel. However, even after executing the basic recover and adding some sleep did not make the the message_unacknowledged count go down by 1 and message_ready count go up by 1.
Finally, when I close the current channel, I see the message_unacknowledged count go down by 1 and message_ready count go up by 1.
My question is, Is it not possible to do a recover the unacknowledged messages on the same consumer unless you close your current channel in RabbitMQ?
Is it possible to disable prefetching with auto-ack=false? I just want to avoid reading message (prefetching) from a queue every time I acknowledge a message. I want to read a message only when I call a 'consume_message'. Setting prefetch_count=0 seems doesn't work and it's treated as 'no specif limit'.
UPDATED:
As I understand 'prefetch_count' is the number of messages cached on the client side (read locally into buffers). For example there is a use case:
(let's assume there is a queue we connect to and it has messages)
Create a connection.
Set Basic.Qos (prefetch_count=1)
Start consuming Basic.Consume
Due to the prefetch_count=1 one message is already transferred to the client and ready to be read and marked as not-ack'd.
Reading message and then processing it.
Then the message is ack'd. And everything starts from step 4.
I thought that setting prefetch_count to 0 would avoid the step 4 and a message is transferred only when you read it - no caching on the client side.
Prefetch and auto-acknowledgment are not related like that. Prefetch count is simply a number of unacknowledged messages prepared to be delivered to a specific consumer.
Let's say you set prefetch count to N. If you set auto-ack to true, these means that these N messages are ACKed upon receiving. If you set it to false, this means that you still get the N messages but they're not ACKed until you manually ACK them.
For the last part - try setting prefetch_count to 1.
Also check this question and both answers.
I want to read the payload, or messageId of unacknowledged messages in a RabbitMQ queue. Is this possible?
The reason I want to do so is I trying to use RabbitMQ dead letter feature to build a cycle to for auto-generating message periodically.
Briefly, create two queues - work queue and delay queue.
Set TTL of the message in delay queue as the time frequency of need to periodically. Can have different messages with different TTL for different job purpose;
put a message into the delay queue. When the message expires, it gets republished into the work queue. The message can sit in the work queue as long as needed until a consumer is up to consume it.
One consumer picks up the message, and process it. If processing succeeds, the consumer needs acknowledge the work queue, and then write the message back to the delay queue; If processing fails (e.g., the thread crashes), no acknowledgement. Then the message would re-appear in the worker queue automatically. Then another consumer can take up the job. When the message sent back to the delay queue gets expired again, it gets republished, then re-consumed by a consumer ...... A cycle constructed, workload distributed.
I want to make sure there is no missing or duplicate messages in the cycle since I do not want missing job or double doing the job at the same time. However, there is tiny tiny chance duplicate messages can happen. Below show the consumer first write back the message to delay queue, and acknowledge the work queue. If the thread crashes right between below two lines, the message would be in the delay queue, and Rabbit republish the message again into work queue. The end up with duplicate messages in the cycle.
channel.basicPublish(DELAY_EXCHANGE, "", null, message.getBytes());
channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
To prevent above, I want to add a dog watch logic after above two line:
Check the total number of messages in the cycle (total messages in both queues) to see whether it is equal my expected number (I expected the number less 10);
If the number does not matches, I want to figure out which one is missing or which one is duplicate, then deal with it. I do not care about the sequence of those messages, or the frequency has been disturbed since this is a really really edge case to consider. I can easily retrieve those messages which are ready and requeue them. But the problem is how to deal with those unacknowledged messages?
Thank you very much in advance!
Roy
It's not possible to read unacknowledged messages from other context the original messages was consumed and held as un-aked.
If I post 3 messages to a topic with 2 active consumers, what will the dequeue count be after all messages are successfully consumed, 3 or 6? From my JConsole I think 6 (it shows enQ=3 and deQ=6) but can you confirm?
Yes your assumption is correct. But keep in mind that it might not always be an exact multiple, if one of the consumers disconnects for a period of time and then reconnects, the dequeue count will not include the messages missed by that client while it was disconnected.