I have an application with RabbitMQ where I get the number of messages in a Rabbit queue using the HTTP API (/api/queues/vhost/name).
However, it appears that this information is refreshed from time to time (by default every 5 seconds). I thought the information was always up to date, and it was the administration page that was updated in a given interval.
Is there any way to get the number of messages in a queue with real-time information?
Thank you
The management database is updated each 5 seconds by default.
use the command line rabbitmqctl list_queues for real-time values.
Try to use:
channel.messageCount(you_queue)
see if it works for you
/**
* Returns the number of messages in a queue ready to be delivered
* to consumers. This method assumes the queue exists. If it doesn't,
* an exception will be closed with an exception.
* #param queue the name of the queue
* #return the number of messages in ready state
* #throws IOException Problem transmitting method.
*/
long messageCount(String queue) throws IOException;
Related
When using StreamMessageListenerContainer a subscription for a consumer group can be created by calling:
receive(consumer, readOffset, streamListener)
Is there a way to configure the container/subscription so that it will always attempt to re-process any PENDING messages before moving on to polling for new messages?
The goal would be to keep retrying any message that wasn't acknowledged until it succeeds, to ensure that the stream of events is always processed in exactly the order it was produced.
My understanding is if we specify the readOffset as '>' then on every poll it will use '>' and it will never see any messages from the PENDING list.
If we provide a specific message id, then it can see messages from the PENDING list, but the way the subscription updates the lastMessageId is like this:
pollState.updateReadOffset(raw.getId().getValue());
V record = convertRecord(raw);
listener.onMessage(record);
So even if the listener throws an exception, or just doesn't acknowledge the message id, the lastMessageId in pollState is still updated to this message id and won't be seen again on the next poll.
I have a Splunk forwarder which sends events to two third-party systems(through TCP) and also index them into a splunk indexer.
The problem I'm facing on is that if any of the two third-pary systems goes down. Splunk stops indexing events and neither sends them to the other system.
The output.conf I have is:
[tcpout]
defaultGroup = default-system1, default-system2
indexAndForward = 0
[tcpout:default-system1]
server = <IP>:<PORT>
[tcpout-server://<IP>:<PORT>]
[tcpout:default-system2]
server = <IP>:<PORT>
sendCookedData = false
Is there any way to avoid such a dependency? If one of the destinatios servers is down, it doesn't affect the other sending. I've been looking at the documentation and there are some options that could be use.
heartbeatFrequency in combination with sendCookedData.
heartbeatFrequency = <integer>
* How often (in seconds) to send a heartbeat packet to the receiving server.
* This setting is a mechanism for the forwarder to know that the receiver
(indexer) is alive. If the indexer does not send a return packet to the
forwarder, the forwarder declares the receiver unreachable and does not
forward data to it.
* The forwarder only sends heartbeats if the 'sendCookedData' setting
is set to "true".
* Default: 30
sendCookedData = <boolean>
* Whether or not to send processed or unprocessed data to the receiving server.
* If set to "true", events are cooked (have been processed by Splunk software).
* If set to "false", events are raw and untouched prior to sending.
* Set to "false" if you are sending events to a third-party system.
* Default: true
But I'm not sure if it's the most correct approach, based on the description of sendCookedData, "Set to "false" if you are sending events to a third-party system."
Don't send cooked data to third-party systems. That way lies sadness.
Unfortunately, the behavior you describe is normal for Splunk. From all outward appearances, there is a single output queue and all destinations are fed from that one queue. To avoid data loss, all sending stops when one destination is unavailable.
I have a http server which receives some messages and must reply 200 when a message is successfully stored in a queue and 500 is the message is not added to the queue.
I would like rabbitmq to refuse my messages when the queue reach a size limit.
How can I do it?
actually you can't configure RabbitMq is such a way. but you may programatically check queue size like:
`DeclareOk queueOkStatus = channel.queueDeclare(queueOutputName, true, false, false, null);
if(queueOkStatus.getMessageCount()==0){//your logic here}`
but be careful, because this method returns number of non-acked messages in queue.
If you want to be aware of this , you can check Q count before inserting. It sends request on the same channel. Asserting Q returns messageCount which is Number of 'Ready' Messages. Note : This does not include the messages in unAcknowledged state.
If you do not wish to be aware of the Q length, then as specified in 1st comment of the question:
x-max-length :
How many (ready) messages a queue can contain before it starts to drop them from its head.
(Sets the "x-max-length" argument.)
I had declare a queue like below:
Map<String, Object> args = new HashMap<String, Object>();
args.put("x-max-length-bytes", 2 * 1024 * 1024); // Max length is 2G
channel.queueDeclare("queueName", true, false, false, args);
When the queue messages count bytes is large than 2G, It will auto remove the message on the head of the queue.
But what I expected is That it reject produce the last message and return exception to the producer.
How can I get it?
A possible workaround is check the queue size before send your message using the HTTP API.
For example if you have a queue called: myqueuetest with max size = 20.
Before send the message you can call the HTTP API in this way:
http://localhost:15672/api/queues/
the result is a JSON like this:
"message_bytes":10,
"message_bytes_ready":10,
"message_bytes_unacknowledged":0,
"message_bytes_ram":10,
"message_bytes_persistent":0,
..
"name":"myqueuetest",
"vhost":"test",
"durable":true,
"auto_delete":false,
"arguments":{
"x-max-length-bytes":20
},
then you cloud read the message_bytes field before send your message and then decide if send or not.
Hope it helps
EDIT
This workaround could kill your application performance
This workaround is not safe if you have multi-threading/more publisher
This workaround is not a very "best practise"
Just try to see if it is ok for your application.
As explained on the official docs:
Messages will be dropped or dead-lettered from the front of the queue to make room for new messages once the limit is reached.
https://www.rabbitmq.com/maxlength.html
If you think RabbitMQ should drop messages form the end of the queue, feel free to open an issue here so we can discuss about it https://github.com/rabbitmq/rabbitmq-server/issues
In a Mule flow, I would like to add an Exception Handler that forwards messages to a "retry queue" when there is an exception. However, I don't want this retry logic to run automatically. Instead, I'd rather receive a notification so I can review the errors and then decide whether to retry all messages in the queue or not.
I don't want to receive a notification for every exception. I'd rather have a scheduled job that runs every 15 minutes and checks to see if there are messages in this retry queue and then only send the notification if there are.
Is there any way to determine how many messages are currently in a persistent VM queue?
Assuming you use the default VM queue persistence mechanism and that the VM connector is named vmConnector, you can do this:
final String queueName = "retryQueue";
int messageCount = 0;
final VMConnector vmConnector = (VMConnector) muleContext.getRegistry()
.lookupConnector("vmConnector");
for (final Serializable key : vmConnector.getQueueProfile().getObjectStore().allKeys())
{
final QueueKey queueKey = (QueueKey) key;
if (queueName.equals(queueKey.queueName))
{
messageCount++;
}
}
System.out.printf("Queue %s has %d pending messages%n", queueName, messageCount);