I have a http server which receives some messages and must reply 200 when a message is successfully stored in a queue and 500 is the message is not added to the queue.
I would like rabbitmq to refuse my messages when the queue reach a size limit.
How can I do it?
actually you can't configure RabbitMq is such a way. but you may programatically check queue size like:
`DeclareOk queueOkStatus = channel.queueDeclare(queueOutputName, true, false, false, null);
if(queueOkStatus.getMessageCount()==0){//your logic here}`
but be careful, because this method returns number of non-acked messages in queue.
If you want to be aware of this , you can check Q count before inserting. It sends request on the same channel. Asserting Q returns messageCount which is Number of 'Ready' Messages. Note : This does not include the messages in unAcknowledged state.
If you do not wish to be aware of the Q length, then as specified in 1st comment of the question:
x-max-length :
How many (ready) messages a queue can contain before it starts to drop them from its head.
(Sets the "x-max-length" argument.)
Related
I'm trying to understand how rabbitmq works with multiple consumer and prefetch_count.
I have three consumers consuming on the same queue and all of these consumers have configured with the QoS prefetch_count = 200.
Now assuming at a certain point I have unlimited backlog messages in the queue and consumers A,B,C are connecting to the queue, would A get message 1-200, B get 201-400, C get 401-600 from the queue simultaneously? That seems like message 1, 201, 401 got processed at the first place compared to the rest. Somehow I don't want that, I'd like to have these messages being processed sequentially.
If that's the case I guess this implies that the messages may be processed disordered based on how consumers are setup, even though the queue follows FIFO.
Or should I set prefetch_count = 1 to make sure of REAL FIFO?
Edited:
Just set up a local env of rabbitmq and experimented a bit. I used a producer to bombard a queue with numbers 0 to 100000 sequentially to accumulate backlog messages in a queue. Later on, I had two consumers A, B consuming messages from that queue with prefetch_count = 200.
From what I observed, A got 0-199 and B got numbers 200-399 at very beginning. However, A started getting numbers {401, 403, 405, 406 ...} and B gets {400, 402, 404, ...} after that.
I guess A and B got non-skipped messages at the beginning was because I wasn't strictly spinning up these two consumers simultaneously. But the following pattern explains well how prefetch_count works. It doesn't necessarily send consumers consecutive messages(I knew it's processed in a round robin fashion, but I guess this is more intuitive with an experiment). There's no guarantee in what order the messages will be processed if using prefetch_count.
I am following this guide- https://spring.io/guides/gs/messaging-jms/
I have few messages with higher priority that needs to be sent before any other message.
I have already tried following -
jmsTemplate.execute(new ProducerCallBack(){
public Object doInJms(Session session,MessageProducer producer){
Message hello1 =session.createTextMessage("Hello1");
producer.send(hello1, DeliveryMode.PERSISTENT,0,0); // <- low priority
Message hello2 =session.createTextMessage("Hello2");
producer.send(hello1, DeliveryMode.PERSISTENT,9,0);// <- high priority
}
})
But the messages are sent in order as they are in the code.What I am missing here?
Thank you.
There are a number of factors that can influence the arrival order of messages when using priority. The first question would be did you enable priority support and the second would be is there a consumer online at the time you sent the message.
There are many good resources for information on using prioritized messages with ActiveMQ, here is one. Keep in mind that if there is an active consumer online when you sent those messages then the broker is just going to dispatch them as they arrive since and the consumer will of course process them in that order.
I am trying to code a simple consumer using librabbitmq. It is working, but when I do execute amqp_basic_consume, it consumes the entire queue.
What I want is for it to get a single message, process it and repeat.
I tried using a basic_qos to have the consumer prefetch 1 at a time, but that seems to have no effect at all.
The basic setup and loop:
// set qos of 1 message at a time
if (!amqp_basic_qos(conn, channel, 0, 1, 0)) {
die_on_amqp_error(amqp_get_rpc_reply(conn), "basic.qos");
}
// Consuming the message
amqp_basic_consume(conn, channel, queue, amqp_empty_bytes, no_local, no_ack, exclusive, amqp_empty_table);
while (run) {
amqp_rpc_reply_t result;
amqp_envelope_t envelope;
amqp_maybe_release_buffers(conn);
result = amqp_consume_message(conn, &envelope, &timeout, 0);
if (AMQP_RESPONSE_NORMAL == result.reply_type) {
strncpy(message, envelope.message.body.bytes, envelope.message.body.len);
message[envelope.message.body.len] = '\0';
printf("Received message size: %d\nbody: -%s-\n", (int) envelope.message.body.len, message );
if ( strncmp(message, "DONE",4 ) == 0 )
{
printf("XXXXXXXXXXXXXXXXXX Cease message received. XXXXXXXXXXXXXXXXXXXXX\n");
run = 0;
}
amqp_destroy_envelope(&envelope);
}else{
printf("Timeout.\n");
run = 0;
}
}
I expect to have a queue filled that I can start processing and if I hit ^C, the remaining messages are still in the queue. Instead, even if I have only processed one message, the entire queue is emptied.
This is the behavior when noAck is true. What will happen is that the messages will be pushed to the connected consumer as fast as the broker can send them, because it assumes that the consumer is able to accept them as they are acknowledged immediately upon delivery.
You would want to change noAck to false, then explicitly ack each message back to the broker in this case.
Alternatively, you could use a basic.get to pull messages from the broker one at a time as opposed to using a push-based consumer (there are folks out there who don't like this idea). Your use case will determine what is most appropriate, but based on the fact that you seem to have a full queue and fairly process-intensive messages, I would assume a basic.get would be just fine in this scenario. The question then would be to decide how often to poll when the queue is empty.
I had declare a queue like below:
Map<String, Object> args = new HashMap<String, Object>();
args.put("x-max-length-bytes", 2 * 1024 * 1024); // Max length is 2G
channel.queueDeclare("queueName", true, false, false, args);
When the queue messages count bytes is large than 2G, It will auto remove the message on the head of the queue.
But what I expected is That it reject produce the last message and return exception to the producer.
How can I get it?
A possible workaround is check the queue size before send your message using the HTTP API.
For example if you have a queue called: myqueuetest with max size = 20.
Before send the message you can call the HTTP API in this way:
http://localhost:15672/api/queues/
the result is a JSON like this:
"message_bytes":10,
"message_bytes_ready":10,
"message_bytes_unacknowledged":0,
"message_bytes_ram":10,
"message_bytes_persistent":0,
..
"name":"myqueuetest",
"vhost":"test",
"durable":true,
"auto_delete":false,
"arguments":{
"x-max-length-bytes":20
},
then you cloud read the message_bytes field before send your message and then decide if send or not.
Hope it helps
EDIT
This workaround could kill your application performance
This workaround is not safe if you have multi-threading/more publisher
This workaround is not a very "best practise"
Just try to see if it is ok for your application.
As explained on the official docs:
Messages will be dropped or dead-lettered from the front of the queue to make room for new messages once the limit is reached.
https://www.rabbitmq.com/maxlength.html
If you think RabbitMQ should drop messages form the end of the queue, feel free to open an issue here so we can discuss about it https://github.com/rabbitmq/rabbitmq-server/issues
I need to set message priority so that High priority messages are consumed before Low priority messages by Receivers.
First I tried with message.setJMSPriority() method to set the priority but it was not working in HornetQ and ActiveMQ so finally I set the priority of the Message Producer using setPriority() method and it works fine now.
Why isn't Messsge.setJMSPriority() working in any of the JMS vendor implementations and why do we need to set the priority of the Producer not the message itself to set the priority of the message? What is the use of Messsge.setJMSPriority() method then?
Any suggestion or comment is appreciated.
To answer this question all you need to do is read the API docs for the setJMSPriority method and it tells you why. Here's the relevant text.
Sets the priority level for this message.
JMS providers set this field when a message is sent. This method can be used to change the value for a message that has been received.
The JMS Provider (ActiveMQ, HornetMQ, etc) set the priority in the producer on send to either the default value of 4, or to whatever value you've set the producer to use, so setting the value before send on the message itself won't have any effect.
The following will not work:
msg.setJMSPriority(9); // Not working!
In this code, the message priority is set to 9, indicating this is a high-priority message.
However, when the message is sent, the message will have a priority of 4 (normal
priority). The reason? Like the message expiration, the JMS provider will look at the
message priority property on the sender, or on the send(..) invocation, and then invoke the setJMSPriority on the message method prior
to placing the message on the queue. Since the default message priority is 4 (normal
priority), the message priority will not be set to a high priority message, as the developer had originally intended.
Like the message expiration, there are two ways of setting the message priority: you
can invoke the setPriority() method on the MessageProducer (QueueSender or Topic
Publisher) or set the message priority when sending the message:
//set the default message priority for all messages to 9 (high)
QueueSender qSender = qSession.createSender(requestQ);
qSender.setPriority(9);
qSender.send(msg1);
//this message is low priority
qSender.send(msg2, DeliveryMode.PERSISTENT, 1, 30000);
In this example, msg1 will be sent with a priority of 9 (high priority), whereas msg2 will
be sent with a priority of 1 (low priority).
This is a JMS Specification requirement.
You should change the priority on the Message Producer.
You can read JmsTemplate http://static.springsource.org/spring/docs/3.0.6.RELEASE/spring-framework-reference/html/jms.html
Some JMS providers allow the setting of default QOS values administratively through the configuration of the ConnectionFactory.
Check isExplicitQosEnabled property.