I am starting with ActiveMQ and I have a usecase. i have n producers sending messages into a Queue Q1. I want to stop the delivery of messages (i.e. i do not want consumers to consume those messages). I want to store the messages for sometime without those being consumed.
I was looking at ways this can be achieved. These two things came into my mind based on what i browsed through.
Using Mirrored queues, so that I can wiretap the messages and save into a virtual queue.
Possibly stop consumers from doing a PULL on the queue.
Another dirty way of doing this is by making consumers not send ack messages once its consumed a message from the queue.
We are currently not happy with either of these.
Any other way you can suggest.
Thanks in advance.
If you always want message delivery to be delayed you can use the scheduler feature of ActiveMQ to delay delivery until a set time or a fixed delay etc.
Other strategies might also work but it really up to you to design something that fits your use case. You can try to use Apache Camel to define a route that implements the logic of your use case to either dispatch a message to a Queue or send it to the scheduler for delayed processing. It all really depends on your use case and requirements.
Related
If I have two queues from which I want to consume messages, and I use a single SimpleMessageQueueListenerContainer for it, in which order would the listeners be invoked/messages consumed when both queues have messages?
I will try to be more specific of the problem I am working on:
I have a consumer application which needs to consume messages from 2 queues – say regular-jobs-queue and infrequent-jobs-queue. If there are any messages in ‘infrequent-jobs-queue’ I want to consume those before consuming messages from ‘regular-jobs-queue’. I might not be able to combine these and put all messages into a single rabbitmq level priority queue and assign higher priority to infrequent-job message because of some upcoming use-cases like purging regular-jobs without affecting infrequent-jobs and others.
I am aware that RabbitMQ has support for consumer priority but I am not very sure if it will be applicable here. I want all instances of my consumer application to first consume messages of infrequent-jobs-queue if any and not prioritize amongst these consumers.
Or should I like have 2 containers, with dedicated consumer thread(s) per queue and have an internal priority-queue data structure into which I can put messages as and when consumed from rabbitmq queue.
Any help would be really appreciated. Thanks.
~Rashida
You can't do what you want; messages will be delivered with equal priority.
Moving them to an internal in-memory queue will risk message loss.
You might want to consider using one of the RabbitTemplate.receive() or receiveAndConvert() methods instead of a message-driven container.
That way you have complete control.
In one of our applications the back pressure did not work and there was a huge pileup in a queue on RabbitMQ. This caused the RMQ node to choke.
Is there a way to apply flow control (manually) on that queue in such cases? That would have slowed down the producer and given us headroom.
In your case the consumers are not fast enough to handle the messages.
Basically you had a load-spike.
So, it does not mean that you need to stop the publishers.
You could:
Increase the number of the consumers
Use the Lazy queues
you didn't see the flow control because RabbitMQ could handle the messages.
I'm trying to achieve load balancing between different types of messages. I would not know in advance what the messages coming in might be until they hit the queue. I know I can try resequencing the messages, but I was thinking that maybe if there was a way to have the various consumers round robin between either queues or between topics, this would solve my problem.
The main problem i'm trying to solve is that I have many services sending messages to one queue with many consumers feeding off one queue. I do not want one type of service monopolizing the entire worker cluster. Again I don't know in advance what the messages that are going to hit the queue are going to be.
To try to clearly repeat my question:
Is there a way to tell the consumers to round robin between either existing queues or topics?
Thank you in advance.
I found the answer to my question on another post just had to know to look there. I resolved my problem by not creating AMQ consumer but a JMS listener with a composite destination as specified in this post: jms-listener-dynamically-choose-destinations. It turns out the JMS listener automatically round robins though all the queues you assign to it.
Consumers on a Queue will already do round robin processing of the messages on the Queue. The one thing to keep in mind is consumer prefetch which can allow one consumer to grab many messages before others arrive on the Queue so you may need to adjust prefetch depending on your scenario.
Read up on the differences between Queue and Topic here.
I have a somewhat unique use case with RabbitMQ and I'm not sure how to go about solving the problem. I want to have one queue with multiple consumers bound to it and then have RabbitMQ send out one message to only one consumer at at time and wait for an ACK before sending out another message to any other consumer.
I realize this kills throughput and can essentially starve the other consumers but for me that's OK. The reason for this odd use case is that the service that the consumers talk to can only handle one concurrent request at a time so I need a way to limit this but consumers can also die unexpectedly and I need another consumer to pick up processing the messages if this happens. I know there is the prefetch option but that still allows multiple users to get a and exclusive queues but I'm not sure those accomplish what I want. Is it possible configure RabbitMQ to do this?
No; there is no way to limit competing consumers on the same queue such that there is one and only one message in process across all consumers until the ack is received.
A similar question came up some time ago; I don't remember if it was here or in the Spring forums but I believe the solution was to have the consumers acquire a global lock of some kind, using something like hazelcast, or even a simple database table row lock (with prefetch=1 so each consumer had only one "in process" message which was processed as and when each one got the lock).
I have two questions about RabbitMQ Work Queues:
As I understand it from the RabbitMQ tutorials, it seems that if I have a basic queue consumer client (just a basic "Hello, World!" consumer) and then I add a second consumer client for the same queue, then RabbitMQ will automatically dispatch the messages between those two queues in a round robin manner. Is that true (without adding in any extra configuration)?
My consumer clients are configured to only ever receive one message at a time, using (GetResponse response = channel.basicGet("my_queue", false). Since I am only ever receiving one message at a time, is it still necessary to set a prefetchCount (channel.basicQos(1)) for fair dispatch?
Answers to your questions:
Yes
No
However, your two questions 1 and 2 are not compatible. If you are using a consumer, it is designed to have messages pushed to it, and you don't use Basic.Get. When you use a consumer, you will need to use Basic.QoS to specify that the consumer can only "own" one unacknowledged message at a time. RabbitMQ will not push additional messages beyond the QoS limit.
Your alternative is to "pull" from the queue using Basic.Get, and you will control your own destiny as far as how many messages you run at a time.
Does this make sense?