Let me explain what I am trying to achieve here:
Create 5 copies of same service each listening to a queue specific to them. The message they listen to would be same
SVC1 listening to Q1
SVC2 listening to Q2
SVC3 listening to Q3
SVC4 listening to Q4
SVC5 listening to Q5
Say they all listen to a message called TestMessage.
Do a Round Robin load balancing between these 5 services and drop the message to the applicable queue based on the output of my round robin logic
My question how do I configure to drop TestMessage to one queue at a point of time.
Thanks in advance
Please take a look at the built-in Distributor. This will perform the load balancing for you and take care of all the work distribution.
The Distributor is what you need to use. It has its own input queue and a queue for available workers. Everytime a worker is available, it will place a message in the distributors input queue and the distributor will send it the next message in its input queue.
Below is a sample application by Mikael Koskinen that demonstrates how to use this:
http://mikaelkoskinen.net/nservicebus-distributor-sample-application/
Related
Does RabbitMQ do round robin distribution of messages in following situation ?
RabbitMQ configuration :
Exchange type - Topic
Routing key - notify.#
Producer is pushing the message to above exchange with following
different topics-
notify.log.# , notify.status.#, notify.priceChange.#
There are 4 consumers are running on different servers.
3 consumers are doing the same processing and running on different instance of same application under a load balancer.(They want to consume every message pushed by producer)
They are listening to "notification_queue" attached with routing key "notify.#".
Remaining 4th consumer is doing some different processing than first 3 consumers and running on different server(of different application).
It is listening to "log_queue" attached with routing key "notify.log.#".
Will RabbitMQ distribute message in round robin fashion in first 3 consumers(in point 1)?
Please help.
Yes indeed , does not matter what is the point of a origin of consumer . When we have multiple consumers on the same queue then the distribution is round robin , you can try it out in the simulator here http://tryrabbitmq.com/
Is there is a way by which we can restrict RabbitMQ Queue to dispatch only a fixed number of messages from the Queue to the consumers?
I have 2 Queues Q1 and Q2 and 10 consumers.Every consumer can process the messages from Q1 and Q2.At any given time, only 2 consumers should process messages from Q2.All the 10 consumers can process message from Q1 simultaneously.
Is there any configuration in RabbitMQ which we can specify, so that RabbitMQ pushes only 2 messages from Q2 to any free consumer and push the next 2 only after they are acknowledged, even though other consumers are free and ready to consume.
More background on the issue:
Why only process 2 messages at a time ? :
Q2 messages are doing a web service call and the web service end point(third party) can only service 2 messages concurrently.
Cant we use concurrency ? :
If we use a ListenerContainer (Spring AMQP) the container is per consumer. We can restrict how many message one consumer can take at a time, but when we have 10 consumers, if there are messages in the Queue, each consumer will get its share.
Can we configure only 2 consumers listening to Q2 ? :
I understand we can achieve this by configuring only 2 consumers for Q2, but I am trying to avoid that. If for some reason these 2 consumers goes down, the processing of Q2 will be halted. If 10 consumers are configured, we can guarantee the processing will happen until the last consumer is down.
Looking to see if there is some config in RabbitMQ which we can make use of or any suggested solution.
Thanks in advance !
I'm pretty sure that consumer prefetch will accomplish what you want. But, Q2 can only have one consumer for this to work. There is no way to coordinate among multiple consumers - you would have to do that yourself, and could use RabbitMQ to do the coordination.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
I think you're getting wrapped up in the problem definition. What you really need is trivial, so let's break this down a bit.
Given two queues, Q1 and Q2
10 consumers
Every consumer can process the messages from Q1 and Q2.
At any given time, only 2 consumers should process messages from Q2.
All the 10 consumers can process message from Q1 simultaneously.
Comments on problem statement
First, queues are assumed to be indepenedent. An independent process P will have queue Q, thus Q1 serves process P1. This is a strict mathematical requirement - you cannot define two queues for a single process P.
Thus, the second constraint is mathematically incorrect, for the same reason that you could not write a valid function that accepts a parameter of type string and bool interchangeably. It must accept one or the other, as they are not compatible types, or it must accept a single common ancestor of the types without regard to the subtypes. This is a variant of the Liskov Substitution Principle.
Redefining the problem
There are a total of 12 consumers in the system:
Q1 has 10 consumers
Q2 has 2 consumers
[Important] Consumers are not shared between queues
Is there any configuration in RabbitMQ which we can specify, so that RabbitMQ pushes only 2 messages from Q2 to any free consumer and push the next 2 only after they are acknowledged, even though other consumers are free and ready to consume.
Based on the new definition of the problem, you have two options:
Use a Basic.Get - pull the next message from the queue as soon as the consumer finishes processing the last message.
Use consumer prefetch with limit 1. This will deliver the first and second messages for each consumer immediately, then deliver additional messages one at a time as the next message for that consumer is acknowledged. This is a bit more complicated, but might make sense if your latency margins are less than 10 milliseconds.
Note that by properly defining the problem space, we have eliminated the fundamental problem of trying to figure out how to ensure only two consumers are processing Q2 messages at any time.
try the new feature Single Active Consumer from version 3.8+.
Single active consumer allows to have only one consumer at a time consuming from a queue and to fail over to another registered consumer in case the active one is cancelled or dies.
Consuming with only one consumer is useful when messages must be consumed and processed in the same order they arrive in the queue.
Single active consumer can be enabled when declaring a queue, with the x-single-active-consumer argument set to true
https://www.rabbitmq.com/consumers.html#single-active-consumer
e.g. with the Java client:
After reading documentation about what is Rabbit and what does, I have some common questions:
Case is: Producer sends one message to some consumers (subscribers).
I need a short explanation for all points of list below, what to use, and what to dig further.
How to clear queue and stop sending message to consumers after
specific time/date?
Can I include to confirmed message user's data like JSON?
Where is stored this data? In the same queue?
How to filter confirmed messages and then clear queue?
How to clear queue after specific time/date?
What happens if not one consumer no confirms message, how long they are stored?
Does consumer subscribe on queue or can subscribe on exchange too?
Using model one to many for sending message, how to set who have to
get message first/last or at the same time, here described that, but not clear is it on client or server side?
If no consumers, how to re-push message to another queue and close
current?
Each consumer has own queue?
Thank you in advance and any comment to this question!
If you can elaborate some of your questions and include what is your use case, I can edit the answer.
1 - As long as consumer is alive rabbitmq sends incoming messages to consumer. You can give TTL to messages/queues if you want them to expire after some time.
https://www.rabbitmq.com/ttl.html
2 - What you mean?
3 - Rabbitmq stores the data in mnesia database.
https://www.rabbitmq.com/persistence-conf.html
https://www.rabbitmq.com/relocate.html
4 - What you mean by filterig messages and clear queue? Successfully consumed messages removed from the queue immediatly.
5 - You can give ttl to queue or declare queue as auto delete
https://www.rabbitmq.com/ttl.html
https://www.rabbitmq.com/queues.html
6 - If consumers don't send ack to rabbit, messages stays unack as long as memory becomes full or rabbit becomes unavailable
7 - Both. A consumer can create its own queue and bind it to an exchange or it can consume from existing queue. It depends on the use case.
8 - It is hard to answer this without knowing details of what you mean by one-to-many. Direct exchange or fanout or whatelse, how many queues etc.
But, in rabbitmq, messages ordered by publish order by default.
According to link you shared, rabbitmq sends messages first to higher priority consumers until consumer prefetch count (unack messages on consumer) becomes its limits.
9 - You need to handle this case in the code. Or you can use management ui with Shovel plugin.
https://www.rabbitmq.com/management.html
https://www.rabbitmq.com/shovel.html
10 - Again, it depends on the design and use case.
I am trying to find a way to load balance requests to a group of similar configuration activemq brokers.
Suppose, i have 3 activemq brokers(A, B, C), each having (say) two queues (X,Y) operating.
Every activemq broker has same queues ie. X & Y.
Every activemq broker queues have a dedicated consumer. which consumes messages only from their respective broker's queue.
Now, I want to load balance requests to my 3 queues load balanced in round-robin. i.e. the 1st request should go to Queue A, then next request to queue B and then next request to queue C and so on.
Any guidance please.
Check out ActiveMQ's Networks of Brokers feature: http://activemq.apache.org/networks-of-brokers.html. I'd use as much of ActiveMQ's built-in functionality as possible before coming up with any custom round-robin code of your own.
I have two subscribers listening to a publisher.All th queues are on the same machine.To make subscriber power off i deleted the input queue of one of the subsriber. I am getting one exception in the generic host command output and no meesages are there in the Outgoing Queues. Is this behaviour is coz all things are in the same machine ?
To bring down a subscriber, just kill the process - no need to delete the queue. The reason you don't see pending messages in the outgoing queue is exactly because the queue is local.