Does RabbitMQ do round robin distribution of messages in following situation ?
RabbitMQ configuration :
Exchange type - Topic
Routing key - notify.#
Producer is pushing the message to above exchange with following
different topics-
notify.log.# , notify.status.#, notify.priceChange.#
There are 4 consumers are running on different servers.
3 consumers are doing the same processing and running on different instance of same application under a load balancer.(They want to consume every message pushed by producer)
They are listening to "notification_queue" attached with routing key "notify.#".
Remaining 4th consumer is doing some different processing than first 3 consumers and running on different server(of different application).
It is listening to "log_queue" attached with routing key "notify.log.#".
Will RabbitMQ distribute message in round robin fashion in first 3 consumers(in point 1)?
Please help.
Yes indeed , does not matter what is the point of a origin of consumer . When we have multiple consumers on the same queue then the distribution is round robin , you can try it out in the simulator here http://tryrabbitmq.com/
Related
After reading documentation about what is Rabbit and what does, I have some common questions:
Case is: Producer sends one message to some consumers (subscribers).
I need a short explanation for all points of list below, what to use, and what to dig further.
How to clear queue and stop sending message to consumers after
specific time/date?
Can I include to confirmed message user's data like JSON?
Where is stored this data? In the same queue?
How to filter confirmed messages and then clear queue?
How to clear queue after specific time/date?
What happens if not one consumer no confirms message, how long they are stored?
Does consumer subscribe on queue or can subscribe on exchange too?
Using model one to many for sending message, how to set who have to
get message first/last or at the same time, here described that, but not clear is it on client or server side?
If no consumers, how to re-push message to another queue and close
current?
Each consumer has own queue?
Thank you in advance and any comment to this question!
If you can elaborate some of your questions and include what is your use case, I can edit the answer.
1 - As long as consumer is alive rabbitmq sends incoming messages to consumer. You can give TTL to messages/queues if you want them to expire after some time.
https://www.rabbitmq.com/ttl.html
2 - What you mean?
3 - Rabbitmq stores the data in mnesia database.
https://www.rabbitmq.com/persistence-conf.html
https://www.rabbitmq.com/relocate.html
4 - What you mean by filterig messages and clear queue? Successfully consumed messages removed from the queue immediatly.
5 - You can give ttl to queue or declare queue as auto delete
https://www.rabbitmq.com/ttl.html
https://www.rabbitmq.com/queues.html
6 - If consumers don't send ack to rabbit, messages stays unack as long as memory becomes full or rabbit becomes unavailable
7 - Both. A consumer can create its own queue and bind it to an exchange or it can consume from existing queue. It depends on the use case.
8 - It is hard to answer this without knowing details of what you mean by one-to-many. Direct exchange or fanout or whatelse, how many queues etc.
But, in rabbitmq, messages ordered by publish order by default.
According to link you shared, rabbitmq sends messages first to higher priority consumers until consumer prefetch count (unack messages on consumer) becomes its limits.
9 - You need to handle this case in the code. Or you can use management ui with Shovel plugin.
https://www.rabbitmq.com/management.html
https://www.rabbitmq.com/shovel.html
10 - Again, it depends on the design and use case.
Here's my current architecture
I have a bunch of IoT devices, that connects through raw duplex persistent TCP to 1 instances of my "worker" that is connected to a RabbitMQ Queue
My publisher publishes some messages that look like that
{
"iot_device_name" : "A",
"command" : "reboot"
}
The worker is then able to map the iot_device_name to the TCP socket.
All is working nice, but if we want to add HA and to scale out a bit, it would be better to have 4 instances of the worker. Load balancing the TCP question is not a problem (with HaProxy or Nginx).
Now the problem is on how to split the load on the Queue part, as the list of IoT devices handled by a worker is dynamic (i.e a device could disconnect and reconnect to an other worker)
So is there a way for a worker to say: "Hmmm no I can't treat this message because I don't know this device, give me an other" so that an other worker can then take it and handle it ?
Other information that may be of help:
the workers are all in the same network, that is also the same than the publisher
the numbers of workers is not dynamic and even if we extrapolate the number of devices for the next years, 8 workers would takes us VERY FAR, as it simply route message/transcode messages, so their cpu load is ridiculous.
So if I understand your architecture correctly, you have commands sent to your publisher on one side, which are pushed into rabbitmq.
On the consumer side, you have multiple workers, to which the messages are dispatched, and each worker has a bunch of devices connected to it.
If indeed this is your architecture, I'd propose the following for your rabbitmq configuration:
use a direct exchange
each worker has it's own queue (exclusive), and manages the bindings between the exchange and its queue dynamically:
each time a device connects to a worker, that worker adds a binding between its queue and the exchange with as routing key the identifier for the device
each time a worker detects that a device is not connected to it anymore, he removes the related binding from the rabbitmq configuration
related to the detection of disconnected devices, I'd expect it common that it's upon receiving a command to push to the device that a worker realize the device isn't connected to it anymore, in such cases in addition to adapting the bindings, the worker would republish the message to the same exchange with the same routing key, so that it can have another shot at being consumed by the proper worker
I'd also consider configuring a TTL on the queues, no point in consuming a message that's too old
The publisher will of course also need to alter the message, including the intended device identification as routing key
I hope the proposal here makes sense, there are a few other cases to be considered: alternate exchange to make sure we don't lose requests if there is a (short) period when the device hasn't reconnected to a worker and we get a command for it anyway, adding a property to a message republished to ensure we do not add an infinite loop in the system, ... but what is indicated above should be a reasonable starting point to achieve your goal
I'm trying to scale-out a RabbitMQ messaging system. The current system is very simple - the producer sends a message to a fanout exchange and the message is handle by multiple consumers - classic fanout routing .
I have multiple consumers from different types (e.g: one that print to screen, one that logs to file, one that saves to DB,...).
My challenge - i'm not sure what's the best way to scale-out the consumers. If i add other consumers from the same type - i'll get double logs or double entries in the DB. ... (think about two DB consumers consuming from the same fanout exchange) .
I guess I can create a consumer that publish to a work-queue but I wonder if there's a better "builtin" solution in rabbitmq.
thanks in advance,
zf
If you need to scale consumers in order to consume faster all the messages coming from the fanout exchange you need competive consuming; so you need more consumers attached to the same queue bound to the fanout exchange.
In this way every consumer will consume a batch of messages indipendently from the others. The number of messages inside the batch is defined with the prefetch count property ( http://www.rabbitmq.com/consumer-prefetch.html ).
In this way, in your case, you should be able to scale consumers avoiding double logs and double entries in the DB.
I have a scenario as below
Exchange is of type "topic" and i have two queues(queue1 and queue2) binded to it with routing key #.
Each queue is having 1 consumer. Now, when i send a message from publisher both queues are getting my message since its binded to routing key # .
Is there any way, we can send messages to only one queue on round robin basis ?
Since, each message is getting into both queues , i am getting duplication of messages.
If you are looking for round robin consuming you shoud see this pattern:
https://www.rabbitmq.com/tutorials/tutorial-two-java.html
So basically you publish to one queue and then you add consumers to the same queue.
through the QoS = 1 you have the round-robin. In this way you can add more consumers without create new queues
I have been using `basicGet` method of RabbitMQ to fetch messages from MQ.
everything is working as expected. However when multiple instances of `JVM` are running i.e when multiple worker servers are present , the same message is being sent to all the worker servers (JVM).
I have used a "direct" exchange on the `Rabbit MQ configurations`.
**Sample code :**
basicGet("SampleQueue", true);
Please help.
**Scenario :** I have 4 messages present in the Rabbit MQ. I have two worker servers defined, which when executed should fetch 2 messages each (2 for each worker server). But currently, I am seeing that the two worker servers is fetching all the messages i.e 4 each.
I would like to ensure that the messages are being sent to Worker servers in a round robbin manner.
So Ideally, the Rabbit mq should not post duplicate messages to all worker servers.
I am using Spring AMQP implementation. The below snippet is being executed in a Spring Job which extends ItemReader.
**Sample code :**
GetResponse message = ApplicationContextProvider.getChannel().basicGet(
"SampleQueue", true);
**Here is my Queue configuration :**
<rabbit:connection-factory id="connectionFactory"
host="host" username="guest" password="guest" />
<rabbit:admin connection-factory="connectionFactory" />
<rabbit:template id="template"
connection-factory="connectionFactory" exchange="Sample" />
The Exchange defined in Rabbit MQ is :
Type : direct
durable : true
name : Sample
The Queue defined in Rabbit MQ :
name :Sample Queue
Binding :
Sample(Exchange) binded to SampleQueue (Queue) with routingKey (Sample).
Thanks.
Looking at the RabbitMQ routing tutorial page I think the Multiple bindings section explains what you are experiencing.
Multiple bindings
It is perfectly legal to bind multiple queues with the same binding
key. In our example we could add a binding between X and Q1 (Q2)
with binding key black. In that case, the direct exchange will behave
like fanout and will broadcast the message to all the matching
queues. A message with routing key black will be delivered to both Q1
and Q2.
If you would like a round-robin kind of dispatch then each worker should subscribe to the same queue like this
That is the concept of work queues. The RabbitMQ page has an excellent tutorial about this concept.
But if you would still like to use a direct exchange then each worker would have to bind its own queue to the exchange with a different routing key.
The routing algorithm behind a direct exchange is simple - a message
goes to the queues whose binding key exactly matches the routing key
of the message.
In this setup, we can see the direct exchange X with two queues bound
to it. The first queue is bound with binding key orange, and the
second has two bindings, one with binding key black and the other one
with green.
In such a setup a message published to the exchange with a routing key
orange will be routed to queue Q1. Messages with a routing key of
black or green will go to Q2. All other messages will be discarded.