I am using RabbitMQ to send a message to 2 or more consumers.
Looking at the simple tutorial I can achieve this very easily.
But I also want to scale some of my consumers and allow some consumers to always process the message and some consumers to only process the message if it has not been processed already by another 'similar' consumer.
|-------> cA
|
P --|-------> cB'
|
|-------> cB"
In my example above:
cA will always process the message, regardless.
cB' will only process the message if cB" has not processed the message already.
And cB" will only process the message if cB' has not processed the message already.
The idea is to process data as fast as possible and, (looking at the picture above), if cB' or cB" need to contact the DB or an API, I want another consumer to pickup the data off the queue as fast as possible.
But I also do not want that data to be picked up more than once.
I could always add another publisher and have the messages removed off the queue, but I am not sure if this is really the most efficient way to achieve what I am after.
|-------> cA
|
P --|-------> p'--|-------> cB'
|
|-------> cB"
Is this something that could be achieved in RabbitMQ?
If I understood your question correctly a setup with 2 queues will solve your problem.
Both queues have to be bound to the exchange(X) with the same routing key.
If you subscribe the queues in the following pattern:
cA as the sole subscriber of queue1
cB' and cB" sharing queue2
Routing and Subscription Overview:
|--<routing key>---queue1-----cA
P---X---|
| |--cB'
|--<routing key>---queue2--|
|--cB"
The message will:
always be process by cA and
in a round robin manner either by cB' OR cB" (in default queue configuration). You can configure the behavior how the load is shared between cB' and cB". See https://www.rabbitmq.com/tutorials/tutorial-two-dotnet.html for further details.
Read Consumer Priorities
And Also Dead Letter Exchanges
Related
Is it possible to implement the aggregator pattern in RabbitMQ?
I have A … N messages that I need to wait for/aggregate before sending off to another queue X.
So I'm thinking I will have some kind of unique ID that ensures that messages are routed exclusively to the same consumer and then wait for all the messages to arrive.
Is this possible in RabbitMQ?
Yes, it is possible.
But this is not RabbitMQ specific. Understanding what to aggregate and how to do it is beyond the responsibility of the message broker.
You need to write a service to subscribe to the relevant messages and then publish the result back. In the context of RabbitMQ, you could use routing keys to ensure the right consumer gets the messages, but that is not the only way.
I have implemented the example from the RabbitMQ website:
RabbitMQ Example
I have expanded it to have an application with a button to send a message.
Now I started two consumer on two different computers.
When I send the message the first message is sent to computer1, then the second message is sent to computer2, the thrid to computer1 and so on.
Why is this, and how can I change the behavior to send each message to each consumer?
Why is this
As noted by Yazan, messages are consumed from a single queue in a round-robin manner. The behavior your are seeing is by design, making it easy to scale up the number of consumers for a given queue.
how can I change the behavior to send each message to each consumer?
To have each consumer receive the same message, you need to create a queue for each consumer and deliver the same message to each queue.
The easiest way to do this is to use a fanout exchange. This will send every message to every queue that is bound to the exchange, completely ignoring the routing key.
If you need more control over the routing, you can use a topic or direct exchange and manage the routing keys.
Whatever type of exchange you choose, though, you will need to have a queue per consumer and have each message routed to each queue.
you can't it's controlled by the server check Round-robin dispatching section
It decides which consumer turn is. i'm not sure if there is a set of algorithms you can pick from, but at the end server will control this (i think round robin algorithm is default)
unless you want to use routing keys and exchanges
I would see this more as a design question. Ideally, producers should create the exchanges and the consumers create the queues and each consumer can create its own queue and hook it up to an exchange. This makes sure every consumer gets its message with its private queue.
What youre doing is essentially 'worker queues' model which is used to distribute tasks among worker nodes. Since each task needs to be performed only once, the message is sent to only one node. If you want to send a message to all the nodes, you need a different model called 'pub-sub' where each message is broadcasted to all the subscribers. The following link shows a simple pub-sub tutorial
https://www.rabbitmq.com/tutorials/tutorial-three-python.html
I am new to RabbitMQ and this is confusing me. I have setup a direct exchange and different queues subscribe to different routing keys on this exchange. What I want is that if a message gets published with one routing key, it gets consumed from one subscriber only, no matter how many queues subscribe to that routing key.
Current scenario:
Exchange (type: direct)
-QueueA1 (receives message A from exchange with routing key of "TypeA")
-QueueA2 (also receive message A from exchange with routing key of "TypeA")
-QueueB (doesn't receive message A because it subscribes to key "TypeB")
Desired:
-QueueA1 (receives message A from exchange with routing key of "TypeA")
-QueueA2 (doesn't receive message A because it's already consumed by QueueA1)
-QueueB (doesn't receive message A because it subscribes to key "TypeB")
Do I need to use a different exchange? How do I achieve desired scenario?
You can achieve what you want by using a single QueueA with multiple consumers subscribed to this queue:
Direct exchange
|
|-- ["TypeA"]--> QueueA
| |-- Consumer A1
| `-- Consumer A2
|
`-- ["TypeB"]--> QueueB
In this case, a message queued on QueueA will be delivered to only one consumer. The consumer which gets the message is undefined however: they are picked in a round-robin manner.
w/ RabbitMQ, all routing keys that match within a given exchange, will have a copy of the message delivered to the specified queue.
So, in your scenario, you will always have QueueA1 and QueueA2 receive messages of TypeA. This is how routing keys work. There is no way around this, using a single exchange.
If you need QueueA1 and QueueA2 to receive different messages, then you either:
need to use different routing keys to bind the queue, or
need to use a different exchange
Regarding the suggestion from Jean-Sebastient...
This scenario will allow either Consumer A1 or Consumer A2 to handle the message in question, but does so by having them both subscribe to the same queue.
If you're trying to ensure there is one 1 consumer that receives the message, then this is what you want to do. But, if you need both queues and need to ensure only one queue receives the message, then you need to go with one of the other options I suggested.
Finally, if you are looking at guaranteeing a message is processed only once, none of these options will do that.
On the surface, it will look like this works most of the time. But there will be scenarios where something goes wrong and you'll have more than one queue or consumer process the same message.
To handle this scenario, you need to look a "idempotence" in your message handling. This is most commonly handled with a database and an ID to say it has been processed already, but there are other solutions out there.
I have a rabbitmq cluster used as a working queue. There are 5 kinds of consumers who want to consume exactly the same data.
What I know for now is using fanout exchange to "copy" the data to 5 DIFFERENT queues. And the 5 consumers can consume different queue. This is kind of wasting resources because the data is the same in file queues.
My question is, does rabbitmq support to push the same data to multi consumers? Just like a message need to be acked for a specified times to be deleted.
I got the following answer from rabbitmq email group. In short, the answer is no... and what I did above is the correct way.
http://rabbitmq.1065348.n5.nabble.com/Does-rabbitmq-support-to-push-the-same-data-to-multi-consumers-td36169.html#a36170
... fanout exchange to "copy" the data to 5 DIFFERENT queues. And the 5 consumers can consume different queue. This is kind of wasting resources because the data is the same in file queues.
You can consume with 5 consumers from one queue if you do not want to duplicate messages.
does rabbitmq support to push the same data to multiple consumers
In AMQP protocol terms you publish message to exchange and then broker (RabbitMQ) decide what to do with messages - assume it figured out the queue message intended for (one or more) and then put that message on top of that queue (queues in RabbitMQ are classic FIFO queues which is somehow break AMQP implementation in RabbitMQ). Only after that message may be delivered to consumer (or die due to queue length limit or per-queue or per-message ttl, if any).
message need to be acked for a specified times to be deleted
There are no way to change message body or attributes after message being published (actually, Dead Letter Exchanges extension and some other may change routing key, for example and add,remove and change some headers, but this is very specific case). So if you want to track ack's number you have to re-publish consumed message with changed body or header (depends on where do you plan to store ack's counter, but headers fits pretty nice for this.
Also note, that there are redeliverd message attribute which denotes whether message was already was consumed, but then redelivered. This flag doesn't count redelivers number so it usage is quite limited.
I'm seeking some advise as to how best configure my rabbitMQ exchanges.
I'm trying to use a topic exchange in a round robin methodology. Each consumer has its own (uniquely) named queue attached to a topic exchange. I would like the exchange to round robin messages to each consumer queue for the "same" topic - lets say *.log for example.
I have tried multiple combinations and only seem to be able to simultaneously deliver messages to the consumer queues which effectively means I'm processing the message twice, once in each consumer.
For clarity I also have a fanout exchange, which I use to "control" the consumers (start, stop etc).this should remain in place in any outcome.
Any guidance on how best to achieve the stated outcome would be great.
Each consumer has its own (uniquely) named queue attached to a topic exchange
The trick is to have every worker/consumer that you want to round-robin between to setup a named queue and all use the same queue instead creating their own.
So you could create a named queue called "log" for all of the "log" workers. You would create a different named queue for say "foo" for all of the "foo" workers. Requests will be delivered round-robin to all consumers looking at the same queue.
To use RabbitMQ in round robin fashion, better to use direct exchange instead of topic exchange.
A direct exchange is ideal for the unicast routing of messages (although they can be used for multicast routing as well).
A queue binds to the exchange with a routing key K.
When a new message with routing key R arrives at the direct exchange, the exchange routes it to the queue if K = R.
Direct exchanges are often used to distribute tasks between multiple workers in a round robin manner. When doing so, it is important to understand that, messages are load balanced between consumers and not between queues.