Are there any best practices of how to organize RabbitMQ exchanges/queues in microservices architecture?
Say I have services A, B, C, D. Service D needs to consume events from A and B, publish commands to C.
My solution - Every Service declares its exchange. ExchangeA, ExchangeB, ExchangeC, ExchangeD.
Clients can publish commands to that exchange and consume events from those exchanges. These are topic exchanges, so for command, the routing key would be ServiceA.Command.Command1, for event ServiceA.Event.Event1.
For the example I've shared:
ServiceD binds queue (with name ServiceD.ServiceA.Event.Queue) to ExchangeA with routing key: ServiceA.Event.#
ServiceD binds queue (with name ServiceD.ServiceB.Event.Queue) to ExchangeB with routing key: ServiceB.Event.#
ServiceD produces command to the exchange ServiceC with routing key ServiceC.Command.Command1
Is this the right approach? Are there some good examples I could take a look?
Microsoft example micro service project here: https://github.com/dotnet-architecture/eShopOnContainers
usually one Exchange and N queues are used.
exchange image
queues image
Your Exchange should not be service based but should be associated to the transaction which requires async processing of the messages. Create the Exchanges for those transactions and create and map your publishers to those exchanges. Create consumer queues and bind the queues to the exchange.
This would be the queue from where your other service will pull the message for processing. You one service can have 1:N exchanges.
You may have only one exchange and each services consumes from their own queues (1 queue per topic-service). All services publish to the same exchange.
Following your example:
ServiceD binds queue (with name ServiceD.ServiceA.Event.Queue) to
Exchange1 with routing key: ServiceA.Event.#
ServiceD binds queue
(with name ServiceD.ServiceB.Event.Queue) to Exchange1 with routing
key: ServiceB.Event.#
ServiceD produces command to the exchange Exchange1
with routing key ServiceC.Command.Command1
ServiceC binds queue (with name ServiceC.Commands.Queue) to
Exchange1 with routing key: ServiceC.Command.#
etc.
You may also have multiple exchanges, 1 exchange per object type (e.g. one exchange with the customer object events and commands, and another exchange with the order events and commands).
Related
I have multiple producers that publish to their specific (durable and limited) queues using the amq.direct exchange and particular routing key
Queues:
producer.06
producer.07
...
Routing keys:
"producer.06" -> producer.06
"producer.07" -> producer.07
...
I also have multiple consumers. When they connect, they create their own (exclusive) queue and routing keys to receive all the messages from the queues that are of interest to them. This way multiple consumers can get the same messages.
Queues:
consumer.a
consumer.b
...
Routing keys:
"producer.06" -> consumer.a
"producer.06" -> consumer.b
"producer.07" -> consumer.b
...
I would like to populate the consumer's queue with a snapshot of messages of the relevant producer's queues, prior to binding the routing keys. Loosing a few messages in the interval between the message copy and routing key binding is acceptable, and a better alternative than out-of-order messages for my application. The consumer should not remove messages from producer's queues (as they would be needed by other consumers).
Is there a way to achieve this? -copying a snapshot of a queue into another one- or does anyone has a suggestion on how to achieve this?
I am running RabbitMQ 3.8.4 on Erlang 23.0.2, and using Rabbit .Net client 6.0.0.0 for the consumers.
I have a situation when I need to create a route for my messages but I would like to use a matching pattern with negation, like !myPattern.
Example:
I have a queue bound in a Topic Exchange and the routing key is #.brazil.#. So it means that this queue will only receive messages when in the message's routing key contains ".brazil." like message.brazil.denmark.
Now I want to create another queue and bind to the same Topic Exchange but I want to receive all messages that don't contain the pattern #.brazil.#, something like !(#.brazil.#).
I was making some tests using Headers Exchange but the x-match argument only can have 2 possible values: any and all, and I need something like except.
Basically it is not possible to use negation in RabbitMQ even in routing key or header's attribute.
As far as I found out, there are 3 options here:
1 - Using an alternate exchange feature
Declare a fanout exchange you'll publish to (let's call it "my-exchange").
Declare a fanout exchange called "junk".
When each consumer declares a queue, it also declares a topic exchange
and a fanout exchange.
The alternate-exchange for the topic exchange should be set to the fanout exchange.
It then binds the topic exchange to "my-exchange", and "junk" to the topic
exchange, with a routing key equal to the topics it doesn't want.
Thus messages with the "bad" routing key go:
[my-exchange] -> [per-consumer-topic] -> [junk]
and the rest go:
[my-exchange] -> [per-consumer-topic] -> [per-consumer-fanout] ->
[per-consumer-queue]
Solution By: Simon MacMullen-2
Thread Reference: http://rabbitmq.1065348.n5.nabble.com/Binding-to-topic-exchange-with-a-negation-wildcard-td21964.html
2 - Using a Router Consumer
On this solution, you will have only 1 consumer bound in your queue, and the
unique responsibility of this consumer will be "redirect" the message to other
exchanges based on your rules.
Now your router logic will be centralized on this "orchestrator" and not in
RabbitMQ anymore (routing keys or header's attr).
3 - Using a Fanout Exchange
This solution is simple but has a huge drawback, scaling.
Basically you will have a Fanout Exchange responsible to deliver the message to
all bound queues and all consumers will receive the message and check if it
should process or discard the message, it means that now the "router logic"
will be on the consumer side.
The problem with this solution is if you want to scale a specific consumer and
your process is not idempotent you will process the message more than 1 time
(the number of instances running of your consumer).
So in my case, the best approach was the Router Consumer.
I am using a RabbitMQ topic exchange to route messages in a microservice environment. I understand how Queues work, but I'm uncertain of the intended relationship between a Queue and a subscription to a routing key. For this example, let's assume that each Queue will only ever have one connection (service) bound to it.
Should each service only have one queue on which every routing key sub is bound?
Should each queue only have one routing key bound to it?
Why do we need routing key to route messages from exchange to queue? Can't we simply use the queue name to route the message? Also, in case of publishing to multiple queues, we can use multiple queue names. Can anyone point out the scenario where we actually need routing key and queue name won't be suffice?
There are several types of exchanges. The fanout exchange ignores the routing key and sends messages to all queues. But pretty much all other exchange types use the routing key to determine which queue, if any, will receive a message.
The tutorials on the RabbitMQ website describes several usecases where different exchange types are useful and where the routing key is relevant.
For instance, tutorial 5 demonstrates how to use a topic exchange to route log messages to different queues depending on the log level of each message.
If you want to target multiple queues, you need to bind them to a fanout exchange and use that exchange in your publisher.
You can't specify multiple queue names in your publisher. In AMQP, you do not publish a message to queues, you publish a message to an exchange. It's the exchange responsability to determine the relevant queues. It's possible that a message is routed to no queue at all and just dropped.
Decoupling queue names from applications is useful for flexibility.
You could establish multiple queues to consume the same message, but queues can't have the same name.
In some cases, message's originator doesn't know the names of queues. (like when you have randomly generated queue names when horizontally scaling a server)
An exchange may be routing messages for more than just one type of consumer. Then you would need some wildcards in your routing keys to route messages to concerned consumers.
As far as I can tell, there is no proper use case for a direct exchange, as anything you can do with it you can do with a fanout exchange, only more expandably.
More specifically, in reading RabbitMQ in Action, the authors numerously refer to the use case that goes something like - "Suppose when a user uploads a picture you need to generate a thumbnail. But then later marketing also tells you to award points for uploading a photo. With RabbitMQ you just have to create another queue and do no work on the producer side!"
But that's only true if you've had the foresight to create a fanout exchange on the producer side. To my understanding a direct exchange cannot accomplish this and is only appropriate when you actually want tight coupling between exchange and queue, (which you don't, because that's the point of messaging systems.)
Is this correct or is there an actual use case?
Compared to the fanout exchange, the direct exchange allows some filtering based on the message's routing key to determine which queue(s) receive(s) the message. With a fanout exchange, there is no such filtering and all messages go to all bound queues.
So if you have a direct exchange with several queues bound with the same routing key, and all messages have this key, then you have the same behavior as the fanout exchange. This is better explained in tutorial 4 on the RabbitMQ website.
In the image upload use case, you can use:
a fanout exchange with two queues (one for the thumbnail worker, one for the score computation worker). The routing key is ignored.
fanout-exchange
|--> queue --> thumbnail-worker
`--> queue --> score-worker
a direct exchange with again two queues. Queues are bound with the image-processing key for instance, and messages with this key will be queued to both queues.
direct-exchange
|--["image-processing"]--> queue --> thumbnail-worker
`--["image-processing"]--> queue --> score-worker
Of course, in this situation, if the message's routing key doesn't match the binding key, none of the queues will receive the message.
You can't put the two workers on the same queue, because messages will be load balanced between them: one worker will see half of the messages.
Do you mean a fanout exchange or a topic exchange? a fanout exchange is very different from a direct exchange. I presume that sending the photo to the exchange is sent with a routing key that specifies that there is a photo. In which case you have a consumer that generates the thumbnail and when you want to add a new consumer you can just add it and get the same message but do something different with it, ie award points.
The use case holds up. I think the point is that the exchange is originally created as a direct exchange.
This answer echoes the previousone and if you refer to this page, I believe you'll that one particular use case described is:
Direct exchanges are often used to distribute tasks between multiple
workers (instances of the same application) in a round robin manner.