I have a scenario where publishers need to send messages to a known exchange (they won't know the downstream queues directly). Most of the messages should be delivered to every bound queue (like a fanout), but some need to be delivered only to a specific bound queue. I'd like to avoid multiple exchanges as well as multiple queue bonds.
My current solution uses header routing, where a message has one of two headers, identify it as "global" or "specific to a particular sub", where the downstream queues are bound on a match-any to both of those headers. I think this will work, but I feel like there should be a simpler solution.
I tried to find an exchange plugin that would "fanout all messages except specific ones", but I couldn't find such a plugin. Outside of that, any ideas on how to implement such a routing strategy?
For what it's worth, my original solution using a "headers" exchange with queues bound using "match any" is the only one I could find, short of writing a new exchange plugin. It does work and so far seems reasonably fast (at least not measurably slower than a typical "topic" exchange--which I could find no way to apply in this scenario).
From my research on this topic, the ideal solution would be a "topic" exchange with the ability to use a RegEx or at least some form of "or-logic". I did find some information that implied that a RegEx was considered but decided against (in favor of the "dotted notation" topic format) because the latter was faster, specifically because the use a a RegEx would require that every binding be evaluated on each new message (i.e. there was no way to construct a "search shortcut").
For now, my "match-any headers exchange" solution will serve my purposes, but in the future, a "topic" exchange that allowed "or-logic" might be worth exploring. It would allow multiple topic patterns to be achieved with single-binding, without the overhead of a RegEx. But I have no experience with Erlang, nor the time to learn enough of it to write the necessary plugin. Please contact me if anyone is interested in collaborating on this.
What you describe really sounds like exchange-to-exchange binding though, as rich routing is actually one of RMQ's strengths.
You could create the entry exchange that would point to either the fanout (for generic case) or topic/direct one (for the special one), and leave all routing to RMQ. The entry-exchange could be a header exchange or a direct one, depending on what you want to put in headers:
entry-exchange -----> fanout-exchange ---*---> multiple-fanout queues
\
\---> the special exchange --> queue-for-special-usecase
Related
I have read so many articles regarding exchanges in RabbitMQ.
But I couldn't find any useful article which explains about usecases of each exchange.
all they are saying is
Direct Exchange - Binding key and routing key should be same.
Topic Exchange - Routing key should match the routing pattern of the binding key.
Fanout Exchange - All the queues which are bounds to that particular exchange will get the message.
Header Exchange - headers should match the key.
can any one explain each exchange and usecases of it in detail?
Well, the choice of exchange type to use is pretty much driven by your use case, it your responsibility to choose the exchange type that suits you best.
Here are some examples:
Lets say you want a broadcast (you have multiple instances of the applications that are supposed to receive the message). For example, something happens in the system (resource becomes available, cache should be invalidated, you name it) and you want that all instances will know it. Then your first bet will be using fanout exchange.
Another example. You want to unicast: send a message and you have multiple consumers, but you want that only one consumer will get a message and will attempt to process it. In this case, you can't use fan-out exchange and you'll opt for other types of exchange (like direct exchange for example).
All-in-all I believe you should be interested to read this article that describes various concepts of fairly rich amqp protocol and provides use case examples for different types of exchanges.
Can anyone explain in which cases I need to create multiple queues (one user -> one queue name), and when one queue name for all clients with different routing keys (one user -> one routing key) and why?
A user should not be able to read messages intended for another user.
I'm using direct exchange type.
First off I am going to assume that when you say "user" you are interchangeably referring to a consumer or producer, and they aren't the same thing so I would read up on that here in rabbitmq's simplest explanation. Walking through that tutorial will definitely help solidify your understanding of rabbit a bit more overall too, which is always good.
In any case, I would recommend doing this:
Create multiple queue's, each one linked to a single consumer. The reason for doing this instead of using a single queue with multiple is discussed here but if you don't want a bunch of programmer jargon, it pretty much says that a single queue is super slow because only one message can be consumed at a time from the queue.
Also, there is a built in "default exchange" that you can use instead of setting up another direct exchange which it sounds like you're putting effort into that you might not need to, obviously I'm not sure what you are doing but I would take that into consideration... hope this helps!
As per my understanding, all the functionality or use-cases of direct and fanout exchanges can be achieved using topic exchange. Topic exchange supports superset of functionality. So the question is, why does RabbitMQ have direct and fanout exchanges? Are there any use-cases that can be achieved using direct/fanout exchange but not with topic exchange?
I would say it's a matter of simplification. If all you need is, say, a topology where each specific routing key maps 1:1 with a queue (round robin distribution of tasks between multiple workers within the same application is an example cited in the RabbitMQ doc), then Direct exchanges may be easier to use and work with, and all you need. That's not to say you couldn't accomplish the same thing using Topic Exchanges (you could).
Similarly, with Fanout exchanges, if you have a situation that calls for the simply broadcasting of messages, you may find a Fanout exchange is easier to work with. Again, that's not to say you couldn't accomplish the same thing using Topic Exchanges (you could).
I generally use Topic exchanges exclusively, because I value the flexibility they provide. As an application expands, they can handle a greater variety of use cases within the same exchange, whereas that may not be the case with the other two types. Thus I can avoid the potential for having to change the topology midstream as an application grows.
As the RabbitMQ doc states about Topic exchanges:
Topic exchanges have a very broad set of use cases. Whenever a problem
involves multiple consumers/applications that selectively choose which
type of messages they want to receive, the use of topic exchanges
should be considered.
For more information about these concepts, including illustrations, this page has quite a bit of info:
https://www.rabbitmq.com/tutorials/amqp-concepts.html
You could replace the functionality of direct and fanout with a topic exchange, but you could also implement a big "Animal" class and don't bother implementing the "cat" and "dog" ones…
The different exchange types offer specificity according to your needs. You could do broadcast with the topic exchange, but then you need to force every client to know about the meaning of # and require them to use that routing key when binding; or just use a fanout exchange.
On the implementation side, implementing a topic exchange is a bit more complex, require a data structure that asks for more book-keeping than a plain fanout or direct exchange.
I'm envisioning a configuration in which one exchange is bound to several queues. Each of these bindings will be unique and thus targeted at particular messages. However, some, and perhaps most of the time I'd like the incoming messages to be randomly distributed among the queues. My current thought is to simply have the publisher choose the routing key at random from among the choices. That, however, requires the publisher to have some knowledge of the sort of bindings that are available. It would be better if the random selection were made at the exchange level.
Is it possible to specify a routing key such that the exchange will make a random choice from among the available queue bindings?
thanks
It is possible to define own exchange types for RabbitMQ. Thus you can build an exchange, which exactly fits your needs.
RabbitMQ Random Exchange seems to have the behavior you request. However, I have no experience with it.
So we will have a topic exchange that looks something like
{class}.{genus}
So we have some consumers that bind with the topic
mammal.*
(or bird.*, etc.)
Now suppose later on we want to include species information so the topic exchange now looks like this:
{class}.{genus}.{species}
Now the old consumers are broken :(
However they could have bound as
mammal.*.#
And been able to listen to whatever future information is added. However, this is something my team came up with on our own which leads me to ask:
Is this good practice?
Are there tradeoffs to this I should be aware of?
Is there an alternate way to have a producer be able to add information without breaking existing consumers, without publishing to multiple exchanges?
Typically if you have a need maximum control on queue delivery and want to do the logic in rabbit, then you should consider header exchanges.
Usually when we code up the publish we know exactly which queue it needs to go to, so whether you want to use a routing key or a boolean to do this might not make much difference depending on your application.
This brings up another design consideration to be aware of: whether you want routing logic in rabbit. Someone people prefer to just use simple routing keys and either direct or topic exchanges, focusing on flexible consumers. Its going to be hard to guess at what is best for your application obviously.
Keep in mind that your consumers will be subscribed, often statically, to the queue(s) that the exchange delivers to. Also mammal.# is the same as mammal.*.# (see: ref)