Is there a way to bind a queue to an exchange by the combination of Routing Key plus Header Value(s)?
Thanks
No, it is not possible to mix routing logic, at least with one exchange (see explanation and workaround below).
Each exchange type has it own routing mechanism, headers and topic exchanges (as well as others) logic can't be mixed.
If you really sure you need at the same time routing key and headers routing, you can try to play with Exchange to Exchange binding (here is good explanation how it works: Exchange to Exchange bindings in RabbitMQ blog).
P.S.: But it would be better if you explain what problem you are trying to solve, while it looks like you really complicate things (behavior you've asked about is not very common).
Related
I have read so many articles regarding exchanges in RabbitMQ.
But I couldn't find any useful article which explains about usecases of each exchange.
all they are saying is
Direct Exchange - Binding key and routing key should be same.
Topic Exchange - Routing key should match the routing pattern of the binding key.
Fanout Exchange - All the queues which are bounds to that particular exchange will get the message.
Header Exchange - headers should match the key.
can any one explain each exchange and usecases of it in detail?
Well, the choice of exchange type to use is pretty much driven by your use case, it your responsibility to choose the exchange type that suits you best.
Here are some examples:
Lets say you want a broadcast (you have multiple instances of the applications that are supposed to receive the message). For example, something happens in the system (resource becomes available, cache should be invalidated, you name it) and you want that all instances will know it. Then your first bet will be using fanout exchange.
Another example. You want to unicast: send a message and you have multiple consumers, but you want that only one consumer will get a message and will attempt to process it. In this case, you can't use fan-out exchange and you'll opt for other types of exchange (like direct exchange for example).
All-in-all I believe you should be interested to read this article that describes various concepts of fairly rich amqp protocol and provides use case examples for different types of exchanges.
I have a scenario where publishers need to send messages to a known exchange (they won't know the downstream queues directly). Most of the messages should be delivered to every bound queue (like a fanout), but some need to be delivered only to a specific bound queue. I'd like to avoid multiple exchanges as well as multiple queue bonds.
My current solution uses header routing, where a message has one of two headers, identify it as "global" or "specific to a particular sub", where the downstream queues are bound on a match-any to both of those headers. I think this will work, but I feel like there should be a simpler solution.
I tried to find an exchange plugin that would "fanout all messages except specific ones", but I couldn't find such a plugin. Outside of that, any ideas on how to implement such a routing strategy?
For what it's worth, my original solution using a "headers" exchange with queues bound using "match any" is the only one I could find, short of writing a new exchange plugin. It does work and so far seems reasonably fast (at least not measurably slower than a typical "topic" exchange--which I could find no way to apply in this scenario).
From my research on this topic, the ideal solution would be a "topic" exchange with the ability to use a RegEx or at least some form of "or-logic". I did find some information that implied that a RegEx was considered but decided against (in favor of the "dotted notation" topic format) because the latter was faster, specifically because the use a a RegEx would require that every binding be evaluated on each new message (i.e. there was no way to construct a "search shortcut").
For now, my "match-any headers exchange" solution will serve my purposes, but in the future, a "topic" exchange that allowed "or-logic" might be worth exploring. It would allow multiple topic patterns to be achieved with single-binding, without the overhead of a RegEx. But I have no experience with Erlang, nor the time to learn enough of it to write the necessary plugin. Please contact me if anyone is interested in collaborating on this.
What you describe really sounds like exchange-to-exchange binding though, as rich routing is actually one of RMQ's strengths.
You could create the entry exchange that would point to either the fanout (for generic case) or topic/direct one (for the special one), and leave all routing to RMQ. The entry-exchange could be a header exchange or a direct one, depending on what you want to put in headers:
entry-exchange -----> fanout-exchange ---*---> multiple-fanout queues
\
\---> the special exchange --> queue-for-special-usecase
I use RabbitMQ as following :
Create a direct exchange "FooExchange"
Connect a client "A" to "FooExchange" with a queue named "client_A_queue"
Connect a client "B" to "FooExchange" with a queue named "client_B_queue"
Connect a client "C" to "FooExchange" with a queue named "client_C_queue"
Now, when client "A" publish a message to the exchange, everyone receive it.
Is there anyway to avoid client "A" to receive its own messages ?
(and the same for every client : a client should not receive its own messages)
For the moment I have added a "sender" header with a sender UniqueID and I filter these messages in the client source code, but I think that a better solution should exist.
(in real world situation, I can have many clients, not all clients knows the existence of all other clients)
Thanks.
EDIT :
Maybe direct exchange is not the good solution. Is there any way to fit my needs only with exchange/queue/routing configuration or should I use code in client applications to filter these messages ?
Of course, if I have 1000 clients connected, I can't really use one routing key for each client and send message to 999 routing keys jsut to exclude one.
The short answer is that this can't be done in RabbitMQ, directly.
There are no negations in routing key matches, so you can't say "all, but not this one" with routing keys or bindings.
For the moment I have added a "sender" header with a sender UniqueID and I filter these messages in the client source code, but I think that a better solution should exist.
this is pretty much what you need to do
From your comment
Every client publish messages the same way : to "FooExchange"
exchange, with routing key "FooKey". Every client bind it's queue to
"FooKey" on "FooExchange
You are not doing the publishing in the correct way. You must define to which exchange and which routing key.So each subscriber with different routing key, since this is what you want. Check the first tutorial on rabbitmq website. Also bare in mind that when using direct exchange, the name of the queue on the subscribing side is the same as the routing key on the publishing side.
Here is how direct exchange works
taken from here.
EDIT to answer the edit in the question
I didn't really understand this part
I can't really use one routing key for each client and send message to
999 routing keys jsut to exclude one.
You would need to specify more precisely what you need.
Anyhow, I suggest that you check out all the types of exchanges:direct, fanout, topic and headers. More info is already in the link I have provided, under the picture.
EDIT2:
I think I finally understood what is the use case.
If there is no other criteria which you could use to mark the messages or clients, then you'd have to use fanout exchange, and simply don't react on the message if it's "self-sent". Potentially you could use the headers exchange and use some kind of mappings, but it seems that it would end up on the same. AFAIK, there is not pattern for topic exchange that would include NOT something.
In the queue I have pushed 10K objects. Timestamp is one of the attribute in object. So, how can I write a consumer code using spring amqp?
can anyone help me on this.
AMQP, unlike JMS, has no notion of message selection for consumers. One solution is to use a topic exchange and set the routing key - let's say consumer 1 binds his queue to the exchange with foo.bar a second one binds with foo.baz; and a third binds with foo.*. The third will get all messages (with routing keys starting with foo.); the others will only get messages with their respective keys.
A direct exchange could also be used; it requires a complete match on the routing key.
You should probably work through all the RabbitMQ tutorials to understand the different exchange types before asking more questions here.
So we will have a topic exchange that looks something like
{class}.{genus}
So we have some consumers that bind with the topic
mammal.*
(or bird.*, etc.)
Now suppose later on we want to include species information so the topic exchange now looks like this:
{class}.{genus}.{species}
Now the old consumers are broken :(
However they could have bound as
mammal.*.#
And been able to listen to whatever future information is added. However, this is something my team came up with on our own which leads me to ask:
Is this good practice?
Are there tradeoffs to this I should be aware of?
Is there an alternate way to have a producer be able to add information without breaking existing consumers, without publishing to multiple exchanges?
Typically if you have a need maximum control on queue delivery and want to do the logic in rabbit, then you should consider header exchanges.
Usually when we code up the publish we know exactly which queue it needs to go to, so whether you want to use a routing key or a boolean to do this might not make much difference depending on your application.
This brings up another design consideration to be aware of: whether you want routing logic in rabbit. Someone people prefer to just use simple routing keys and either direct or topic exchanges, focusing on flexible consumers. Its going to be hard to guess at what is best for your application obviously.
Keep in mind that your consumers will be subscribed, often statically, to the queue(s) that the exchange delivers to. Also mammal.# is the same as mammal.*.# (see: ref)