In trying to understanding the difference between direct, fanout and topic exchanges, I want to confirm that the advantage of a topic exchange is that a producer pushes to an exchange and specifies a fully specific routing key, and queues can bind to multiple routing keys via wildcards. e.g. topic pushes to...
$channel->basic_publish($msg, 'logs-exchange', 'error.critical.ram')
And a queue that would message the on-call team on all critical errors would bind like...
$channel->queue_bind('on-call-team', 'logs-exchange', 'error.critical.*')
And a similar queue with error.*.ram would message the sysops team responsible for increasing ram.
Is this the correct use case for a topic exchange, and is the topic exchange the only exchange that can support this flexibility? Conversely are both topic and fanout exchanges implementable by a topic exchange?
Is this the correct use case for a topic exchange,
Yes absolutely correct usage for this exchange. You will see that it is very powerful
and is the topic exchange the only exchange that can support this flexibility?
Yes, the direct exchange which is similar will not support wildcards so you have to specify the full routing key when you bind to it otherwise it will not receive messages. There maybe custom messages that use topic exchanges as basis but I cannot think of one off hand
Conversely are both topic and fanout exchanges implementable by a topic exchange?
I am not sure what you mean here. Yes you could have mutiple queue's bind to the exchange with the same binding key, so they will all receive the same messages. But if you do not need that functionality then you might as well use the fanout exchange itself.
Additionally I have a fanout exchange that is bound to some queues and a topic exchange. The publisher to this exchange uses a routing key, even though it is not needed for the fanout exchange it is needed for the topic exchange that it gets routed to. This is good because it has no effect on the fanout with our causing problems for the topic exchanges
Related
I have a situation when I need to create a route for my messages but I would like to use a matching pattern with negation, like !myPattern.
Example:
I have a queue bound in a Topic Exchange and the routing key is #.brazil.#. So it means that this queue will only receive messages when in the message's routing key contains ".brazil." like message.brazil.denmark.
Now I want to create another queue and bind to the same Topic Exchange but I want to receive all messages that don't contain the pattern #.brazil.#, something like !(#.brazil.#).
I was making some tests using Headers Exchange but the x-match argument only can have 2 possible values: any and all, and I need something like except.
Basically it is not possible to use negation in RabbitMQ even in routing key or header's attribute.
As far as I found out, there are 3 options here:
1 - Using an alternate exchange feature
Declare a fanout exchange you'll publish to (let's call it "my-exchange").
Declare a fanout exchange called "junk".
When each consumer declares a queue, it also declares a topic exchange
and a fanout exchange.
The alternate-exchange for the topic exchange should be set to the fanout exchange.
It then binds the topic exchange to "my-exchange", and "junk" to the topic
exchange, with a routing key equal to the topics it doesn't want.
Thus messages with the "bad" routing key go:
[my-exchange] -> [per-consumer-topic] -> [junk]
and the rest go:
[my-exchange] -> [per-consumer-topic] -> [per-consumer-fanout] ->
[per-consumer-queue]
Solution By: Simon MacMullen-2
Thread Reference: http://rabbitmq.1065348.n5.nabble.com/Binding-to-topic-exchange-with-a-negation-wildcard-td21964.html
2 - Using a Router Consumer
On this solution, you will have only 1 consumer bound in your queue, and the
unique responsibility of this consumer will be "redirect" the message to other
exchanges based on your rules.
Now your router logic will be centralized on this "orchestrator" and not in
RabbitMQ anymore (routing keys or header's attr).
3 - Using a Fanout Exchange
This solution is simple but has a huge drawback, scaling.
Basically you will have a Fanout Exchange responsible to deliver the message to
all bound queues and all consumers will receive the message and check if it
should process or discard the message, it means that now the "router logic"
will be on the consumer side.
The problem with this solution is if you want to scale a specific consumer and
your process is not idempotent you will process the message more than 1 time
(the number of instances running of your consumer).
So in my case, the best approach was the Router Consumer.
I have read so many articles regarding exchanges in RabbitMQ.
But I couldn't find any useful article which explains about usecases of each exchange.
all they are saying is
Direct Exchange - Binding key and routing key should be same.
Topic Exchange - Routing key should match the routing pattern of the binding key.
Fanout Exchange - All the queues which are bounds to that particular exchange will get the message.
Header Exchange - headers should match the key.
can any one explain each exchange and usecases of it in detail?
Well, the choice of exchange type to use is pretty much driven by your use case, it your responsibility to choose the exchange type that suits you best.
Here are some examples:
Lets say you want a broadcast (you have multiple instances of the applications that are supposed to receive the message). For example, something happens in the system (resource becomes available, cache should be invalidated, you name it) and you want that all instances will know it. Then your first bet will be using fanout exchange.
Another example. You want to unicast: send a message and you have multiple consumers, but you want that only one consumer will get a message and will attempt to process it. In this case, you can't use fan-out exchange and you'll opt for other types of exchange (like direct exchange for example).
All-in-all I believe you should be interested to read this article that describes various concepts of fairly rich amqp protocol and provides use case examples for different types of exchanges.
Why do we need routing key to route messages from exchange to queue? Can't we simply use the queue name to route the message? Also, in case of publishing to multiple queues, we can use multiple queue names. Can anyone point out the scenario where we actually need routing key and queue name won't be suffice?
There are several types of exchanges. The fanout exchange ignores the routing key and sends messages to all queues. But pretty much all other exchange types use the routing key to determine which queue, if any, will receive a message.
The tutorials on the RabbitMQ website describes several usecases where different exchange types are useful and where the routing key is relevant.
For instance, tutorial 5 demonstrates how to use a topic exchange to route log messages to different queues depending on the log level of each message.
If you want to target multiple queues, you need to bind them to a fanout exchange and use that exchange in your publisher.
You can't specify multiple queue names in your publisher. In AMQP, you do not publish a message to queues, you publish a message to an exchange. It's the exchange responsability to determine the relevant queues. It's possible that a message is routed to no queue at all and just dropped.
Decoupling queue names from applications is useful for flexibility.
You could establish multiple queues to consume the same message, but queues can't have the same name.
In some cases, message's originator doesn't know the names of queues. (like when you have randomly generated queue names when horizontally scaling a server)
An exchange may be routing messages for more than just one type of consumer. Then you would need some wildcards in your routing keys to route messages to concerned consumers.
As far as I can tell, there is no proper use case for a direct exchange, as anything you can do with it you can do with a fanout exchange, only more expandably.
More specifically, in reading RabbitMQ in Action, the authors numerously refer to the use case that goes something like - "Suppose when a user uploads a picture you need to generate a thumbnail. But then later marketing also tells you to award points for uploading a photo. With RabbitMQ you just have to create another queue and do no work on the producer side!"
But that's only true if you've had the foresight to create a fanout exchange on the producer side. To my understanding a direct exchange cannot accomplish this and is only appropriate when you actually want tight coupling between exchange and queue, (which you don't, because that's the point of messaging systems.)
Is this correct or is there an actual use case?
Compared to the fanout exchange, the direct exchange allows some filtering based on the message's routing key to determine which queue(s) receive(s) the message. With a fanout exchange, there is no such filtering and all messages go to all bound queues.
So if you have a direct exchange with several queues bound with the same routing key, and all messages have this key, then you have the same behavior as the fanout exchange. This is better explained in tutorial 4 on the RabbitMQ website.
In the image upload use case, you can use:
a fanout exchange with two queues (one for the thumbnail worker, one for the score computation worker). The routing key is ignored.
fanout-exchange
|--> queue --> thumbnail-worker
`--> queue --> score-worker
a direct exchange with again two queues. Queues are bound with the image-processing key for instance, and messages with this key will be queued to both queues.
direct-exchange
|--["image-processing"]--> queue --> thumbnail-worker
`--["image-processing"]--> queue --> score-worker
Of course, in this situation, if the message's routing key doesn't match the binding key, none of the queues will receive the message.
You can't put the two workers on the same queue, because messages will be load balanced between them: one worker will see half of the messages.
Do you mean a fanout exchange or a topic exchange? a fanout exchange is very different from a direct exchange. I presume that sending the photo to the exchange is sent with a routing key that specifies that there is a photo. In which case you have a consumer that generates the thumbnail and when you want to add a new consumer you can just add it and get the same message but do something different with it, ie award points.
The use case holds up. I think the point is that the exchange is originally created as a direct exchange.
This answer echoes the previousone and if you refer to this page, I believe you'll that one particular use case described is:
Direct exchanges are often used to distribute tasks between multiple
workers (instances of the same application) in a round robin manner.
I'm looking for a way to secure my websites messaging system so that users only get data they should have access to. With this in mind, I thought of a system where I have a master topic exchange which my server will send all messages to.
The web site holds a sessionId for each user. When a user is authenticated, another exchange is created with a name of sessionId. The client side user is allowed to bind to all exchanges other then the master. Since sessionID's are unique it would be very hard to guess another users sessionID and bind to get their messages.
each message will have a routing key of sessionID.destination. The client side will know all of the potential destinations.
To help visualize:
-> SessionID Exchange -> client
Server -> master Exchange | -> SessionID Exchange -> client
-> SessionID Exchange -> client
My question is two fold. Is it possible to bind an exchange to an exchange in rabbitmq? Also, has someone set up a system like this one previously? Rather, does anyone with experience on this topic already have a working system which I may use?
Thanks in advanced.
Yes it is possible to bind and exchange to and exchange. You can even have different types of exchanges. You need to used channel.exchangeBind() instead of channel.queueBind(). But it works in a similar way.
I have a topic exchange bound to a fanout exchange in my system. I make sure to send a routing key with the messages sent to the fanout exchange. Its no effect at the fanout exchange level but when it gets routed to the topic exchange the routing key is then used to determine which queues it is sent to.
I found this blog which talked about something similar to my design. It's not quite the same thing, but it let's me know that it's at least possible.
http://blog.springsource.org/2011/04/01/routing-topologies-for-performance-and-scalability-with-rabbitmq/
In case of single exchange and multiple queues bound, messages sent by client will reach via exchange to possible queues based upon bindings.Which means message from client will reach destination queue via exchange in one hop.
Consider a case where you want messages to be sent to possible destination queues based upon bindings, ALSO take more hops to different exchange and its queue bound. In such cases exchange to exchange binding will fit the purpose.
Binding queue to exchange is a costly process.Exchange to exchange binding is more flexible and a better scalability solution. Client can create its own private exchange and bind to special purpose exchange at the server. Exchange to exchange binding is better in performance and solves scalability issues.