My use case is following: I have several users, that subscribed on countries, other words, one user can be related to some countries.
So, producer sends message for country USA, it means that subscribed users on USA should recieve this message. Like a fanout type.
I was thinking about architecture of transport messages in Rabbit MQ broker.
So, I think it is ugly way to create exchanges equal number of countries.
Instead that I suppose to place one exchange with name country binded with some queues, one queue for one country.
Then on client side I determine on which queues users subscribed (Countries).
A continuation:
What if there are cities assigned to countries, and user also subscribes on cities of country?
Should I create another exchange cities and listen messages from queue? Can I send message from country exchange to city exchange directly?
Scheme is fr one country and some relative cities:
Producer -> CountryExhange(USA) ->
CityExchange(Washington) -> Queue (Washington) ->
Consumes (USA, Washington)
Generally, message routing is dependent upon the type of message being processed. So, if each country has its own message type, then by all means set up one queue per country. Otherwise, let country-specific logic be dealt with by message consumers.
Edit:
The question asks whether additional exchanges should be created. It should be noted that the job of an exchange is to route each message published to one or more queues. Standard exchange types are direct, topic, and fanout. These exchanges exist by default in a newly-configured RabbitMQ server. Therefore, unless an additional exchange type is desired, there are very few cases where adding another exchange makes any sense.
Related
TTL can be set on queues, so they will expire after a period of time if they are not used. Is there a similar option for exchanges?
I'm trying to build a social application and each exchange represents a user. Each time someone wants to send a message to this user, he would send the message to the user exchange. If number of the users become large, like 20 million users, there would be 20 million exchanges in the system. I'm afraid that much exchanges degrade the system. Instead I want to only keep exchanges for online users.
By the way the messages are only valuable if the user is online and I don't want to store messages for later delivery.
Having a separate exchange for every user ideed would be overkill. Try a different approach.
Use a single direct exchange.
When a client comes online it creates a new exclusive, auto-delete queue and consumes from it.
The client also binds the single exchange to its queue using the name of the user as the routing key.
Producers publish messages to the single exhange with the name of the user as the routing key of the message.
This will automatically
only keep queues for online users and
discard messages for offline users.
Edit: If a user shall be able to use multiple clients, that's possible using the above approach.
Every client creates a new exclusi auto-delete queue and consumes from it as above.
It binds the single exchange to this queue as above.
Note that it is possible to have multiple bindings from an exchange using identical routing keys. Every client has its own queue and its own binding, even if the routing key on this binding is the same routing key as on another binding created by a different client.
RabbitMQ allows distribute messages over several queues based on routing_key matching. For exchange of type='topic' one can specify wildcard symbols which allows more flexible filtering (than in case of type='direct' exchange), they are:
* (star) can substitute for exactly one word.
# (hash) can substitute for zero or more words.
E.g., for queue with routing_key='A.B.*' the exchange is pushing all the messages with routing_key of the patterns 'A.B.A', 'A.B.1', 'A.B.XXX', etc.
In my use case I need two queues bound to the same exchange, one queue receives all messages with routing_key='A.B.A' and another queue receives all other messages which is not matching to 'A.B.A'. It is not a big deal to do it for the first queue but I cannot really find anything in tutorials which helps me with the second queue...
Please, would be nice to have an example in python + pika library.
Could you clarify your requirement of bound to the same exchange?
If it's "just" having the messages pushed through the same exchange instead, you could check out alternate exchange
Basically it's a configuration on your "main" exchange which states that any message that doesn't match any of the bindings to queues bound to it will be transferred to an alternate exchange (can be of different type) for processing.
A simple setup would be to have that alternate exchange of type fanout, routing all of those unbound messages to a given queue.
I have been learning AMQP using rabbitMQ and I came across this concept called fanout exchanges. From the illustration diagram, all I could see is that it's some kind of load balancer. Could anyone please explain what is it's actual purpose?
I assume that you mean that only one queue will get a message once it arrives to fanout exchange. So from that point of view:
No, I don't think its a load-balancer (I admit that terminology can be confusing).
In Rabbit MQ there are different types of exchanges, its true and fanout exchange is only one type of them. The basic model of Rabbit MQ assumes that you can connect as many queues as you want to the same exchange. Now, all the queues that are connected to the exchange will get the message (Rabbit MQ just replicates the message) - so exchange can't act as a load balancer.
The only difference between the exchange types is an algorithm of matching routing key. It's like a "to" field in a regular envelope. When a message arrives to exchange, it checks the routing key (a.k.a. binding) and depending on type of exchange "finds" to which queue the message should be routed.
When queue gets registered to exchange it always uses this binding. It like queue says to the binding "hey, all messages which are supposed to arrive to John Smith (its a routing key), please pass them to me". Then when the message arrives, it always has a "to" field in the envelope - so exchange checks whether the message is intended to be sent to John Smith, and if so - just routes it to the queue.
It's possible that there will be many queues interested to get a message from John Smith, in this case the message will be replicated. As for fanout exchange - it just doesn't pay any attention to the routing key and instead just sends the message to all the connected queues.
Now, there is another abstraction called consumer. Consumers can be connected to the single queue (again, many consumers can be connected to the queue).
The trick is that only one consumer can get the message for processing at a time.
So if you want a load balancer - you can use a single queue, connected to your exchange (it can be fanout of course), but then connect many consumers to that queue, and rabbit will send the message to the first consumer (it uses round robin internally to pick the first consumer) - if the consumer can't handle it, the message will be re-queued and rabbit will attempt to send it to another consumer.
Why do we need routing key to route messages from exchange to queue? Can't we simply use the queue name to route the message? Also, in case of publishing to multiple queues, we can use multiple queue names. Can anyone point out the scenario where we actually need routing key and queue name won't be suffice?
There are several types of exchanges. The fanout exchange ignores the routing key and sends messages to all queues. But pretty much all other exchange types use the routing key to determine which queue, if any, will receive a message.
The tutorials on the RabbitMQ website describes several usecases where different exchange types are useful and where the routing key is relevant.
For instance, tutorial 5 demonstrates how to use a topic exchange to route log messages to different queues depending on the log level of each message.
If you want to target multiple queues, you need to bind them to a fanout exchange and use that exchange in your publisher.
You can't specify multiple queue names in your publisher. In AMQP, you do not publish a message to queues, you publish a message to an exchange. It's the exchange responsability to determine the relevant queues. It's possible that a message is routed to no queue at all and just dropped.
Decoupling queue names from applications is useful for flexibility.
You could establish multiple queues to consume the same message, but queues can't have the same name.
In some cases, message's originator doesn't know the names of queues. (like when you have randomly generated queue names when horizontally scaling a server)
An exchange may be routing messages for more than just one type of consumer. Then you would need some wildcards in your routing keys to route messages to concerned consumers.
As far as I can tell, there is no proper use case for a direct exchange, as anything you can do with it you can do with a fanout exchange, only more expandably.
More specifically, in reading RabbitMQ in Action, the authors numerously refer to the use case that goes something like - "Suppose when a user uploads a picture you need to generate a thumbnail. But then later marketing also tells you to award points for uploading a photo. With RabbitMQ you just have to create another queue and do no work on the producer side!"
But that's only true if you've had the foresight to create a fanout exchange on the producer side. To my understanding a direct exchange cannot accomplish this and is only appropriate when you actually want tight coupling between exchange and queue, (which you don't, because that's the point of messaging systems.)
Is this correct or is there an actual use case?
Compared to the fanout exchange, the direct exchange allows some filtering based on the message's routing key to determine which queue(s) receive(s) the message. With a fanout exchange, there is no such filtering and all messages go to all bound queues.
So if you have a direct exchange with several queues bound with the same routing key, and all messages have this key, then you have the same behavior as the fanout exchange. This is better explained in tutorial 4 on the RabbitMQ website.
In the image upload use case, you can use:
a fanout exchange with two queues (one for the thumbnail worker, one for the score computation worker). The routing key is ignored.
fanout-exchange
|--> queue --> thumbnail-worker
`--> queue --> score-worker
a direct exchange with again two queues. Queues are bound with the image-processing key for instance, and messages with this key will be queued to both queues.
direct-exchange
|--["image-processing"]--> queue --> thumbnail-worker
`--["image-processing"]--> queue --> score-worker
Of course, in this situation, if the message's routing key doesn't match the binding key, none of the queues will receive the message.
You can't put the two workers on the same queue, because messages will be load balanced between them: one worker will see half of the messages.
Do you mean a fanout exchange or a topic exchange? a fanout exchange is very different from a direct exchange. I presume that sending the photo to the exchange is sent with a routing key that specifies that there is a photo. In which case you have a consumer that generates the thumbnail and when you want to add a new consumer you can just add it and get the same message but do something different with it, ie award points.
The use case holds up. I think the point is that the exchange is originally created as a direct exchange.
This answer echoes the previousone and if you refer to this page, I believe you'll that one particular use case described is:
Direct exchanges are often used to distribute tasks between multiple
workers (instances of the same application) in a round robin manner.