I'm envisioning a configuration in which one exchange is bound to several queues. Each of these bindings will be unique and thus targeted at particular messages. However, some, and perhaps most of the time I'd like the incoming messages to be randomly distributed among the queues. My current thought is to simply have the publisher choose the routing key at random from among the choices. That, however, requires the publisher to have some knowledge of the sort of bindings that are available. It would be better if the random selection were made at the exchange level.
Is it possible to specify a routing key such that the exchange will make a random choice from among the available queue bindings?
thanks
It is possible to define own exchange types for RabbitMQ. Thus you can build an exchange, which exactly fits your needs.
RabbitMQ Random Exchange seems to have the behavior you request. However, I have no experience with it.
Related
I have read so many articles regarding exchanges in RabbitMQ.
But I couldn't find any useful article which explains about usecases of each exchange.
all they are saying is
Direct Exchange - Binding key and routing key should be same.
Topic Exchange - Routing key should match the routing pattern of the binding key.
Fanout Exchange - All the queues which are bounds to that particular exchange will get the message.
Header Exchange - headers should match the key.
can any one explain each exchange and usecases of it in detail?
Well, the choice of exchange type to use is pretty much driven by your use case, it your responsibility to choose the exchange type that suits you best.
Here are some examples:
Lets say you want a broadcast (you have multiple instances of the applications that are supposed to receive the message). For example, something happens in the system (resource becomes available, cache should be invalidated, you name it) and you want that all instances will know it. Then your first bet will be using fanout exchange.
Another example. You want to unicast: send a message and you have multiple consumers, but you want that only one consumer will get a message and will attempt to process it. In this case, you can't use fan-out exchange and you'll opt for other types of exchange (like direct exchange for example).
All-in-all I believe you should be interested to read this article that describes various concepts of fairly rich amqp protocol and provides use case examples for different types of exchanges.
I'm looking for a way to buffer messages received by the exchange as long as there is at least one queue bind to that exchange.
Is it supported by RabbitMQ?
Maybe there are some workarounds (I didn't find any).
EDIT
My use case:
I've got one data producer (which reads real-time data from an external system)
I've got one fanout exchange which receives data from the producer
On system startup, there might be no consumer, but after a few moments, there should be at least one which creates his own queue and binds it to the exchange from 2.
The problem is this short time between step 2. and 3. where there are no queues bound to the exchange created in step 1.
Of course, it's an edge case and after system initialization queues and exchanges are bound and everything works as expected.
Why queues and bindings has to be created by consumers (not by the producer)? Because I need a flexible setup where I can add consumers without any changes in other components code (e.g. producer).
EDIT 2
I'm processing the output from another system which stores both real-time and historical data. There are the cases where I want to read historical data first (on initialization) and then continue to handle real-time data.
I may mislead you by saying that there are multiple consumers. In the case where I need a buffer on exchange there is only one consumer (which writes everything to time series DB as it appears in queue).
The RabbitMQ team monitors this mailing list and only sometimes answers questions on StackOverflow.
Why queues and bindings has to be created by consumers (not by the producer)?
Queues and bindings can be created by producers or consumers or both. The requirement is that the exact same arguments are used when creating them if a client application tries to "re-create" a queue or binding. If different arguments are used, a channel-level error will happen.
As you have found, if a producer publishes to an exchange that can't route messages, they will be lost. Olivier's suggestion to use an alternate exchange is a good one, but I recommend you have your producers create queues and bindings as well.
If you mean to avoid throwing away messages because there is no destination configured for it, yes.
You should look at alternate exchange.
This assume that before (or when) you start (or when), the alternate exchange is created (would typically go for fanout) and a queue is binded to it (let's call it notroutedq).
So the messages are not lost, they will be stored in notroutedq.
From there you can possibly setup a mechanism that would reprocess messages in that queue - reinjecting them into the main exchange most likely - once a given time has passed or when a binding has been added to your main exchange.
-- EDIT --
Thanks for the updated info.
Could you indicate how long typically you'd expect the past messages to be useful to the consumers?
In your description, you mention real-time data and possibly multiple consumers coming and going. Based on that, I'm not sure how much of the data kept in the notroutedq would be of value, and with which frequency you'd expect to resend them to the consumers.
The cases I had with alternate exchange where mostly focused on identifying missing bindings, so that one could easily correct the bindings and reprocess the messages without loss.
If the number of consumers varies through time and the data content is real-time, I'd wonder a bit about the benefit of keeping the data.
So we decided to go with RabbitMQ as a message/event bus in our migration to micro-services architecture, but we couldn't find a definite answer on what is the best way of putting our queues, we have two options to go with:
One main exchange which will be a Fanout exchange, which in turn will fan messages out to a main queue for logging and other purposes and another sub exchange which will be a topic exchange and route the messages to each desired queue using the message routing key. We expect the number of queues behind the sub-exchange to be some how a large number. This can be explained by this graph:
One main exchange, which will be a Topic exchange, with still one main queue bound to that exchange using "#" routing key. That main exchange will also handles main routing to other sub exchanges, so routing keys might be "agreements.#", "assignments.#", "messages.#", which are then used to bind multiple topic sub-exchanges, each will handle sub routing, so one sub exchange might be handling all "assignments" and queues bound to that exchange could be bound by routing keys like "assignments.accepted", "assignments.deleted"...In this scenario, we feel like the huge number of queues will be less per exchange, they will be somehow distributed between exchanges.
So, which of these scenarios could be the best approach? Faster on RabbitMQ, less overhead.
Taking in mind, all queues, exchanges and bindings will be done on the fly from the service which will be either publishing or subscribing.
You can find some explanation in this topic: RabbitMQ Topic exchanges: 1 Exchange vs Many Exchanges
I am using RabbitMQ in a very similar way that you showed in the case 2, as I found the same benefits as described in this article: https://skillachie.com/2014/06/27/rabbitmq-exchange-to-exchange-bindings-ampq/
Exchange-to-exchange bindings are much more flexible in terms of the topology that you can design, promotes decoupling & reduce binding churn
Exchange-to-exchange bindings are said to be very light weight and as a result help to increase performance *
Based in my own experience with exchange-to-exchange, the case 2 is great and it will allows to create/change messages flow topologies in a very fast way.
I'm going to first re-summarize what I think is your question, since I'm sure it's buried somewhere in your post.
It is desirable to have a tracer/logging queue, in addition to a series of work-specific queues for actual message processing. What exchange topology is best for this scenario?
First off, neither option makes much sense given your application. Option 1 will create an exchange that will publish a message to every queue bound to it, regardless. This is clearly not what you want. Option 2 will give you a rather complex routing topology for which the benefit is unclear, and the drawback is painful maintenance and a steep learning curve. (Just because you can do something does not mean you should do it.)
What should be done?
It is important to remember that in RabbitMQ, it is the queues which consume the resources of the broker. Exchanges merely connect queues with publishers. The exchange is a means to an end, while the queue is the end itself.
What instead I think you should do is set up a single topic exchange. Bind your tracing queue to routing key # so that you receive all messages. Then, bind your work queues appropriately so that they receive only the messages that need to flow into them. For example, it is common to route messages by message type, where each queue holds exactly one type of message. This is both simple and effective.
The advantage of a single topic exchange is that you get the benefits of both a Direct Exchange and a Fanout Exchange depending on the binding key used. Further, configuration changes are easy to achieve and can often be done without disrupting any system processing at all (let's say that you want to stop tracing certain messages - this can be done with ease using a topic exchange, assuming your routing keys are rational).
Exchange-to-exchange bindings is semantically identical to exchange-to-queue bindings.
https://www.rabbitmq.com/e2e.html
As per my understanding, all the functionality or use-cases of direct and fanout exchanges can be achieved using topic exchange. Topic exchange supports superset of functionality. So the question is, why does RabbitMQ have direct and fanout exchanges? Are there any use-cases that can be achieved using direct/fanout exchange but not with topic exchange?
I would say it's a matter of simplification. If all you need is, say, a topology where each specific routing key maps 1:1 with a queue (round robin distribution of tasks between multiple workers within the same application is an example cited in the RabbitMQ doc), then Direct exchanges may be easier to use and work with, and all you need. That's not to say you couldn't accomplish the same thing using Topic Exchanges (you could).
Similarly, with Fanout exchanges, if you have a situation that calls for the simply broadcasting of messages, you may find a Fanout exchange is easier to work with. Again, that's not to say you couldn't accomplish the same thing using Topic Exchanges (you could).
I generally use Topic exchanges exclusively, because I value the flexibility they provide. As an application expands, they can handle a greater variety of use cases within the same exchange, whereas that may not be the case with the other two types. Thus I can avoid the potential for having to change the topology midstream as an application grows.
As the RabbitMQ doc states about Topic exchanges:
Topic exchanges have a very broad set of use cases. Whenever a problem
involves multiple consumers/applications that selectively choose which
type of messages they want to receive, the use of topic exchanges
should be considered.
For more information about these concepts, including illustrations, this page has quite a bit of info:
https://www.rabbitmq.com/tutorials/amqp-concepts.html
You could replace the functionality of direct and fanout with a topic exchange, but you could also implement a big "Animal" class and don't bother implementing the "cat" and "dog" ones…
The different exchange types offer specificity according to your needs. You could do broadcast with the topic exchange, but then you need to force every client to know about the meaning of # and require them to use that routing key when binding; or just use a fanout exchange.
On the implementation side, implementing a topic exchange is a bit more complex, require a data structure that asks for more book-keeping than a plain fanout or direct exchange.
So we will have a topic exchange that looks something like
{class}.{genus}
So we have some consumers that bind with the topic
mammal.*
(or bird.*, etc.)
Now suppose later on we want to include species information so the topic exchange now looks like this:
{class}.{genus}.{species}
Now the old consumers are broken :(
However they could have bound as
mammal.*.#
And been able to listen to whatever future information is added. However, this is something my team came up with on our own which leads me to ask:
Is this good practice?
Are there tradeoffs to this I should be aware of?
Is there an alternate way to have a producer be able to add information without breaking existing consumers, without publishing to multiple exchanges?
Typically if you have a need maximum control on queue delivery and want to do the logic in rabbit, then you should consider header exchanges.
Usually when we code up the publish we know exactly which queue it needs to go to, so whether you want to use a routing key or a boolean to do this might not make much difference depending on your application.
This brings up another design consideration to be aware of: whether you want routing logic in rabbit. Someone people prefer to just use simple routing keys and either direct or topic exchanges, focusing on flexible consumers. Its going to be hard to guess at what is best for your application obviously.
Keep in mind that your consumers will be subscribed, often statically, to the queue(s) that the exchange delivers to. Also mammal.# is the same as mammal.*.# (see: ref)