Can I have two queues with the same name and same routingKey yet each bound to another exchange?
You can't have two queues with the same name at all, (well you can if they're in different virtual hosts but I don't think that's what you're talking about).
However, you can bind one queue to many exchanges, or to one exchange with many routing keys. If you're aiming to have a single consumer pick up messages from several exchanges, that's what you want to do.
You can't do that, unless those queues are in different vhosts.
Related
RabbitMQ allows distribute messages over several queues based on routing_key matching. For exchange of type='topic' one can specify wildcard symbols which allows more flexible filtering (than in case of type='direct' exchange), they are:
* (star) can substitute for exactly one word.
# (hash) can substitute for zero or more words.
E.g., for queue with routing_key='A.B.*' the exchange is pushing all the messages with routing_key of the patterns 'A.B.A', 'A.B.1', 'A.B.XXX', etc.
In my use case I need two queues bound to the same exchange, one queue receives all messages with routing_key='A.B.A' and another queue receives all other messages which is not matching to 'A.B.A'. It is not a big deal to do it for the first queue but I cannot really find anything in tutorials which helps me with the second queue...
Please, would be nice to have an example in python + pika library.
Could you clarify your requirement of bound to the same exchange?
If it's "just" having the messages pushed through the same exchange instead, you could check out alternate exchange
Basically it's a configuration on your "main" exchange which states that any message that doesn't match any of the bindings to queues bound to it will be transferred to an alternate exchange (can be of different type) for processing.
A simple setup would be to have that alternate exchange of type fanout, routing all of those unbound messages to a given queue.
Why do we need routing key to route messages from exchange to queue? Can't we simply use the queue name to route the message? Also, in case of publishing to multiple queues, we can use multiple queue names. Can anyone point out the scenario where we actually need routing key and queue name won't be suffice?
There are several types of exchanges. The fanout exchange ignores the routing key and sends messages to all queues. But pretty much all other exchange types use the routing key to determine which queue, if any, will receive a message.
The tutorials on the RabbitMQ website describes several usecases where different exchange types are useful and where the routing key is relevant.
For instance, tutorial 5 demonstrates how to use a topic exchange to route log messages to different queues depending on the log level of each message.
If you want to target multiple queues, you need to bind them to a fanout exchange and use that exchange in your publisher.
You can't specify multiple queue names in your publisher. In AMQP, you do not publish a message to queues, you publish a message to an exchange. It's the exchange responsability to determine the relevant queues. It's possible that a message is routed to no queue at all and just dropped.
Decoupling queue names from applications is useful for flexibility.
You could establish multiple queues to consume the same message, but queues can't have the same name.
In some cases, message's originator doesn't know the names of queues. (like when you have randomly generated queue names when horizontally scaling a server)
An exchange may be routing messages for more than just one type of consumer. Then you would need some wildcards in your routing keys to route messages to concerned consumers.
As per my understanding, all the functionality or use-cases of direct and fanout exchanges can be achieved using topic exchange. Topic exchange supports superset of functionality. So the question is, why does RabbitMQ have direct and fanout exchanges? Are there any use-cases that can be achieved using direct/fanout exchange but not with topic exchange?
I would say it's a matter of simplification. If all you need is, say, a topology where each specific routing key maps 1:1 with a queue (round robin distribution of tasks between multiple workers within the same application is an example cited in the RabbitMQ doc), then Direct exchanges may be easier to use and work with, and all you need. That's not to say you couldn't accomplish the same thing using Topic Exchanges (you could).
Similarly, with Fanout exchanges, if you have a situation that calls for the simply broadcasting of messages, you may find a Fanout exchange is easier to work with. Again, that's not to say you couldn't accomplish the same thing using Topic Exchanges (you could).
I generally use Topic exchanges exclusively, because I value the flexibility they provide. As an application expands, they can handle a greater variety of use cases within the same exchange, whereas that may not be the case with the other two types. Thus I can avoid the potential for having to change the topology midstream as an application grows.
As the RabbitMQ doc states about Topic exchanges:
Topic exchanges have a very broad set of use cases. Whenever a problem
involves multiple consumers/applications that selectively choose which
type of messages they want to receive, the use of topic exchanges
should be considered.
For more information about these concepts, including illustrations, this page has quite a bit of info:
https://www.rabbitmq.com/tutorials/amqp-concepts.html
You could replace the functionality of direct and fanout with a topic exchange, but you could also implement a big "Animal" class and don't bother implementing the "cat" and "dog" ones…
The different exchange types offer specificity according to your needs. You could do broadcast with the topic exchange, but then you need to force every client to know about the meaning of # and require them to use that routing key when binding; or just use a fanout exchange.
On the implementation side, implementing a topic exchange is a bit more complex, require a data structure that asks for more book-keeping than a plain fanout or direct exchange.
I have a Topic exchange from which I'd like to distribute messages to two queues on two servers part of a cluster, in order to reduce memory pressure on any particular server. My consumers are periodically slow, and I sometimes run into the high memory watermark.
The way I tried to resolve this is by routing messages using an intermediate direct exchange, with two queues bound to the exchange:
a (topic) -> a1 (direct) -> q1/q2 (bound to routing key "a")
But the messages were routed to both queues, as AMQP intends. Anyone has ideas? What I need is an exchange that routes to one and only one queue, even if the routing key matches many queues. I'd prefer not to change my routing keys, but that could be arranged.
I found Selective routing with RabbitMQ, which may mean I'll need to implement my own routing logic. Hopefully, this already exists somewhere else.
You could perhaps use the Shovel plugin - http://www.rabbitmq.com/shovel.html - to move messages from your intermediate exchange to the two queues.
If you set up two shovels, both consuming from a single queue on the direct intermediate exchange, they should be able to fight over the messages coming in (I'm assuming that you don't care too much if the two recipient queues don't get the incoming messages in a strict round robin fashion). The shovels then each publish to one of the two end queues, and can send through the ACKs from the end consumer.
Pretty new to RabbitMQ and we're still in the investigation stage to see if it's a good fit for our use cases--
We've readily come to the conclusion that our desired topology would have us deploying a few topic based exchanges, and then filtering from there to specific queues. For example, let's say we have a user and an upload exchange, where the user queue might receive messages where the topic is "new-registration" or "friend-request" and the upload exchange might receive messages like "video-upload" or "picture-upload".
Creating the queues, getting them routed to the appropriate queue, and then building listeners to handle the messages for the various queues has been quite straight forward.
What's unclear to me however is if it's possible to do a fanout on a topic exchange?
I.e. I have named queues that are bound to my topic exchange, but I'd like to be able to just throw tons of instances of my listeners at those queues to prevent single points of failure. But to the best of my knowledge, RabbitMQ treats these listeners in a straight forward round robin fashion--e.g. every Nth message always go to the same Nth listener rather than dispatching messages to the first available consumer. This is generally acceptable to us but given the load we anticipate, we'd like to avoid the possibility of hot spots developing amongst our consumer farm.
So, is there some way, either in the queue or exchange configuration or in the consumer code, where we can point our listeners to a topic queue but have the listeners treated in a fanout fashion?
Yes, by having the listeners bind using different queue names, they will be treated in a fanout fashion.
Fanout is 1:N though, i.e. each task can be delivered to multiple listeners like pub-sub. Note that this isn't restricted to a fanout exchange, but also applies if you bind multiple queues to a direct or topic exchange with the same binding key. (Installing the management plugin and looking at the exchanges there may be useful to visualize the bindings in effect.)
Your current setup is a task queue. Each task/message is delivered to exactly one worker/listener. Throw more listeners at the same queue name, and they will process the tasks round-robin as you say. With "fanout" (separate queues for a topic) you will process a task multiple times.
Depending on your platform there may be existing work queue solutions that meet your requirements, such as Resque or DelayedJob for Ruby, Celery for Python or perhaps Octobot or Akka for the JVM.
I don't know for a fact, but I strongly suspect that RabbitMQ will skip consumers with unacknowledged messages, so it should never bottleneck on a single stuck consumer. The comments on their FAQ seem to suggest that RabbitMQ will make an effort to keep things chugging along even in the presence of troublesome consumers.
This is a late answer, but in case others come across this question...
It sounds like what you want is fair dispatch rather than a fan out model (which would publish a given message to every queue).
Fair dispatch will give a message to the next available worker rather than using a simple round-robin approach. This should avoid the "hotspots" you are concerned about, without delivering the same message to multiple consumers.
If this is what you are looking for, then see the "Fair Dispatch" section on this page in the Rabbit docs. A prefetch count of 1 is the key here.