In RabbitMQ Fifth Tutorial (for Ruby, but they're the same for all languages), they mention:
We created three bindings: Q1 is bound with binding key "*.orange.*"
and Q2 with "*.*.rabbit" and "lazy.#"
and then:
"lazy.pink.rabbit" will be delivered to the second queue only once,
even though it matches two bindings.
How is this possible? For all I know, if a routing key matches 45 queues, it will go to all 45 queues. Why isn't this the case here as well?
Q2 has two bindings that both will match single message. According to
AMQP 0.9.1 specification, section 1.7.2.3. Method queue.bind (page 35):
A server MUST not deliver the same message more than once to a queue, even if the queue has multiple bindings that match the message.
Test scenario:
A client declares a named queue and binds it using multiple bindings to the
amq.topic exchange. The client then publishes a message that matches all its bindings.
so while lazy.pink.rabbit routing key matches both *.*.rabbit and lazy.# routing keys, the message won't be duplicated to Q2 and only one copy of that message will reside in Q2.
P.S.:
There are nice Compatibility and Conformance RabbitMQ documentation page, which contains all specs in one place.
Related
I have a situation when I need to create a route for my messages but I would like to use a matching pattern with negation, like !myPattern.
Example:
I have a queue bound in a Topic Exchange and the routing key is #.brazil.#. So it means that this queue will only receive messages when in the message's routing key contains ".brazil." like message.brazil.denmark.
Now I want to create another queue and bind to the same Topic Exchange but I want to receive all messages that don't contain the pattern #.brazil.#, something like !(#.brazil.#).
I was making some tests using Headers Exchange but the x-match argument only can have 2 possible values: any and all, and I need something like except.
Basically it is not possible to use negation in RabbitMQ even in routing key or header's attribute.
As far as I found out, there are 3 options here:
1 - Using an alternate exchange feature
Declare a fanout exchange you'll publish to (let's call it "my-exchange").
Declare a fanout exchange called "junk".
When each consumer declares a queue, it also declares a topic exchange
and a fanout exchange.
The alternate-exchange for the topic exchange should be set to the fanout exchange.
It then binds the topic exchange to "my-exchange", and "junk" to the topic
exchange, with a routing key equal to the topics it doesn't want.
Thus messages with the "bad" routing key go:
[my-exchange] -> [per-consumer-topic] -> [junk]
and the rest go:
[my-exchange] -> [per-consumer-topic] -> [per-consumer-fanout] ->
[per-consumer-queue]
Solution By: Simon MacMullen-2
Thread Reference: http://rabbitmq.1065348.n5.nabble.com/Binding-to-topic-exchange-with-a-negation-wildcard-td21964.html
2 - Using a Router Consumer
On this solution, you will have only 1 consumer bound in your queue, and the
unique responsibility of this consumer will be "redirect" the message to other
exchanges based on your rules.
Now your router logic will be centralized on this "orchestrator" and not in
RabbitMQ anymore (routing keys or header's attr).
3 - Using a Fanout Exchange
This solution is simple but has a huge drawback, scaling.
Basically you will have a Fanout Exchange responsible to deliver the message to
all bound queues and all consumers will receive the message and check if it
should process or discard the message, it means that now the "router logic"
will be on the consumer side.
The problem with this solution is if you want to scale a specific consumer and
your process is not idempotent you will process the message more than 1 time
(the number of instances running of your consumer).
So in my case, the best approach was the Router Consumer.
RabbitMQ allows distribute messages over several queues based on routing_key matching. For exchange of type='topic' one can specify wildcard symbols which allows more flexible filtering (than in case of type='direct' exchange), they are:
* (star) can substitute for exactly one word.
# (hash) can substitute for zero or more words.
E.g., for queue with routing_key='A.B.*' the exchange is pushing all the messages with routing_key of the patterns 'A.B.A', 'A.B.1', 'A.B.XXX', etc.
In my use case I need two queues bound to the same exchange, one queue receives all messages with routing_key='A.B.A' and another queue receives all other messages which is not matching to 'A.B.A'. It is not a big deal to do it for the first queue but I cannot really find anything in tutorials which helps me with the second queue...
Please, would be nice to have an example in python + pika library.
Could you clarify your requirement of bound to the same exchange?
If it's "just" having the messages pushed through the same exchange instead, you could check out alternate exchange
Basically it's a configuration on your "main" exchange which states that any message that doesn't match any of the bindings to queues bound to it will be transferred to an alternate exchange (can be of different type) for processing.
A simple setup would be to have that alternate exchange of type fanout, routing all of those unbound messages to a given queue.
Why do we need routing key to route messages from exchange to queue? Can't we simply use the queue name to route the message? Also, in case of publishing to multiple queues, we can use multiple queue names. Can anyone point out the scenario where we actually need routing key and queue name won't be suffice?
There are several types of exchanges. The fanout exchange ignores the routing key and sends messages to all queues. But pretty much all other exchange types use the routing key to determine which queue, if any, will receive a message.
The tutorials on the RabbitMQ website describes several usecases where different exchange types are useful and where the routing key is relevant.
For instance, tutorial 5 demonstrates how to use a topic exchange to route log messages to different queues depending on the log level of each message.
If you want to target multiple queues, you need to bind them to a fanout exchange and use that exchange in your publisher.
You can't specify multiple queue names in your publisher. In AMQP, you do not publish a message to queues, you publish a message to an exchange. It's the exchange responsability to determine the relevant queues. It's possible that a message is routed to no queue at all and just dropped.
Decoupling queue names from applications is useful for flexibility.
You could establish multiple queues to consume the same message, but queues can't have the same name.
In some cases, message's originator doesn't know the names of queues. (like when you have randomly generated queue names when horizontally scaling a server)
An exchange may be routing messages for more than just one type of consumer. Then you would need some wildcards in your routing keys to route messages to concerned consumers.
In the queue I have pushed 10K objects. Timestamp is one of the attribute in object. So, how can I write a consumer code using spring amqp?
can anyone help me on this.
AMQP, unlike JMS, has no notion of message selection for consumers. One solution is to use a topic exchange and set the routing key - let's say consumer 1 binds his queue to the exchange with foo.bar a second one binds with foo.baz; and a third binds with foo.*. The third will get all messages (with routing keys starting with foo.); the others will only get messages with their respective keys.
A direct exchange could also be used; it requires a complete match on the routing key.
You should probably work through all the RabbitMQ tutorials to understand the different exchange types before asking more questions here.
I have to implement this scenario:
An external application publish message to rabbitmq.
This message has a client_id property. We can place this id to routing key or message header or some other property.
I have to implement sharding in a exchange routng logic - the message should be delivered to specific queue based on the client_id range.
Is it possible to implement in a standard exchanges?
If not what exchange should I take as the base?
How to dynamicly change client_id ranges?
Take a look at the rabbitmq plugin. It's included in the RabbitMQ distribution from v3.6.0 onwards.
Just have your producer put enough info into the routing key that causes the message to go into the right queue on the other side of the Exchange.
So for example, create two queues called 1 and 2 and bind them with routing keys matching the names. Then have your producer decide which routing key to use when producing the event message. Customers with names starting with letters a-m go to 1, n-z go to 2, you get the idea. It pushes the sharding to the producer but that might be OK for your application.
AMQP doesn't have any explicit implementation of sharding, but its architecture should help you to do that.
Spreading messages to several queues is just a rabbitmq challenge (and part of amqp specification), and with routing, way you can attach hetereogeneous consumers to handle specific messages routed via the same exchange. Therefore, producer should push a specific key to be consumed by specific queue/consumer...
You can decide to make a static sharding, perhaps you have 10 queues with one consumer per queue. You could implement a consistent hashing function such that key is CLIENT_ID % 10.
Another ways and none static solutions could be propoused, and you can try to over this architecture.