I have a situation when I need to create a route for my messages but I would like to use a matching pattern with negation, like !myPattern.
Example:
I have a queue bound in a Topic Exchange and the routing key is #.brazil.#. So it means that this queue will only receive messages when in the message's routing key contains ".brazil." like message.brazil.denmark.
Now I want to create another queue and bind to the same Topic Exchange but I want to receive all messages that don't contain the pattern #.brazil.#, something like !(#.brazil.#).
I was making some tests using Headers Exchange but the x-match argument only can have 2 possible values: any and all, and I need something like except.
Basically it is not possible to use negation in RabbitMQ even in routing key or header's attribute.
As far as I found out, there are 3 options here:
1 - Using an alternate exchange feature
Declare a fanout exchange you'll publish to (let's call it "my-exchange").
Declare a fanout exchange called "junk".
When each consumer declares a queue, it also declares a topic exchange
and a fanout exchange.
The alternate-exchange for the topic exchange should be set to the fanout exchange.
It then binds the topic exchange to "my-exchange", and "junk" to the topic
exchange, with a routing key equal to the topics it doesn't want.
Thus messages with the "bad" routing key go:
[my-exchange] -> [per-consumer-topic] -> [junk]
and the rest go:
[my-exchange] -> [per-consumer-topic] -> [per-consumer-fanout] ->
[per-consumer-queue]
Solution By: Simon MacMullen-2
Thread Reference: http://rabbitmq.1065348.n5.nabble.com/Binding-to-topic-exchange-with-a-negation-wildcard-td21964.html
2 - Using a Router Consumer
On this solution, you will have only 1 consumer bound in your queue, and the
unique responsibility of this consumer will be "redirect" the message to other
exchanges based on your rules.
Now your router logic will be centralized on this "orchestrator" and not in
RabbitMQ anymore (routing keys or header's attr).
3 - Using a Fanout Exchange
This solution is simple but has a huge drawback, scaling.
Basically you will have a Fanout Exchange responsible to deliver the message to
all bound queues and all consumers will receive the message and check if it
should process or discard the message, it means that now the "router logic"
will be on the consumer side.
The problem with this solution is if you want to scale a specific consumer and
your process is not idempotent you will process the message more than 1 time
(the number of instances running of your consumer).
So in my case, the best approach was the Router Consumer.
Related
In my system, I use Topic Exchanges with lots of consumer queues. Each queue has it's own non-unique routing key (f.e. 'add.#' for all new entities or just '#' to consume all events).
I want to add support for retrying failed messages with some delay. The biggest issue that I see with Dead Letter Exchange approach is to send a message directly to the queue in which it failed. Routing keys for Queues are not unique, and even if I resubmit a message to the Exchange with the original routing key, it will be consumed by other queues.
One solution is having a "retry" exchange and every application will be subscribed to it with unique routing key (f.e. original queue name). But it sounds too complicated and I want to hide this infrastructure complexity from developers.
I came up with the idea to have a filter that will check the 'x-death' header, get the first queue (the queue where the error occurred in a first place), and process a message only for the appropriate queue. Otherwise - acknowledge the message.
Is it possible to implement this behavior with Spring AMQP? I'm looking into MessagePostProcessor, but how to Acknowledge a message from it?
If you really worry about only the target queue, so you need to consider a variant with republishing in the default exchange which has these abilities:
The default exchange is implicitly bound to every queue, with a routing key equal to the queue name. It is not possible to explicitly bind to, or unbind from the default exchange. It also cannot be deleted.
Pay attention to the routing key equal to the queue name part. I would consider to deal with a AmqpHeaders.CONSUMER_QUEUE and use its value as a routing key for republishing to the default exchange ("") during retry process.
Why do we need routing key to route messages from exchange to queue? Can't we simply use the queue name to route the message? Also, in case of publishing to multiple queues, we can use multiple queue names. Can anyone point out the scenario where we actually need routing key and queue name won't be suffice?
There are several types of exchanges. The fanout exchange ignores the routing key and sends messages to all queues. But pretty much all other exchange types use the routing key to determine which queue, if any, will receive a message.
The tutorials on the RabbitMQ website describes several usecases where different exchange types are useful and where the routing key is relevant.
For instance, tutorial 5 demonstrates how to use a topic exchange to route log messages to different queues depending on the log level of each message.
If you want to target multiple queues, you need to bind them to a fanout exchange and use that exchange in your publisher.
You can't specify multiple queue names in your publisher. In AMQP, you do not publish a message to queues, you publish a message to an exchange. It's the exchange responsability to determine the relevant queues. It's possible that a message is routed to no queue at all and just dropped.
Decoupling queue names from applications is useful for flexibility.
You could establish multiple queues to consume the same message, but queues can't have the same name.
In some cases, message's originator doesn't know the names of queues. (like when you have randomly generated queue names when horizontally scaling a server)
An exchange may be routing messages for more than just one type of consumer. Then you would need some wildcards in your routing keys to route messages to concerned consumers.
In the queue I have pushed 10K objects. Timestamp is one of the attribute in object. So, how can I write a consumer code using spring amqp?
can anyone help me on this.
AMQP, unlike JMS, has no notion of message selection for consumers. One solution is to use a topic exchange and set the routing key - let's say consumer 1 binds his queue to the exchange with foo.bar a second one binds with foo.baz; and a third binds with foo.*. The third will get all messages (with routing keys starting with foo.); the others will only get messages with their respective keys.
A direct exchange could also be used; it requires a complete match on the routing key.
You should probably work through all the RabbitMQ tutorials to understand the different exchange types before asking more questions here.
As far as I can tell, there is no proper use case for a direct exchange, as anything you can do with it you can do with a fanout exchange, only more expandably.
More specifically, in reading RabbitMQ in Action, the authors numerously refer to the use case that goes something like - "Suppose when a user uploads a picture you need to generate a thumbnail. But then later marketing also tells you to award points for uploading a photo. With RabbitMQ you just have to create another queue and do no work on the producer side!"
But that's only true if you've had the foresight to create a fanout exchange on the producer side. To my understanding a direct exchange cannot accomplish this and is only appropriate when you actually want tight coupling between exchange and queue, (which you don't, because that's the point of messaging systems.)
Is this correct or is there an actual use case?
Compared to the fanout exchange, the direct exchange allows some filtering based on the message's routing key to determine which queue(s) receive(s) the message. With a fanout exchange, there is no such filtering and all messages go to all bound queues.
So if you have a direct exchange with several queues bound with the same routing key, and all messages have this key, then you have the same behavior as the fanout exchange. This is better explained in tutorial 4 on the RabbitMQ website.
In the image upload use case, you can use:
a fanout exchange with two queues (one for the thumbnail worker, one for the score computation worker). The routing key is ignored.
fanout-exchange
|--> queue --> thumbnail-worker
`--> queue --> score-worker
a direct exchange with again two queues. Queues are bound with the image-processing key for instance, and messages with this key will be queued to both queues.
direct-exchange
|--["image-processing"]--> queue --> thumbnail-worker
`--["image-processing"]--> queue --> score-worker
Of course, in this situation, if the message's routing key doesn't match the binding key, none of the queues will receive the message.
You can't put the two workers on the same queue, because messages will be load balanced between them: one worker will see half of the messages.
Do you mean a fanout exchange or a topic exchange? a fanout exchange is very different from a direct exchange. I presume that sending the photo to the exchange is sent with a routing key that specifies that there is a photo. In which case you have a consumer that generates the thumbnail and when you want to add a new consumer you can just add it and get the same message but do something different with it, ie award points.
The use case holds up. I think the point is that the exchange is originally created as a direct exchange.
This answer echoes the previousone and if you refer to this page, I believe you'll that one particular use case described is:
Direct exchanges are often used to distribute tasks between multiple
workers (instances of the same application) in a round robin manner.
I have exactly 2 types of messages that I want to be sent via RabbitMQ. So I have 2 options how I can do this:
sent a message to default empty-named exchange with routing_key corresponding to the queue name
use direct exchange's routing_key parameter corresponding to consumer's routing_key parameter in queue binding
So which option is preferable and why?
A default exchange is a direct exchange. RabbitMQ creates the default exchange by default, but it uses an empty string for the name. If you look at the RabbitMQ AMQP concepts page, under Default Exchange:
The default exchange is a direct exchange with no name (empty string)
pre-declared by the broker.
You can see this by running rabbitmqctl list_exchanges as well:
direct
Foo direct < Same thing as the above
amq.direct direct
amq.fanout fanout
...and so on
As far as I'm aware, there isn't any benefits of using one over the other. I would stick with the default exchange if you only need to route based on routing keys.
Let's say you direct-bind to an exchange broadcasting logs to routing keys of "info", "warn", and "error". Using the default exchange, you would need to create three different queues with those names to receive all logs; and adjustments to which log levels you receive would require changing your queue declarations. By using a named exchange, you can simply change your queue's bindings and continue processing things as normal.
In short, it provides one extra level of abstraction.
As I see it, the default direct exchange give the possibility for the consumers and the producers to not know about each other, by binding a queue (used by a consumer) to an exchange (used by a producer) implicitly using the queue's name.
I use the default direct-exchange for a specific case: the consumer and producers don't know about each other. In my case, each consumer have its proper queue. From the producer, I cannot know by advance which queues are going to be declared and used, as it depends on the consumers. So it is impossible to define the bindings between a custom direct-exchange and the queues on the producer side.
One way to solve it with a custom (user-defined) direct-exchange would be to define the binding-key on consumer side. But it would means to know about the producer from the consumer side as I need to know the exchange name used by the producer.
Therefore, automatically binding a queue by its name on the default direct-exchange makes it possible, in my case, to only declare a queue on consumer side and send message to it from producer by only knowing the queue's name.
Of course it implies to know the queue's name at runtime, when invoking the producer, as it would be required to know the binding-key of a custom direct-exchange (in my case, the queue's name is given by the application using the producer). But when configuring the broker, producers and consumers don't have to know about each other.
some description in rabbitmq web manager.
//default exchange 's Bindings
Default exchange
The default exchange is implicitly bound to every queue,
with a routing key equal to the queue name.
It is not possible to
explicitly bind to, or
unbind from the default exchange.
It also cannot be deleted.