I try to make chat application using RabbitMQ.
Every conversation group has fanoutexchange, to which group members push messages and bind their queues.
But with such routing members receive their own messages, so I have to filter them in the client code.
Is there any way how to make exclude pattern matching, so exchange would not send message to the queue with the same key?
Is there any way how to make exclude pattern matching
nope.
routing keys only match, not exclude
You could use headers exchange instead of fanout.
A headers exchange is designed for routing on multiple attributes that
are more easily expressed as message headers than a routing key.
Headers exchanges ignore the routing key attribute. Instead, the
attributes used for routing are taken from the headers attribute. A
message is considered matching if the value of the header equals the
value specified upon binding.
Related
I have a situation when I need to create a route for my messages but I would like to use a matching pattern with negation, like !myPattern.
Example:
I have a queue bound in a Topic Exchange and the routing key is #.brazil.#. So it means that this queue will only receive messages when in the message's routing key contains ".brazil." like message.brazil.denmark.
Now I want to create another queue and bind to the same Topic Exchange but I want to receive all messages that don't contain the pattern #.brazil.#, something like !(#.brazil.#).
I was making some tests using Headers Exchange but the x-match argument only can have 2 possible values: any and all, and I need something like except.
Basically it is not possible to use negation in RabbitMQ even in routing key or header's attribute.
As far as I found out, there are 3 options here:
1 - Using an alternate exchange feature
Declare a fanout exchange you'll publish to (let's call it "my-exchange").
Declare a fanout exchange called "junk".
When each consumer declares a queue, it also declares a topic exchange
and a fanout exchange.
The alternate-exchange for the topic exchange should be set to the fanout exchange.
It then binds the topic exchange to "my-exchange", and "junk" to the topic
exchange, with a routing key equal to the topics it doesn't want.
Thus messages with the "bad" routing key go:
[my-exchange] -> [per-consumer-topic] -> [junk]
and the rest go:
[my-exchange] -> [per-consumer-topic] -> [per-consumer-fanout] ->
[per-consumer-queue]
Solution By: Simon MacMullen-2
Thread Reference: http://rabbitmq.1065348.n5.nabble.com/Binding-to-topic-exchange-with-a-negation-wildcard-td21964.html
2 - Using a Router Consumer
On this solution, you will have only 1 consumer bound in your queue, and the
unique responsibility of this consumer will be "redirect" the message to other
exchanges based on your rules.
Now your router logic will be centralized on this "orchestrator" and not in
RabbitMQ anymore (routing keys or header's attr).
3 - Using a Fanout Exchange
This solution is simple but has a huge drawback, scaling.
Basically you will have a Fanout Exchange responsible to deliver the message to
all bound queues and all consumers will receive the message and check if it
should process or discard the message, it means that now the "router logic"
will be on the consumer side.
The problem with this solution is if you want to scale a specific consumer and
your process is not idempotent you will process the message more than 1 time
(the number of instances running of your consumer).
So in my case, the best approach was the Router Consumer.
Is it possible to get a list of consumers for a particular routing key when using wildcards?
I have two consumers creating these two routing keys:
customer.created.#
customer.created.from.template.#
I want to find out which routing keys match for a customer.created.from.template event.
RabbitMQ has a management API. One of the methods you can call on this is the /consumers endpoint, which will list all consumers on a particular RabbitMQ cluster.
While I'm sure there would be a way to use this information to get what you need here, I'm not sure what the particular use case is. If you could supply additional detail, it might be possible to advise further.
One possible way of doing this is by the use of Firehose Tracer.
The firehose publishes messages to the topic exchange amq.rabbitmq.trace. In this section we refer to the messages consumed and inspected via the Firehose mechanism as "traced messages".
Traced message routing key will be either "publish.{exchangename}" (for messages entering the node), or "deliver.{queuename}" (for messages that are delivered to consumers).
The Trace queue can then be consumed to extract the desired information.
I use RabbitMQ as following :
Create a direct exchange "FooExchange"
Connect a client "A" to "FooExchange" with a queue named "client_A_queue"
Connect a client "B" to "FooExchange" with a queue named "client_B_queue"
Connect a client "C" to "FooExchange" with a queue named "client_C_queue"
Now, when client "A" publish a message to the exchange, everyone receive it.
Is there anyway to avoid client "A" to receive its own messages ?
(and the same for every client : a client should not receive its own messages)
For the moment I have added a "sender" header with a sender UniqueID and I filter these messages in the client source code, but I think that a better solution should exist.
(in real world situation, I can have many clients, not all clients knows the existence of all other clients)
Thanks.
EDIT :
Maybe direct exchange is not the good solution. Is there any way to fit my needs only with exchange/queue/routing configuration or should I use code in client applications to filter these messages ?
Of course, if I have 1000 clients connected, I can't really use one routing key for each client and send message to 999 routing keys jsut to exclude one.
The short answer is that this can't be done in RabbitMQ, directly.
There are no negations in routing key matches, so you can't say "all, but not this one" with routing keys or bindings.
For the moment I have added a "sender" header with a sender UniqueID and I filter these messages in the client source code, but I think that a better solution should exist.
this is pretty much what you need to do
From your comment
Every client publish messages the same way : to "FooExchange"
exchange, with routing key "FooKey". Every client bind it's queue to
"FooKey" on "FooExchange
You are not doing the publishing in the correct way. You must define to which exchange and which routing key.So each subscriber with different routing key, since this is what you want. Check the first tutorial on rabbitmq website. Also bare in mind that when using direct exchange, the name of the queue on the subscribing side is the same as the routing key on the publishing side.
Here is how direct exchange works
taken from here.
EDIT to answer the edit in the question
I didn't really understand this part
I can't really use one routing key for each client and send message to
999 routing keys jsut to exclude one.
You would need to specify more precisely what you need.
Anyhow, I suggest that you check out all the types of exchanges:direct, fanout, topic and headers. More info is already in the link I have provided, under the picture.
EDIT2:
I think I finally understood what is the use case.
If there is no other criteria which you could use to mark the messages or clients, then you'd have to use fanout exchange, and simply don't react on the message if it's "self-sent". Potentially you could use the headers exchange and use some kind of mappings, but it seems that it would end up on the same. AFAIK, there is not pattern for topic exchange that would include NOT something.
In the queue I have pushed 10K objects. Timestamp is one of the attribute in object. So, how can I write a consumer code using spring amqp?
can anyone help me on this.
AMQP, unlike JMS, has no notion of message selection for consumers. One solution is to use a topic exchange and set the routing key - let's say consumer 1 binds his queue to the exchange with foo.bar a second one binds with foo.baz; and a third binds with foo.*. The third will get all messages (with routing keys starting with foo.); the others will only get messages with their respective keys.
A direct exchange could also be used; it requires a complete match on the routing key.
You should probably work through all the RabbitMQ tutorials to understand the different exchange types before asking more questions here.
I have to implement this scenario:
An external application publish message to rabbitmq.
This message has a client_id property. We can place this id to routing key or message header or some other property.
I have to implement sharding in a exchange routng logic - the message should be delivered to specific queue based on the client_id range.
Is it possible to implement in a standard exchanges?
If not what exchange should I take as the base?
How to dynamicly change client_id ranges?
Take a look at the rabbitmq plugin. It's included in the RabbitMQ distribution from v3.6.0 onwards.
Just have your producer put enough info into the routing key that causes the message to go into the right queue on the other side of the Exchange.
So for example, create two queues called 1 and 2 and bind them with routing keys matching the names. Then have your producer decide which routing key to use when producing the event message. Customers with names starting with letters a-m go to 1, n-z go to 2, you get the idea. It pushes the sharding to the producer but that might be OK for your application.
AMQP doesn't have any explicit implementation of sharding, but its architecture should help you to do that.
Spreading messages to several queues is just a rabbitmq challenge (and part of amqp specification), and with routing, way you can attach hetereogeneous consumers to handle specific messages routed via the same exchange. Therefore, producer should push a specific key to be consumed by specific queue/consumer...
You can decide to make a static sharding, perhaps you have 10 queues with one consumer per queue. You could implement a consistent hashing function such that key is CLIENT_ID % 10.
Another ways and none static solutions could be propoused, and you can try to over this architecture.