I'm using the RabbitMQ Direct reply-to feature for my RPC calls and I'm using Correlation Id as it mentions here.
My question is do I need to use the Correlation Id if I'm using the Direct reply-to?
reply-to value is known to be unique (it looks like amq.rabbitmq.reply-to.g2dkAA9yYWJiaXRAbWVyY3VyaW8AAAFJAAAAAAI=.41Feb1a6XK29NG5/qnF24w==), so using correlation-id is not mandatory, but you MAY keep using if you plan to switch back from Direct reply-to for some reason.
Related
I use RabbitMQ as following :
Create a direct exchange "FooExchange"
Connect a client "A" to "FooExchange" with a queue named "client_A_queue"
Connect a client "B" to "FooExchange" with a queue named "client_B_queue"
Connect a client "C" to "FooExchange" with a queue named "client_C_queue"
Now, when client "A" publish a message to the exchange, everyone receive it.
Is there anyway to avoid client "A" to receive its own messages ?
(and the same for every client : a client should not receive its own messages)
For the moment I have added a "sender" header with a sender UniqueID and I filter these messages in the client source code, but I think that a better solution should exist.
(in real world situation, I can have many clients, not all clients knows the existence of all other clients)
Thanks.
EDIT :
Maybe direct exchange is not the good solution. Is there any way to fit my needs only with exchange/queue/routing configuration or should I use code in client applications to filter these messages ?
Of course, if I have 1000 clients connected, I can't really use one routing key for each client and send message to 999 routing keys jsut to exclude one.
The short answer is that this can't be done in RabbitMQ, directly.
There are no negations in routing key matches, so you can't say "all, but not this one" with routing keys or bindings.
For the moment I have added a "sender" header with a sender UniqueID and I filter these messages in the client source code, but I think that a better solution should exist.
this is pretty much what you need to do
From your comment
Every client publish messages the same way : to "FooExchange"
exchange, with routing key "FooKey". Every client bind it's queue to
"FooKey" on "FooExchange
You are not doing the publishing in the correct way. You must define to which exchange and which routing key.So each subscriber with different routing key, since this is what you want. Check the first tutorial on rabbitmq website. Also bare in mind that when using direct exchange, the name of the queue on the subscribing side is the same as the routing key on the publishing side.
Here is how direct exchange works
taken from here.
EDIT to answer the edit in the question
I didn't really understand this part
I can't really use one routing key for each client and send message to
999 routing keys jsut to exclude one.
You would need to specify more precisely what you need.
Anyhow, I suggest that you check out all the types of exchanges:direct, fanout, topic and headers. More info is already in the link I have provided, under the picture.
EDIT2:
I think I finally understood what is the use case.
If there is no other criteria which you could use to mark the messages or clients, then you'd have to use fanout exchange, and simply don't react on the message if it's "self-sent". Potentially you could use the headers exchange and use some kind of mappings, but it seems that it would end up on the same. AFAIK, there is not pattern for topic exchange that would include NOT something.
Can anyone give me examples of how in production a correlation id can be used?
I have read it is used in request/response type messages but I don't understand where I would use it?
One example (which maybe wrong) I can think off is in a publish subscribe scenario where I could have 5 subscribers and if I get 5 replies with the same correlation id then I could say all my subscribers have received it. Not sure if this would the be correct usage of it.
Or if I send a simple message, the I can use the correlation to guarantee that the client received it.
Any other examples?
A web application that is providing HTTP API for outsiders for performing a processing task and you want to give the results for the caller as a response to the HTTP request they made.
A request comes in, message describing the task is pushed to queue by the frontend server. After that the frontend server blocks to wait for response message with the same correlation id. A pool of worker machines are listening on queue and one of them picks up the task, performs it and returns the result as message. Once a message with right correlation id comes in, frontend server continues to return the response to the caller.
In the context of CQRS and EventSourcing a command message correlation id will most likely get stored togehter with the corresponding events from the domain. This information can later be used to form an audit trail.
Streaming engines like Apache Flink use correlation ids, much like you said, to guarantee exactness of processing.
In the queue I have pushed 10K objects. Timestamp is one of the attribute in object. So, how can I write a consumer code using spring amqp?
can anyone help me on this.
AMQP, unlike JMS, has no notion of message selection for consumers. One solution is to use a topic exchange and set the routing key - let's say consumer 1 binds his queue to the exchange with foo.bar a second one binds with foo.baz; and a third binds with foo.*. The third will get all messages (with routing keys starting with foo.); the others will only get messages with their respective keys.
A direct exchange could also be used; it requires a complete match on the routing key.
You should probably work through all the RabbitMQ tutorials to understand the different exchange types before asking more questions here.
I have to implement this scenario:
An external application publish message to rabbitmq.
This message has a client_id property. We can place this id to routing key or message header or some other property.
I have to implement sharding in a exchange routng logic - the message should be delivered to specific queue based on the client_id range.
Is it possible to implement in a standard exchanges?
If not what exchange should I take as the base?
How to dynamicly change client_id ranges?
Take a look at the rabbitmq plugin. It's included in the RabbitMQ distribution from v3.6.0 onwards.
Just have your producer put enough info into the routing key that causes the message to go into the right queue on the other side of the Exchange.
So for example, create two queues called 1 and 2 and bind them with routing keys matching the names. Then have your producer decide which routing key to use when producing the event message. Customers with names starting with letters a-m go to 1, n-z go to 2, you get the idea. It pushes the sharding to the producer but that might be OK for your application.
AMQP doesn't have any explicit implementation of sharding, but its architecture should help you to do that.
Spreading messages to several queues is just a rabbitmq challenge (and part of amqp specification), and with routing, way you can attach hetereogeneous consumers to handle specific messages routed via the same exchange. Therefore, producer should push a specific key to be consumed by specific queue/consumer...
You can decide to make a static sharding, perhaps you have 10 queues with one consumer per queue. You could implement a consistent hashing function such that key is CLIENT_ID % 10.
Another ways and none static solutions could be propoused, and you can try to over this architecture.
This is similar to this other question but with a bit of a twist: I read in the specification that the message-id for AMQP messages should be set by the application itself, so in theory I could use that to guarantee a certain degree of uniqueness, right?
My main question is now: In what scope is that message-id garantueed to be unique? For the messages currently enqueued inside a specific queue? Over all queues? Over the universe? :-)
And is this behaviour standardized? I plan to use RabbitMQ here, but it would be nice to have something not vendor specifc :-)
Thanks.
Another suggestion is according to the dump pipes - smart endpoints school of thought.
You could handle uniqueness in your application, using some sort of shared state.
We had the same problem when switching from Gearman to RabbitMQ. We use memcached to keep track of unique message ID's posted and consumers drop messages for which the message ID is already stored in memcache (duplicates). You could also check memcache before putting it on the queue altogether.
This frees you from using this feature in your message bus layer (so you can more easily switch between brokers, also those who do not guarantee uniqueness)
Message Id is application-specific only and may be not unique at all. You have to take care of uniqueness by yourself.