I'm not entirely certain this is the right place to ask, but I have questions about the functioning of RabbitMQ to implement a publish/subscribe messaging system. I am investigating RabbitMQ to see if it fits the needs of the company I work for, and I can't find any answer to these questions either in the official doc or on the internet :
Is it possible for a consumer to subscribe to a group of publishers? I guess it could be possible, if all publishers from the group we want to subscribe to send their messages to an exchange that then forwards them to the subscriber(s). What would be the most suitable exhange type to implement this?
Is it possible to subscribe to a group of topics? For example if we have several topics starting by "data", like data_1, data_2 etc, would it be possible to have consumers subscribe to "data*" ? I didn't find anything on this but from what I gathered on how exchanges work it doesn't seem possible.
Is it possible to have sub-topics, and even sub-sub-sopics?
From what I understand so far, using a topic or header exchange, sub-topics and sub-sub-topics would basically be parts of the routing key, like data.diag.signal for example.
I hope I made myself clear enough and I thank you in advance for your answers.
Will use Java API examples in my answer.
Is it possible for a consumer to subscribe to a group of publishers?
RMQ consumers are not aware of publishers. The entities between them are queues (and exchanges, from producer(publisher) side).
The Java API com.rabbitmq.client.Channel.basicConsume(String, Consumer) allows for consumption from one queue only. But you could pass the same Consumer callback to multiple queues
An alternative would be to create per-consumer queue, and use channel.queueBind to bind it to exchanges producing interesting messages. Or even easier, have them publish to some generic exchange and bind to that one.
Is it possible to subscribe to a group of topics? For example if we have several topics starting by "data", like data_1, data_2 etc, would it be possible to have consumers subscribe to "data*" ? I didn't find anything on this but from what I gathered on how exchanges work it doesn't seem possible.
Not explicitly, as you need to pass in the queue name.
But you can work around with exchange routing:
If your current state is:
producer1 -> exchange-data-1 -> queue-data-1
producer2 -> exchange-data-2 -> queue-data-2
producer3 -> exchange-data-3 -> queue-data-3
You could also set up a new exchange
producer1 -> exchange-data-1 --\
producer2 -> exchange-data-2 -> exchange-data-all -> queue-per-consumer -> your-consumer
producer3 -> exchange-data-3 --/
Is it possible to have sub-topics, and even sub-sub-sopics? From what I understand so far, using a topic or header exchange, sub-topics and sub-sub-topics would basically be parts of the routing key, like data.diag.signal for example.
There is no sub-topic concept in RMQ, but, as you had mentioned, you can use topic (or even headers, for richer designs) exchanges.
Related
I would like to confront my understanding of google pubSub/lite vs RabbitMQ (using MQTT over WSS).
My use case is that I need something like a topic exchange. To send messages individually or to all or to some.
Having RabbitMQ I understand that I can create a topic have multiple queues linked via routingKey. E.g. amqTopic.routingKey1-10.
And I can push a message to a specific queue e.g. like this amqTopic.routingKey8
or push to the entire topic(all queues routed) like this amqTopic.*
Is it possible to create topic exchange structure with Google PubSub and if so how? I am not sure if I miss something. But from what I read I am inclined to say no, because google works like a direct exchange.
Thank you for helping..
This kind of topic exchange structure is possible to re-create using Cloud Pub/Sub filters. You can attach attributes to your messages when they are published (e.g. "routingKey": "8" or "routingKey": "all") and configure filters on your subscriptions to receive only messages meant for a particular routing key (attributes.routingKey="8" OR attributes.routingKey="all" in this scenario).
It's not currently possible to create this kind of topic exchange structure in Pub/Sub Lite.
Here's an example:
TYPE : TOPIC
exchange.v1 -> queue.order
exchange.v2 -> queue.log
so when the apps running it's must configure the exchange first right? and in a single service only can have 1 exchange?
I have 1 service for logging and 1 service for ordering. all proses will be sent into logging service and then forward another event. in this case to queue.order
So it's possible to publish an event from a different exchange? or I miss something? please let me know :(
Exchanges are not tied to “services”, much less in a 1:1 manner.
Exchanges in RabbitMQ are message sinks. Any existing exchanges can be published to by any number of applications (“services”) with adequate permissions.
Exhanges can either be pre-deployed or created automatically by an application. Pre-deployment is usually more common. This may or may not be outside the lifecycle of a single “service”.
Exchanges (depending on type) may also route to any number of queues on the same vhost.
Now, with all of that out of the way..
It is very possible to forward a message from a queue to another exchange: read from queues (stores), publish to exchanges (sinks). This can be done in code or even from a tool like the Shovel plugin - the “correct” approach depends significantly based on semantics, just as the choice of routing.
Personally, I recommend keeping RabbitMQ processing chains to as limited a scope as allowed by the application domain.
I have a use-case where the producers publish some events and the consumers are initially bound to the producers' exchanges. Over time, certain conditions change and the binding or the routing keys (for consumers) needs to be updated, since their subscription interests have changed. Can I handle this dynamic routing scenario in RabbitMQ?
Yes, this is fairly straightforward.
RabbitMQ provides the Queue.Bind and Queue.Unbind methods. When your consumers wish to receive information about a particular topic, they would Bind the queue to that topic via a topic exchange. When they are no longer interested in a particular topic, the binding would be removed for that topic using Unbind.
As per my understanding, all the functionality or use-cases of direct and fanout exchanges can be achieved using topic exchange. Topic exchange supports superset of functionality. So the question is, why does RabbitMQ have direct and fanout exchanges? Are there any use-cases that can be achieved using direct/fanout exchange but not with topic exchange?
I would say it's a matter of simplification. If all you need is, say, a topology where each specific routing key maps 1:1 with a queue (round robin distribution of tasks between multiple workers within the same application is an example cited in the RabbitMQ doc), then Direct exchanges may be easier to use and work with, and all you need. That's not to say you couldn't accomplish the same thing using Topic Exchanges (you could).
Similarly, with Fanout exchanges, if you have a situation that calls for the simply broadcasting of messages, you may find a Fanout exchange is easier to work with. Again, that's not to say you couldn't accomplish the same thing using Topic Exchanges (you could).
I generally use Topic exchanges exclusively, because I value the flexibility they provide. As an application expands, they can handle a greater variety of use cases within the same exchange, whereas that may not be the case with the other two types. Thus I can avoid the potential for having to change the topology midstream as an application grows.
As the RabbitMQ doc states about Topic exchanges:
Topic exchanges have a very broad set of use cases. Whenever a problem
involves multiple consumers/applications that selectively choose which
type of messages they want to receive, the use of topic exchanges
should be considered.
For more information about these concepts, including illustrations, this page has quite a bit of info:
https://www.rabbitmq.com/tutorials/amqp-concepts.html
You could replace the functionality of direct and fanout with a topic exchange, but you could also implement a big "Animal" class and don't bother implementing the "cat" and "dog" ones…
The different exchange types offer specificity according to your needs. You could do broadcast with the topic exchange, but then you need to force every client to know about the meaning of # and require them to use that routing key when binding; or just use a fanout exchange.
On the implementation side, implementing a topic exchange is a bit more complex, require a data structure that asks for more book-keeping than a plain fanout or direct exchange.
In the queue I have pushed 10K objects. Timestamp is one of the attribute in object. So, how can I write a consumer code using spring amqp?
can anyone help me on this.
AMQP, unlike JMS, has no notion of message selection for consumers. One solution is to use a topic exchange and set the routing key - let's say consumer 1 binds his queue to the exchange with foo.bar a second one binds with foo.baz; and a third binds with foo.*. The third will get all messages (with routing keys starting with foo.); the others will only get messages with their respective keys.
A direct exchange could also be used; it requires a complete match on the routing key.
You should probably work through all the RabbitMQ tutorials to understand the different exchange types before asking more questions here.