From my reading around rabbitmq and AMQP, most exchanges, queues and bindings are done by the application e.g.
App1(Producer) declares exchange1
App2(Consumer) declares queue1 and binds to exchange1
I have seen then in order not to have missing messages that both should declare additional config, so now would be:
App1(Producer) declares exchange1, declares queue1 and binds to exchange1
App2(Consumer) declares exchange1, declares queue1 and binds to exchange1
Now the Producer App needs to know details it never needed to known.
Is this acceptable? It feels wrong
For the cosumer to be able to bind an exchange to a queue he wants to consume messages from, the exchange must exist and have the expected type (fanout, direct, topic) and configuration.
Consider a situation where the consumer would rely on the exchange being created by somebody else. If he would just create the queue and then try to bind it to the missing exchange, he would get an error.
In a scenario where specificicly configured things (exchange, queue, binding) must exist before a consumer can do his part of the setup, you have three options:
Configure the required things by a third party that is guaranteed to finish before producers and consumers run from application code.
Configure the required things by the producer or the consumer and make sure this part of the application code is run first.
Let the producer and the consumer perform identical steps to configure the required things. This makes the order in which the different parts of the application code are executed irrelevant.
The third option works if the configuration steps are identical in all aspects. For example, RabbitMQ accepts the repeated creation of an exchange, if this configuration always is identical. It throws an error, if an existing exchange is configured again with differing configuration.
I would say the the third option is advisiable for a scenario where the application code does the setup. It feels right to me.
Related
Here's an example:
TYPE : TOPIC
exchange.v1 -> queue.order
exchange.v2 -> queue.log
so when the apps running it's must configure the exchange first right? and in a single service only can have 1 exchange?
I have 1 service for logging and 1 service for ordering. all proses will be sent into logging service and then forward another event. in this case to queue.order
So it's possible to publish an event from a different exchange? or I miss something? please let me know :(
Exchanges are not tied to “services”, much less in a 1:1 manner.
Exchanges in RabbitMQ are message sinks. Any existing exchanges can be published to by any number of applications (“services”) with adequate permissions.
Exhanges can either be pre-deployed or created automatically by an application. Pre-deployment is usually more common. This may or may not be outside the lifecycle of a single “service”.
Exchanges (depending on type) may also route to any number of queues on the same vhost.
Now, with all of that out of the way..
It is very possible to forward a message from a queue to another exchange: read from queues (stores), publish to exchanges (sinks). This can be done in code or even from a tool like the Shovel plugin - the “correct” approach depends significantly based on semantics, just as the choice of routing.
Personally, I recommend keeping RabbitMQ processing chains to as limited a scope as allowed by the application domain.
In all the examples I find online, I see the exchange and the queue being declared before a messages are consumed.
Declaring the exchange seems weird, because, why would I do it? I'm consuming a queue, which might be bound to multiple exchanges (or to none, maybe it just have old messages waiting in it).
also, I can't think of why I would declare a queue. This will require me to know information about the queue that I don't need to know to consume it (like auto_delete and durability).
When I tested it locally, I can consume a queue without declaring anything. It works. So I'm left wondering, why does every example I've seen online, declare the exchange and queue, even if it just consumes it?
thanks!!!
"All" the example you saw are self contained. And they trying to give you a working example. Because in case you don't have all the components setted up, your example will fail.
In terms of "why I would declare a queue". The real-life example is when your consumer wants to consume messages that are relevant with the current configuration. In this case it will create an exclusive ( no one else can connect to this queue ) and will start consume messages.
Back to your answer. No you don't need to do this. You can pre-create exchange, binding and queue ahead of time and then just pass the names to the code.
In general, you don’t need declare exchange and queue in consumer. You must assemble "exchanges/queues" topology somewhere else. It's like schema in database.
But always there are exceptions.
When you need "private" queue (exclusive=true) for real-time processing, consumer must know (by configuration) about source exchange and bind own queue to it.
In other case i can imagine situations where publisher declare exchange and consumers can discover it using some convention (pattern) for exchange naming.
Rebus has flexible system that allows me to specify different endpoints for different message types, either in web.config or by implementing a custom IDetermineMessageOwnership.
As far as I can tell, message ownership is represented simply by a string. Using the MSMQ transport, this string points to a queue to which the message is delivered. With RabbitMQ, the string is used as a topic for the message, which is then delivered to a generic exchange named "Rebus". Rebus is a nice fellow, so he also sets up a queue in the RabbitMQ server, using the same name, and makes a binding from topic to queue within the Rebus exchange.
My question is this: Is it possible to have Rebus not create queues and bindings, but still deliver the messages to an exchange with a relevant topic set for each message?
Declaring the queues and bindings manually will allow me to set up an awesome topic exchange, using bindings with wildcards and what not. Here is a nice illustration of a topic exchange with funky bindings, just to make my question look more sleek and sexy:
Sounds to me like you want to do something like this:
Configure.With(yourFavoriteContainer)
.Transport(t => t.UseRabbitMq(...)
.ManageSubscriptions()) //< BAM!!1
.(...)
which lets Rebus take advantage of the fact that Rebus' RabbitMqMessageQueue implements IMulticastTransport, which in turn turns handling of all things multicast over to Rabbit.
It's just important that all of your Rabbit-enabled Rebus endpoints agree on letting Rabbit ManageSubscriptions - otherwise, weird stuff might happen ;)
It means that
when you bus.Subscribe<SomeEvent>, you bind a topic with the type name to the subscriber's input queue - e.g. "SomeEvent.SomeNamespace" -> myInputQueue
publishers publish events on a topic that is the type name - e.g. "SomeEvent.SomeNamespace"
message ownership is disregarded when subscribing
Rabbit will do the heavy lifting when doing multicast (which is what Rabbit users are mostly doing)
If you require even more flexibility, you can even take responsibility of deciding the topic to publish to for each .NET type, like so:
Configure.With(yourFavoriteContainer)
.Transport(t => t.UseRabbitMq(...)
.ManageSubscriptions()
.AddEventNameResolver(type => DecideTopic(type))
.(...)
You can add multiple event name resolvers if you want - they will be run in sequence until one of them returns something that is not null.
Does it make sense?
I have exactly 2 types of messages that I want to be sent via RabbitMQ. So I have 2 options how I can do this:
sent a message to default empty-named exchange with routing_key corresponding to the queue name
use direct exchange's routing_key parameter corresponding to consumer's routing_key parameter in queue binding
So which option is preferable and why?
A default exchange is a direct exchange. RabbitMQ creates the default exchange by default, but it uses an empty string for the name. If you look at the RabbitMQ AMQP concepts page, under Default Exchange:
The default exchange is a direct exchange with no name (empty string)
pre-declared by the broker.
You can see this by running rabbitmqctl list_exchanges as well:
direct
Foo direct < Same thing as the above
amq.direct direct
amq.fanout fanout
...and so on
As far as I'm aware, there isn't any benefits of using one over the other. I would stick with the default exchange if you only need to route based on routing keys.
Let's say you direct-bind to an exchange broadcasting logs to routing keys of "info", "warn", and "error". Using the default exchange, you would need to create three different queues with those names to receive all logs; and adjustments to which log levels you receive would require changing your queue declarations. By using a named exchange, you can simply change your queue's bindings and continue processing things as normal.
In short, it provides one extra level of abstraction.
As I see it, the default direct exchange give the possibility for the consumers and the producers to not know about each other, by binding a queue (used by a consumer) to an exchange (used by a producer) implicitly using the queue's name.
I use the default direct-exchange for a specific case: the consumer and producers don't know about each other. In my case, each consumer have its proper queue. From the producer, I cannot know by advance which queues are going to be declared and used, as it depends on the consumers. So it is impossible to define the bindings between a custom direct-exchange and the queues on the producer side.
One way to solve it with a custom (user-defined) direct-exchange would be to define the binding-key on consumer side. But it would means to know about the producer from the consumer side as I need to know the exchange name used by the producer.
Therefore, automatically binding a queue by its name on the default direct-exchange makes it possible, in my case, to only declare a queue on consumer side and send message to it from producer by only knowing the queue's name.
Of course it implies to know the queue's name at runtime, when invoking the producer, as it would be required to know the binding-key of a custom direct-exchange (in my case, the queue's name is given by the application using the producer). But when configuring the broker, producers and consumers don't have to know about each other.
some description in rabbitmq web manager.
//default exchange 's Bindings
Default exchange
The default exchange is implicitly bound to every queue,
with a routing key equal to the queue name.
It is not possible to
explicitly bind to, or
unbind from the default exchange.
It also cannot be deleted.
I have to implement this scenario:
An external application publish message to rabbitmq.
This message has a client_id property. We can place this id to routing key or message header or some other property.
I have to implement sharding in a exchange routng logic - the message should be delivered to specific queue based on the client_id range.
Is it possible to implement in a standard exchanges?
If not what exchange should I take as the base?
How to dynamicly change client_id ranges?
Take a look at the rabbitmq plugin. It's included in the RabbitMQ distribution from v3.6.0 onwards.
Just have your producer put enough info into the routing key that causes the message to go into the right queue on the other side of the Exchange.
So for example, create two queues called 1 and 2 and bind them with routing keys matching the names. Then have your producer decide which routing key to use when producing the event message. Customers with names starting with letters a-m go to 1, n-z go to 2, you get the idea. It pushes the sharding to the producer but that might be OK for your application.
AMQP doesn't have any explicit implementation of sharding, but its architecture should help you to do that.
Spreading messages to several queues is just a rabbitmq challenge (and part of amqp specification), and with routing, way you can attach hetereogeneous consumers to handle specific messages routed via the same exchange. Therefore, producer should push a specific key to be consumed by specific queue/consumer...
You can decide to make a static sharding, perhaps you have 10 queues with one consumer per queue. You could implement a consistent hashing function such that key is CLIENT_ID % 10.
Another ways and none static solutions could be propoused, and you can try to over this architecture.