The document said
Published messages are routed to a receive endpoint queue by message type, using exchanges and exchange bindings. A service's receive endpoints do not affect other services or their receive endpoints, as long as they do not share the same queue.
As I know, create one ReceiveEndpoint like below will then create one exchange and one queue with the same name (e.g. some-queue), and will bind this exchange to the message type's exchange.
services.AddMassTransit(x =>
{
x.AddConsumer<EventConsumer>();
x.UsingRabbitMq((ctx, cfg) =>
{
cfg.ReceiveEndpoint("some-queue", e =>
{
e.ConfigureConsumer<EventConsumer>(ctx);
});
});
});
However, I don't get the point why bother have additional "some-queue" exchange. Any example usecase will be helpful.
I cover the reasons for the topology in several videos, including this one.
I was looking for the answer to this question myself. For the benefit of the answer I paste some revised quotes of the linked video from Chris here:
MassTransit has done this since the very first versions.
If you would want to to send directly to a queue you would have to either:
Specify a blank exchange name and set the routing key equal to the queue name.
or
Send to an exchange that's bound to the queue.
You can't send to a queue directly.
[...]
When we looked at the topology for the broker for MassTransit, the approach we took is to create an exchange with the same name as the queue. This gives us some actually really cool features:
Let's say I want to keep a copy of every message sent to my endpoint. I can do that by just creating another queue and binding it to that exchange. That lets me do like a wiretap which is actually a messaging pattern.
[...]
When you're troubleshooting and ask yourself: "Why didn't the service do what I suspected?": With this I can go at that queue and then I could go
to my wiretap queue and look at every message that was received. This is a really cool way to kind of steal traffic and look at it and figure out what's going on.
Related
I have a hard time understanding the basic concepts of RabbitMQ. I find the online documentation not perfectly clear.
So far I understand, what a channel, a queue, a binding etc. is.
But how would the following use case be implemented:
Use Case: Sender posts to one exchange with different topics. On the receiver side, depending on the topic, different receivers should be notified.
So the following should somehow be feasible with a topic exchange:
create a channel
within this channel, create a topic exchange
for each topic to be subscribed to, create a queue and a queue binding with this topic as property
My difficulty is that the callback would be related to the channel, not to the queue or the queue binding. I am not 100 % sure if I am right here.
So that's my question: in order to have multiple callbacks, IOW: different message handlers, depending on the subscribed topic - do you have to create multiple channels, one for each "different message handling"? All these channels should grab the same exchange and define their own queue + queue binding for that specific topic?
Please confirm if this is correct or if I am straying from the canonic path of AMPQ ... "queue" sounds so light-weight, so I intuitively thought of a queue or a queue binding as the right point to attach a consuming event handler to, but it seems that, instead, channel is my friend in this. Right?
Another aspect of my question:
If I really have to use multiple channels for this, do I have to declare the same exchange (exchange name and exchange type of "topic") for each channel? I hoped there was something like:
define the exchange with this name and the type of "topic" once
for each channel, "grab" this predefined exchange and use it by adding queues and queue bindings to this exchange
I find it helpful to think about the roles of the broker (RabbitMQ) and the clients (your applications) separately.
The broker, RabbitMQ, will receive messages from your publishers, route them to queues, and eventually send them to consumers. The message routing can be simple or complex. In your case, the routing is topic based with a few different queues.
You haven't said much about the publishers, likely because their job is simple. They send messages with a routing key to RabbitMQ.
The consumer side is where things can get interesting. At the simplest level, a consumer subscribes to a queue, receives messages from RabbitMQ, and processes them. The consumer opens a connection to RabbitMQ and will use a channel for a particular use (e.g., subscribing to a queue). The power of message brokers is that they allow designers to break up processes into separate apps if desired.
You don't give much insight into your application, other than the presence of different message topics. An important design choice for you to make is how to define the application(s). Are the different topics suitable for separate applications, or will a single application handle all types of messages.
For the former case, you would have one application for each queue. A single channel that subscribes to the queue is probably the most sensible decision unless your application needs to be threaded. For threaded applications, each thread would have its own channel and all threads can be subscribed to the same queue. Each application would have its own callback function for processing that type of message.
For the latter case (single application with multiple queues), the best approach would be to have at least one channel per queue. It sounds like each queue would require its own callback function, and you would assign the functions to the channels according to its subscription. You might have multiple channels per queue if your application can process multiple messages (of each topic) simultaneously.
Regarding your question about declaring exchanges, queues, and bindings, these items only need to be created once. But it is reasonable practice to have your clients declare them at connection time. Advantages of declaring them are that they will be created again if they were deleted and that any discrepancies between your declaration and what is on the broker will trigger errors.
I want to create a UI to see all the messages that are flowing through all exchanges in RabbitMQ server (of course other than the management console).
I am also using Mass Transit over rabbit but i am not sure if this matters.
Is this at all possible without having to code a consumer for each one of them one by one? If yes, any starting points?
The message exchanges used for publishing, as well as sending, are all bound to an exchange that has the same name as the queue for message delivery. So you could bind your own wire tap exchange on the broker to any queue exchange, and wiretap the messages to another queue of your choosing.
You can view the RabbitMQ topology layout in the documentation.
It was specifically done this way to make it easy to wiretap any endpoint, since all messages flow through a single fanout exchange.
This is a pretty broad question because it's not entirely obvious what you mean by "see", but regardless, you could create an observer on your bus. It's documented here and I think it's fairly straightforward: https://masstransit-project.com/MassTransit/usage/observers.html
In the observer you can handle various events when any message hits the MT message bus, and perform some kind of operation (like print the message, add logging, metrics, etc). If you have a microservice scenario it might be a good idea to add an observer to your shared library and add it to the bus in your individual applications.
I've been trying to find a way to CC messages from a Qpid Exchange to another Queue for testing/monitoring purposes. I noticed that a RabbitMQ user out there was having a similar problem, and the solution seemed to be RabbitMQ's Firehose feature. Is there a similar solution in Qpid?
Here's some more details for the curious.
Let's call the Exchange "App.Ex", and through it are flowing messages for a single other intended recipient (let's call him "Bob")
I connect to App.Ex, initiate a session, start a receiver, and start fetching (using code adapted from the QPID Book's "A Simple Messaging Program in Python")
I start seeing the messages I want to see. HOWEVER, in doing so I've robbed Bob of the messages he needs!
So there's the rub, how can I get the messages CC'd to me, but in a way that Bob still receives his messages?
I have permission to modify the messaging configuration, so I can create my own Queues and Exchanges if need be. Thoughts appreciated!
A direct exchange is probably most appropriate because you can have some queues with CC like behavior and some without, and you can change it anytime on a live exchange.
You can have two queues bound to the same subject/routing key. When a message is sent to the exchange with that particular subject/routing key, both bound queues will receives copies of the same message.
Both queues bar1 and bar2 are bound to routing_key foo. When producer B posts messages to the exchange with routing_key = foo, both bar1 and bar1 receive copies of all messages.
Ask if you need commands for creating the exchange and appropriate bindings.
However there are more ways to do the same thing:
You can also achieve the similar behavior using a topic queue, with exact matches on topic name
Lastly, you can also use a fanout exchange, where any message you send to the queue, a copy is routed to all the queues bound to the exchange.
Note that all of these exchange types are from the AMQP spec, so they are not qpid specific, you could do the same thing or something very similar in any AMQP implementation.
I have a code which has both consumer and producer. I want to differentiate or find the exact exchange name through which the consumer has consumed the message. For example, I have almost 5 exchanges and I want to know through which exchange out of that 5 the consumer has received it's message. How can this be achieved?
I have done lot of homework but couldn't find a solution.
Messages are consumed from queues, not exchanges.
The way to figure out original exchange that message was published to is to use Firehose Tracer plugin (maybe even with rabbitmq-tracing
plugin alongside).
Alternatively, you may figure out original exchange by comparing queues bindings with message routing key. This usually work well in most cases, unless you have really wired publishers and routing logic.
P.S.: finally, if you have at least read access to publishers code you can figure out where each messages goes from.
Rebus has flexible system that allows me to specify different endpoints for different message types, either in web.config or by implementing a custom IDetermineMessageOwnership.
As far as I can tell, message ownership is represented simply by a string. Using the MSMQ transport, this string points to a queue to which the message is delivered. With RabbitMQ, the string is used as a topic for the message, which is then delivered to a generic exchange named "Rebus". Rebus is a nice fellow, so he also sets up a queue in the RabbitMQ server, using the same name, and makes a binding from topic to queue within the Rebus exchange.
My question is this: Is it possible to have Rebus not create queues and bindings, but still deliver the messages to an exchange with a relevant topic set for each message?
Declaring the queues and bindings manually will allow me to set up an awesome topic exchange, using bindings with wildcards and what not. Here is a nice illustration of a topic exchange with funky bindings, just to make my question look more sleek and sexy:
Sounds to me like you want to do something like this:
Configure.With(yourFavoriteContainer)
.Transport(t => t.UseRabbitMq(...)
.ManageSubscriptions()) //< BAM!!1
.(...)
which lets Rebus take advantage of the fact that Rebus' RabbitMqMessageQueue implements IMulticastTransport, which in turn turns handling of all things multicast over to Rabbit.
It's just important that all of your Rabbit-enabled Rebus endpoints agree on letting Rabbit ManageSubscriptions - otherwise, weird stuff might happen ;)
It means that
when you bus.Subscribe<SomeEvent>, you bind a topic with the type name to the subscriber's input queue - e.g. "SomeEvent.SomeNamespace" -> myInputQueue
publishers publish events on a topic that is the type name - e.g. "SomeEvent.SomeNamespace"
message ownership is disregarded when subscribing
Rabbit will do the heavy lifting when doing multicast (which is what Rabbit users are mostly doing)
If you require even more flexibility, you can even take responsibility of deciding the topic to publish to for each .NET type, like so:
Configure.With(yourFavoriteContainer)
.Transport(t => t.UseRabbitMq(...)
.ManageSubscriptions()
.AddEventNameResolver(type => DecideTopic(type))
.(...)
You can add multiple event name resolvers if you want - they will be run in sequence until one of them returns something that is not null.
Does it make sense?