I understand that Rebus is perfectly capable of transporting messages from point A to B (using MSMQ as the transport layer). To make things perfectly clear, is Rebus also capable of doing one-to-many messaging, i.e. messages sent from point A should end at both point B and C?
And if it is possible, how does it do it? I cannot see any centralised distribution site (a post-office), so I assume that the communication will consist of a channel from every endpoint to every other endpoint (so that in a network where a process has to communicate with 5 other endpoints, there will be 5 channels radiating out of this process). Can you confirm this assumption?
Yes, Rebus is indeed capable of publishing messages to virtually any number of subscribers. It's true that MSMQ (at least in its most basic mode of operation) is a simple point-to-point channel, which is why there's a layer on top in order to implement true pub/sub.
The way it works, is that each subscriber has an endpoint mapping pointing to the publisher, and then each subscriber goes
bus.Subsribe<SomethingInterestingHappened>();
which causes an internal SubscriptionMessage to be sent the the publisher. The publisher must then remember who subscribed to each given message type, typically by storing this information in an SQL Server. All this happens automatically, it just requires that you configure some kind of subscription storage.
And then, when the time comes to publish something, the publisher goes
bus.Publish(new SomethingInterestingHappened { ... });
which will make Rebus look up all the subscribers of the given message type. This may be 0, 1 or more, and then the event will be sent to each subscriber's input queue.
You can read more about these things in the Rebus docs on the page about routing.
To give you a hint on how subscribers and publishers might be configured, check this out - this is a subscriber:
Configure.With(...)
.Transport(t => t.UseMsmq....)
.MessageOwnership(t => t.FromRebusConfigurationSection())
(...)
which also has an endpoint mapping that maps a bunch of events to a specific publisher:
<endpoints>
<add messages="SomePublisher.Messages" endpoint="publisher_input_queue" />
</endpoint>
and then the publisher might look like this:
Configure.With(...)
.Transport(t => t.UseMsmq....)
.Subscriptions(s => s.StoreInSqlServer(theConnectionString, "subscriptions")
.EnsureTableIsCreated())
(...)
Related
As I have been able to verify, in MassTransit with Azure Service Bus, each type of object consumed by a "Consumer" generates a Topic for that type regardless of whether it is only consumed in a specific "receive endpoint" (queue). When sending a message of this type with the "Send()" method, the message is sent directly to the "receive endpoint" (queue) without going through the topic. If this same message is published with the "Publish()" method, it is published in the Topic, and is forwarded to the receive endpoint (queue) from the corresponding subscriber.
My application uses a CQRS pattern where the messages are divided into commands and events. Commands use the send-receive pattern and are therefore always dispatched in MassTransit with the "Send()" method. The events, however, are based on the publish-subscribe pattern, and therefore are always dispatched in MassTransit with the "Publish()" method. As a result, a large number of topics are created on the bus that are never used (one for each type of command), since the messages belonging to these topics are sent directly to the receiver's queue.
For all these reasons, the question I ask is whether it is possible to configure MassTransit so that it does not automatically create the topics of some types of messages consumed because they will only be sent using the "Send()" method? Does this make sense in MassTransit or is it not possible/recommended?
Thank you!
Regards
Edited 16/04/2021
After doing some testing, I edit this topic to clarify that the intention is to configure MassTransit so that it does not automatically create the topics of some types of messages consumed, all of them received on the same receive endpoint. That is, the intention is to configure (dynamically if possible, through the type of object) which types of messages consumed create a topic and which do not in the same receive endpoint. Let's imagine that we have a receive endpoint (a queue) associated with a service, and this service is capable of consuming both commands and events, since the commands are only dispatched through Send(), it is not necessary to create the topic for them, however the events that are dispatched via Publish(), they need their topic (and their subscribers) to exist in order to deliver the message and be consumed.
Thanks in advance
Yes, for a receive endpoint hosting a consumer that will only receive Sent messages, you can specify ConfigureConsumeTopology = false for that receive endpoint. You can do that via a ConsumerDefinition, or when configuring the receive endpoint directly.
UPDATE
It is also possible to disable topology configuration per message type using an attribute on the message contract:
[ConfigureConsumeTopology(false)]
public interface SomeCommand
{
}
This will prevent the topic/exchange from being created and bound to the receive endpoint.
While I can understand the desire to be "pure to the CQRS mantra" and only Send commands, I'd suggest you read this answer and take it into consideration before overburdening your developers with knowing every single endpoint in the system by name...
Requirement
A system undergoes some state change, and multiple other parts of the system has to know this(lets call them observers) so that they can perform some actions based on the current state, the actions of the observers are important, if some of the observers are not online(not listening currently due to some trouble, but will be back soon), the message should not be discarded till all the observers gets the message.
Trying to accomplish this with pub/sub model, here are my findings, (please correct if this understanding is wrong) -
The publisher creates an event on specific topic, and multiple subscribers can consume the same message. This model either provides no delivery guarantee(in redis), or delivery is guaranteed once(with messaging queues), ie. when one of the consumer acknowledges a message, the message is discarded(rabbitmq).
Example
A new Person Profile entity gets created in DB
Now,
A background verification service has to know this to trigger the verification process.
Subscriptions service has to know this to add default subscriptions to the user.
Now both the tasks are important, unrelated and can run in parallel.
Now In Queue model, if subscription service is down for some reason, a BG verification process acknowledges the message, the message will be removed from the queue, or if it is fire and forget like most of pub/sub, the delivery is anyhow not guaranteed for both the services.
One more point is both the tasks are unrelated and need not be triggered one after other.
In short, my need is to make sure all the consumers gets the same message and they should be able to acknowledge them individually, the message should be evicted only after all the consumers acknowledged it either of the above approaches doesn't do this.
Anything I am missing here ? How should I approach this problem ?
This scenario is explicitly supported by RabbitMQ's model, which separates "exchanges" from "queues":
A publisher always sends a message to an "exchange", which is just a stateless routing address; it doesn't need to know what queue(s) the message should end up in
A consumer always reads messages from a "queue", which contains its own copy of messages, regardless of where they originated
Multiple consumers can subscribe to the same queue, and each message will be delivered to exactly one consumer
Crucially, an exchange can route the same message to multiple queues, and each will receive a copy of the message
The key thing to understand here is that while we talk about consumers "subscribing" to a queue, the "subscription" part of a "pub-sub" setup is actually the routing from the exchange to the queue.
So a RabbitMQ pub-sub system might look like this:
A new Person Profile entity gets created in DB
This event is published as a message to an "events" topic exchange with a routing key of "entity.profile.created"
The exchange routes copies of the message to multiple queues:
A "verification_service" queue has been bound to this exchange to receive a copy of all messages matching "entity.profile.#"
A "subscription_setup_service" queue has been bound to this exchange to receive a copy of all messages matching "entity.profile.created"
The consuming scripts don't know anything about this routing, they just know that messages will appear in the queue for events that are relevant to them:
The verification service picks up the copy of the message on the "verification_service" queue, processes, and acknowledges it
The subscription setup service picks up the copy of the message on the "subscription_setup_service" queue, processes, and acknowledges it
If there are multiple consuming scripts looking at the same queue, they'll share the messages on that queue between them, but still completely independent of any other queue.
Here's a screenshot from this interactive visualisation tool that shows this scenario:
As you mentioned it is not something that you can control with Redis Pub/Sub data structure.
But you can do it easily with Redis Streams.
Streams will allow you to post messages using the XADD command and then control which consumers are dealing with the message and acknowledge that message has been processed.
You can look at these sample application that provides (in Java) example about:
posting and consuming messages
create multiple consumer groups
manage exceptions
Links:
Getting Started with Redis Streams and Java
Redis Streams in Action ( Project that shows how to use ADD/ACK/PENDING/CLAIM and build an error proof streaming application with Redis Streams and SpringData )
I have read many different definitions of ESB (enterprise service bus) and it is not clear for me.
Here is my own definition: An ESB is an architecture and not a tool that allows heterogeneous applications to communicate with each other through a BUS. The particularity of an ESB is that it can have producers and consumers. For example, a producer can send a message to a topic/queue inside the bus and three consumers who are subscribers will receive the same message, so it avoids point-to-point flows.
The second particularity of the ESB is that it allows managing the security and logs in one place as everything goes inside the ESB.
I've also heard about "routes" that set rules in moving a message (with Talend ESB), but I don't really see the point (if you have any examples I'm interested). And of course, Web services can be created to expose data. These services must be scalable and resistant to "Single Point of Failure".
I created an architecture and would have liked to know if it's an ESB architecture.
(I made a mistake on my draw, it's not a Queue but a Topic!)
The steps of the process above:
Producer: it listens the changes (update, insert, ...) in different databases and as soon as there is a change, it retrieves the data and sends it to the queue.
Queue: The queue contains all the messages sent by the producer and will send them to the consumers.
Consumers: Consumers will make the data quality and insert the new data into a database.
For me, this architecture respects ESB because activeMQ acts like a bus. He acts here as mediator. What do you think ?
I think you are on the right track. However, I think there is an important distinction to make sure each message flow is using different queues. It is generally a best practice to have a queue per-message type.
The message flows can all co-exist on the same broker infrastructure, allowing you to have higher density, better utilization, and the ability to wiretap message flows in one place as needed.
In your case:
Database A -> queue://A -> Consumer A
Database B -> queue://B -> Consumer B
Database C -> queue://C -> Consumer C
The undelying use case
It is typical pubsub use case: Consider we have M news sources, and there are N subscribers who subscribe to the desired news sources, and who want to get news updates. However, we want these updates to land up in mongodb - essentially maintain most recent 'k' updates (and can be indexed and searched etc.). We want to design for M to scale upto million publishers, N to scale to few millions.
Subscribers' updates are finally received and stored in more than one hosts and their native mongodbs.
Modeling in rabbitmq
Rabbitmq will be used to persist the mappings (who subscribes to which news source).
I have setup a pubsub system in this way: We create publisher exchanges (each mapping to one news source) and of type 'fanout'.
For modelling subscribers, there are two options.
In the first option, have one queue for each subscriber bound to relevant publisher exchanges. And let the client process open connections to all these subscriber queues and receive the updates (and persist them to mongodb). Note that in this option, when the client is restarted, it has to manage list of all susbcribers, and open connections to all subscriber queues it is responsible for.
In the second option, we want to be able to remove overhead of having to explicitly open on each user queue upon startup. Instead, we want to listen to only one queue - representative of all subscribers who will send updates to this client host.
For achieving this, we first create one exchange for each subscriber and let it bind to the publisher exchange(s) that it follows. We let a single queue for each client, and let the subscriber exchange bind to this queue (type=direct) if the subscriber belongs to that client.
Once the client receives the update message, it should come to know which subscriber exchange it came from. Only then we can add it to mongodb for relevant subscriber. Presumably the subscriber exchange should add this information as a new header on the message.
As per rabbitmq docs, I believe there is no way to get achieve this. (Or more specifically, to get the 'delivery path' property from the delivered message, from which we can get this information).
My questions:
Is it possible to add a new header to message as it passes through exchange?
If this is not possible, then can we achieve it through custom exchange and relevant plugin? Any plugin that I can readily use for this purpose?
I am curious as to why rabbitmq is not providing delivery path property as an optional configuration?
Is there any other way I can achieve the same? (See pubsubhubbub note below)
PubSubHubBub
The use case is very similar to what pubsubhubbub protocol provides for. And there is rabbitmq plugin too called rabbithub. However, our system will be a closed system, and I believe that the webhook approach of the protocol is going to be too much of overhead compared to listening on single queue (and from performance perspective.)
The producer (RMQ Client) of the message should add all the required headers (including the originator's identity) before producing (publishing) it on RMQ. These headers are used for routing.
If, while in transit, the message (including headers) needs to be transformed (e.g. adding new headers), it needs to be sent to the transformer (another RMQ Client). This transformer will essentially become the new publisher.
The actual consumer should receive its intended messages (for which it has subscribed to) through single queue. The routing of all its subscribed messages should be arranged on the RMQ Exchange.
Managing the last 'K' updates should neither be the responsibility of the producer nor the consumer. So, it should be done in the transformer. Producers' messages should be routed to this transformer (for storage) before further re-routing to exchange(s) from where consumers consume.
Rebus has flexible system that allows me to specify different endpoints for different message types, either in web.config or by implementing a custom IDetermineMessageOwnership.
As far as I can tell, message ownership is represented simply by a string. Using the MSMQ transport, this string points to a queue to which the message is delivered. With RabbitMQ, the string is used as a topic for the message, which is then delivered to a generic exchange named "Rebus". Rebus is a nice fellow, so he also sets up a queue in the RabbitMQ server, using the same name, and makes a binding from topic to queue within the Rebus exchange.
My question is this: Is it possible to have Rebus not create queues and bindings, but still deliver the messages to an exchange with a relevant topic set for each message?
Declaring the queues and bindings manually will allow me to set up an awesome topic exchange, using bindings with wildcards and what not. Here is a nice illustration of a topic exchange with funky bindings, just to make my question look more sleek and sexy:
Sounds to me like you want to do something like this:
Configure.With(yourFavoriteContainer)
.Transport(t => t.UseRabbitMq(...)
.ManageSubscriptions()) //< BAM!!1
.(...)
which lets Rebus take advantage of the fact that Rebus' RabbitMqMessageQueue implements IMulticastTransport, which in turn turns handling of all things multicast over to Rabbit.
It's just important that all of your Rabbit-enabled Rebus endpoints agree on letting Rabbit ManageSubscriptions - otherwise, weird stuff might happen ;)
It means that
when you bus.Subscribe<SomeEvent>, you bind a topic with the type name to the subscriber's input queue - e.g. "SomeEvent.SomeNamespace" -> myInputQueue
publishers publish events on a topic that is the type name - e.g. "SomeEvent.SomeNamespace"
message ownership is disregarded when subscribing
Rabbit will do the heavy lifting when doing multicast (which is what Rabbit users are mostly doing)
If you require even more flexibility, you can even take responsibility of deciding the topic to publish to for each .NET type, like so:
Configure.With(yourFavoriteContainer)
.Transport(t => t.UseRabbitMq(...)
.ManageSubscriptions()
.AddEventNameResolver(type => DecideTopic(type))
.(...)
You can add multiple event name resolvers if you want - they will be run in sequence until one of them returns something that is not null.
Does it make sense?