Background
We are trying to introduce a new architectural pattern in our company and are considering CQRS with Event Sourcing using a Service Bus. Technologies we are currently developing our POC with are NServiceBus, Event Store, and MSMQ. We would like to have a single endpoint in NServiceBus defined with two different transports, MSMQ for our commands and Event Store for our events. The current state of our enterprise does not permit us to easily switch everything to Event Store presently as we have significant investment in our legacy apps using MSMQ, which is a reason why we are considering the hybrid approach.
Question
Is it possible to create a single NServiceBus endpoint that uses different transports? If yes, how? If no, what alternatives are there?
Aaron,
I think the best option would be to use MSMQ as a transport in NServiceBus. Here's how it may look like:
send a command via MSMQ
in a command handler (re)create an aggregate which is the target of the command
invoke the operations
store the resulting events in the EventStore along with the command's message id to ensure idempotence. The aggregate itself will be responsible for knowing the commands it already processed
in a separate component (event processor) use EventStore persistent subscription APIs to hook to all events stream. Some of these processed events should cause sending a command. Such a command might be send via NServiceBus send-only endpoint hosted inside this event processor.
in that event processor you can also re-publish all events via NServiceBus and MSMQ. Such events should not be subscribed by other services (see note on autonomy below)
NServiceBus Sagas (process managers) should live inside your service boundary and react on commands and/or events sent or re-published by these event processors over MSMQ.
One remark regarding the service boundaries is that you have to decide what level of service autonomy suits you:
* Weak, where services can directly subscribe to other service event streams. In this design events that cross service boundary are obviously allowed to carry data.
* Strong, where services use higher-level events for communication and these events only carry the identity of things and no data. If you want something like this, you can use your event processors to map from ES events to these "higher level" events.
Related
as I was reading thru documentation on nservicebus, I wasnt able to find what is persisted under Persistence section.
If nservicebus is a loosely coupled distributed library sending self-contained messages, what is there to persist? I dont understand.
With web app, when a user has a Session, we may choose to persist the Session in SQL Server, in Memory or somehow else, but with nservicebus there is no session to persist.
So, what is actually the Persistence in nservicebus?
What sort of data that could be persisted and for what reason?
Transports like RabbitMQ and Azure Service Bus natively support publish/subscribe. If an endpoint wants to receive published events, a 'subscription' to those events is stored inside those queuing technologies. Other queuing technologies don't support publish/subscribe natively, like MSMQ and Azure Storage Queues. NServiceBus mimics the behavior, but needs to store those subscriptions somewhere else.
Other things we can store are timeout and deferred messages and saga state. A saga is kind of a state machine (a workflow) and this state needs to be stored somewhere. Another feature NServiceBus supports is the outbox, which removes the need for distributed transactions by putting the message transaction and business transaction in the same database.
If you only use certain features, some transports allow you to do that natively. This removes the need for a persister. Sagas and Outbox always need persistence.
I am getting my head around messaging. Currently we are spiking a few scenarios using Rebus. We are also considering NServiceBus.
The scenario we are trying to build is a proof of concept for a background task processing system. Today we have a handful of backend services hosted in different ways. (web, windows services, console apps) I am looking to hook them up to rebus and start consuming messages using competing consumer, some mesages will have one listener and some will share the load of messages. Elegant :)
I got a pretty good start from this other question How should I set rebus up for one producer and many consumers and it is working nicely in the proof of concept.
Now I want to start reporting progress. My intital approach is to set up pub/sub as well and spin up a service that listens to progress events from all the services. And if a service is interrested in a specific progress in the future it is easy to subscripe of interrest to the messages and start listening.
But how shall I approach setting up both competing consumer and pub/sub? it is dimply two separate things? (In the rebus case one adapter using UseSqlServerInOneWayClientMode / UseSqlServer and another adapter that is set up for the pub/sub using whatever protocol we want?)
Or is there a better solution then having two "buses" here?
I've built something like that myself a couple of times, and I've had pretty good results with using SignalR to report progress from this kind of backend worker processes.
Our setup had a bunch of WPF clients, one single SignalR hub, and a bunch of backend worker processes. All WPF clients and all backend workers would then establish a connection to the hub, allowing workers to send progress reports while doing their work.
SignalR has some nice properties that makes it very suitable for this exact kind of problem:
The published messages "escape" the Rebus unit of work, allowing progress report messages to be sent several times from within one single message handler even though it could take a long time to complete
It was easy to get the messages all the way to the clients because they subscribe directly
We could use the hub groups functionality to group users so we could target progress/status messages from the backend at either all users or a single user (could also be used for departments, etc.)
The most important point, I guess, is that this progress reporting thing (at least in our case) was not as important as our Rebus messages, i.e. it didn't require the same reliability etc, which we could use to our advantage and then pick a technology with some other nice properties that turned out to be cool.
I have a question about an event driven design architecture.
I have a server application that contains many domain services. each service is a WCF service, and I want the services to communicate using raising events via a dedicated pub-sub infrastructure that I've created.
Now everything works well, but I have a question regarding the orchestration..
I have a manager service which accepts calls from the consumers and starts the process, sending a message to component A which in turn sends a message to component B and so on.
Eventually some of these services raise events for the manager service which keeps no state of incoming requests.
What is the best way to orchestrate or manage this flow?
Thanks
Take a look at NServiceBus and NServiceBus sagas in particular. A saga (the meaning in NSB is more akin to 'process manager') maintains state and correlates messages by some ID. In your case, the manager service is a sort of saga. It should maintain sate for each process that it manages. This state should be keyed by a correlation ID to be shared by all messages sent between components.
I am using Windows Service Bus 1.0 to communicate between different processes, each context event stream exists on the bus as a topic.
Using the service bus to link events between bounded contexts I need a method to sync events (or in other words request a replay of past events) when a bounded context comes back online but want to limit the potential flood of messages coming back to only go to the endpoint that requested it, at least if this is something that can be easily done by using existing Service Bus features.
So given an imaginary ContextC sends a message to request all previous events from ContextA and ContextB, is there any way for these replay messages to be sent only to ContextC?
What would be the best way to map a context to be the owner of the topic (or in other words, an individual bus subscriber to a bus topic), to facilitate the unicast replaying above?
In my world, I keep this stuff loosely coupled - each context puts stuff onto a topic and anyone that needs stuff subscribes.
Each SB subscription can use the filtering facilities of Service Bus based on properties (e.g. you could tag events by adding Properties on the Messages and then have a filtering condition on the subscription meaning that only whitelisted classes of events ever apply to each consumer).
That plus the fact that you're already seggregating by topic.
The subscription and the topic then allow you to process the events without losing any or having the publisher go around worrying about or chasing subscribers.
You also mentioned you are tying this to an Event Store in other questions - in that case there is a chance your messages need to be consumed in order. If that is the case, you need to put a session id on your messages.
I could speculate as to why you want this subscriber driven redelivery but won't for now. You need to first explain / verify that concept and requirement (by asking questions which explain your higher level goals) a lot further before anyone answers how that would best be achieved using Service Bus.
My team are looking for ways to separate the various components of our system to decoupled services. What we'd like to achieve is an event-driven pattern where services subscribe to receive information sent when events occur in other systems.
Since we're already using WCF for our request-reply messaging, we'd like to continue using it for this particular pattern. Ideally, the messages would be managed via MSMQ to enable us to use reliable messaging to give us fault tolerance in the event of a service failure.
We're not looking for anything complicated like transactional support across service boundaries. Really, we just need a simple subscription-based message dispatch system. Are there any simple frameworks or services which can help us work to this pattern?
Probably the easiest is NServiceBus (http://www.nservicebus.com/PubSub.aspx) but this does not use WCF.
However from a integration perspective sending and receiving messages is far simpler than the messaging semantics on web services, so you don't need WCF to abstract that away.
Edit: In order to enable this using NetMsmqBinding you will have to implement the subscription infrastructure yourself. It would also be fairly easy to do.
Your publisher would need to have a database to store the subscriptions. When your subscribers start up, the first thing they do is send a subscription message to the publisher, who logs the subscription in it's subscription db.
The subscription messages should contain:
The message types I am interested in
My queue address
Then when your publisher wants to publish a message it retrieves the subscriptions and evaluates each one to see if the message matches the subscription and to retrieve the address to send to. Then it just sends the messages.
This is a standard pattern for implementing pub sub.