as I was reading thru documentation on nservicebus, I wasnt able to find what is persisted under Persistence section.
If nservicebus is a loosely coupled distributed library sending self-contained messages, what is there to persist? I dont understand.
With web app, when a user has a Session, we may choose to persist the Session in SQL Server, in Memory or somehow else, but with nservicebus there is no session to persist.
So, what is actually the Persistence in nservicebus?
What sort of data that could be persisted and for what reason?
Transports like RabbitMQ and Azure Service Bus natively support publish/subscribe. If an endpoint wants to receive published events, a 'subscription' to those events is stored inside those queuing technologies. Other queuing technologies don't support publish/subscribe natively, like MSMQ and Azure Storage Queues. NServiceBus mimics the behavior, but needs to store those subscriptions somewhere else.
Other things we can store are timeout and deferred messages and saga state. A saga is kind of a state machine (a workflow) and this state needs to be stored somewhere. Another feature NServiceBus supports is the outbox, which removes the need for distributed transactions by putting the message transaction and business transaction in the same database.
If you only use certain features, some transports allow you to do that natively. This removes the need for a persister. Sagas and Outbox always need persistence.
Related
I'm currently developing a REST api. The api performs basic crud operations. Data is synced to a legacy system using RabbitMQ. The api is running on SQL Server as a DB.
I'm wondering how to make sure data is saved in the DB and a message is put on the bus.
The fact you are missing distributed transactions looks like a very general issue to me so I'm wondering if there are any best practices using NServiceBus to solve this issue?
RabbitMQ doesn't support distributed transactions on its own, so there isn't much NServiceBus can do in this scenario. One option though is:
The endpoint is configured to use the Outbox feature
when the HTTP request is received by the REST endpoint a message is sent locally to self. No DB operations are performed at this stage
when the sent-to-self message is received you're now in the context of an incoming message and you can:
execute CRUD operations
send outgoing messages
The outbox will guarantee consistency even if there are no distributed transactions
Background
We are trying to introduce a new architectural pattern in our company and are considering CQRS with Event Sourcing using a Service Bus. Technologies we are currently developing our POC with are NServiceBus, Event Store, and MSMQ. We would like to have a single endpoint in NServiceBus defined with two different transports, MSMQ for our commands and Event Store for our events. The current state of our enterprise does not permit us to easily switch everything to Event Store presently as we have significant investment in our legacy apps using MSMQ, which is a reason why we are considering the hybrid approach.
Question
Is it possible to create a single NServiceBus endpoint that uses different transports? If yes, how? If no, what alternatives are there?
Aaron,
I think the best option would be to use MSMQ as a transport in NServiceBus. Here's how it may look like:
send a command via MSMQ
in a command handler (re)create an aggregate which is the target of the command
invoke the operations
store the resulting events in the EventStore along with the command's message id to ensure idempotence. The aggregate itself will be responsible for knowing the commands it already processed
in a separate component (event processor) use EventStore persistent subscription APIs to hook to all events stream. Some of these processed events should cause sending a command. Such a command might be send via NServiceBus send-only endpoint hosted inside this event processor.
in that event processor you can also re-publish all events via NServiceBus and MSMQ. Such events should not be subscribed by other services (see note on autonomy below)
NServiceBus Sagas (process managers) should live inside your service boundary and react on commands and/or events sent or re-published by these event processors over MSMQ.
One remark regarding the service boundaries is that you have to decide what level of service autonomy suits you:
* Weak, where services can directly subscribe to other service event streams. In this design events that cross service boundary are obviously allowed to carry data.
* Strong, where services use higher-level events for communication and these events only carry the identity of things and no data. If you want something like this, you can use your event processors to map from ES events to these "higher level" events.
all
Is it possible to store akka.net actors inbox messages in database?
What will happen if host with akka.net system crash?
Persisting messages is only part of the bigger issue, which is reliable message processing. In short the goal is not only to persist messages, but usually to guarantee that message has been received and correctly processed. By default Akka.NET uses at-most-once delivery semantic, which means, that messages are processed using best effort politics. This allows to keep high throughput and keep actors behavior away from being idempotent. However sometimes we need a higher reliability for some of the messages.
One of the techniques is to use another reliable queue (such as RabbitMQ or Azure Service Bus) in front of your actor system and use it for reliable messaging.
Other solution is to use AtLeastOnceDeliverySemantic actors from Akka.Persistence library. Here you may specify actor responsible for re-sending and confirming processed messages. From there you may decide to persist incoming messages using eventsourcing primitives build into Akka.Persistence itself. Persistence backend is plugable in this scenario.
I have been working on a CQRS project (my first) for over the last 9 months which has been a heavy learning curve. I am currently using JOliver's excellent EventStore in my write model and using PostGresSql for my read model.
Both my read and write databases are on the same machine which means that when a change is made to the write database, in the same synchronous call a change is made to the read model.
As I was learning CQRS I felt this was the best way to go as I had no experience with message queue/service bus frameworks such as MassTransit, NServiceBus etc.
I am now at a point with most of my architecture in place to introduce a message queue framework.
Today, I came across Redis MQ which is part of ServiceStack and as we are already using ServiceStack for our Rest based HTTP clients, this seems like the right way to go.
My question is more about understanding what I need to know (or if I have any misunderstandings) to implement Redis MQ and whether Redis MQ is the right choice?
Now from what I understand, I would use Redis MQ as a durable queue between the write and read database. Once my event store has recorded that something has happened in my domain then it will publish to Redis MQ. The services listening for events/messages would receive the event/message from Redis MQ and once it has processed it (i.e. update or write to the read model), a notification/response goes back to the event store to tell the event store that the message has been received and processed by the listener/subscriber.
Does this sound correct?
Also would the Redis MQ architecture give me everything that NSB, RavenDB, MassTransit etc offer?
Also, I will be deploying to windows 2008 and 2003 server. Is Redis stable for these OSs?
I think the ServiceStack implementation of message queueing in Redis is more appropriate for job-queue scenarios - it pushes a message onto the end of a Redis list and then uses Redis pub-sub to notify listening subscribers that there is a message to pull from the queue. Any consumers would be competing for messages.
For event sourcing, you may be more interested in a type of fanout or topic based messaging topology as offered by RabbitMQ, not that that precludes you from building that sort of thing using Redis data structures yourself.
Now from what I understand, I would use Redis MQ as a durable queue
between the write and read database.
Yes this is correct.
Once my event store has recorded that something has happened in my
domain then it will publish to Redis MQ.
Yes and this can be done in several ways. It can either happen as part of the transaction which persists to the event store or you can have an out of band process which continuously publishes events from the event store.
a notification/response goes back to the event store to tell the event
store that the message has been received and processed by the
listener/subscriber.
The response back to the event publisher is usually omitted. This truly decouples the publishers from subscribers. You make the assumption that once the message is published, all interested subscribers will handle it. If something happens, an error should be logged.
Also would the Redis MQ architecture give me everything that NSB,
RavenDB, MassTransit etc offer?
I don't have experience running Redis MQ, but I do know that Redis supports pub/sub which is one of the value propositions of NSB and MassTransit (as opposed to say bare-bones MSMQ). What MT and NSB offer beyond pub/sub are sagas and it doesn't seem like Redis MQ support those out of the box at least. You may not ever have a need for sagas so this should not automatically be a deterrent. RavenDB is not a message queue so it doesn't apply here.
Also, I will be deploying to windows 2008 and 2003 server. Is Redis
stable for these OSs?
I've run Redis on 2008 R2 and it has been stable so I would think Redis MQ would be stable as well.
You may be interested in a little side project of mine on GitHub which is a queue and persistence implementation for NServiceBus using Redis. https://github.com/mackie1001/NServicebus.Redis
I'd not call it production ready and I want to port it to NSB 4 and do some thorough testing but the meat of it is done.
My team are looking for ways to separate the various components of our system to decoupled services. What we'd like to achieve is an event-driven pattern where services subscribe to receive information sent when events occur in other systems.
Since we're already using WCF for our request-reply messaging, we'd like to continue using it for this particular pattern. Ideally, the messages would be managed via MSMQ to enable us to use reliable messaging to give us fault tolerance in the event of a service failure.
We're not looking for anything complicated like transactional support across service boundaries. Really, we just need a simple subscription-based message dispatch system. Are there any simple frameworks or services which can help us work to this pattern?
Probably the easiest is NServiceBus (http://www.nservicebus.com/PubSub.aspx) but this does not use WCF.
However from a integration perspective sending and receiving messages is far simpler than the messaging semantics on web services, so you don't need WCF to abstract that away.
Edit: In order to enable this using NetMsmqBinding you will have to implement the subscription infrastructure yourself. It would also be fairly easy to do.
Your publisher would need to have a database to store the subscriptions. When your subscribers start up, the first thing they do is send a subscription message to the publisher, who logs the subscription in it's subscription db.
The subscription messages should contain:
The message types I am interested in
My queue address
Then when your publisher wants to publish a message it retrieves the subscriptions and evaluates each one to see if the message matches the subscription and to retrieve the address to send to. Then it just sends the messages.
This is a standard pattern for implementing pub sub.