Can any developers/architects with experience with NServiceBus offer guidance and help on the following?
We have a requirement in the business (and not a lot of money) to create a robust interface between an externally hosted application and our internal ERP's (yup, more than one).
When certain activities take place in the third party application they will send us the message. i.e. call a web service passing various fields of information in the message etc. We are not in control nor can we change this third party application.
My responsibility is creating this web service and the processing of the messages into each ERP. The third party dictates how the web service will look, but not what its responsible for. We have to accept that if they get a response back of 'success' then we at this point have taken responsibility for that message! i.e. we need to ensure as close to perfect no data loss takes place.
This is where I'm interested in the use of NServiceBus. Use it to store/accept a message at first. At this point I get lost, I can't tell what should happen, i.e. what design follows. Does another machine (process) subscribe and grab the message to process it into an ERP, if so since each ERPs integration logic differs do I make a subscriber per ERP? A message may have two destination ERP targets however, so is it best the message is sent and not subscribed to.
Obviously in the whole design, I need to have some business rules which help determine the destination ERP's and then business rules that determine what actually takes place with in each ERP. So I also have a question on BRE's but this can wait although still may be a driver for what the message has to do.
so:
Third party > web service call > store message (& return success) > determine which ERP is target > process each into ERP > mark message complete
If anything fails along the lines making sure the message does not get lost. p.s. how does MSMQ prevent loss since the whole machine may die ? is this just disk resilience etc?
Many thanks if you've read and even more for any advice.
This sounds like a perfect application for NServiceBus.
Your web service should ONLY parse the request from the third and translate it into an NServiceBus message, which it should Bus.Send(). You don't respond with a 200 status code until that message is on the Bus, at which point, you are responsible for it, and NServiceBus's built-in error/retry and error queue facilities become your best friend.
This message should be received by another endpoint, but it needs to be able to account for duplicate messages or use idempotence so that duplicates aren't a problem. If the third party hits your web service, and the message is successfully placed on the bus, but then some error prevents them from receiving the 200 response code, you will get duplicates from them.
At this point, the endpoint receiving the MessageFromWebServiceCommand message could Bus.Publish() a SomeBusinessEventHappenedEvent that contains the command data.
For each ERP, create an additional endpoint that subscribes to the SomeBusinessEventHappenedEvent and uses your business logic to decide what to do respective to that ERP. In some cases, that "something" may be "nothing". Keep idempotence in mind here too, because if the message fails it will be retried.
All the other things you're worried about (preventing loss of messages, what happened if machines die) will be taken care of thanks to NServiceBus and MSMQ being naturally resilient to such problems.
Here is a blog post, including a sample project, that shows how to receive messages from an external partner via a web service and handle them with NServiceBus, and a link straight to the sample project on GitHub:
Robust 3rd Party Integrations with NServiceBus
Project Source Code on GitHub
Related
So I'm thinking of using RabbitMQ to send messages between all the varied apps in our organization. In the attached image is essentially the picture in my mind of how things would work.
So the message goes into the exchange, and splits out into three queues.
Payloads are always JSON text.
The consumers are long-running windows services whose only job is to sit and listen for messages destined for their particular application.When a message comes in, they look at the header to determine how this payload JSON should be interpreted, and which REST endpoint it should be sent to. e.g., "When I see a 'WORK_ORDER_COMPLETE' header I am going to parse this as a WorkOrderCompleteDto and send it as a POST to the CompletedWorkOrder WebAPI method at timelabor-api.mycompany.com. If the API returns other than 200, I reject the message and let rabbit handle it. If I get a 200 back from the API, then I ack the message to rabbit."
Then end applications are simply our internal line-of-business apps that we use for inventory, billing, etc. Those applications are then responsible for performing their respective function (decrementing inventory, creating a billing record, yadda yadda.
Does this in any way make a sensible understanding of a proper way to use Rabbit?
Conceptually, I believe you may be relying on RabbitMQ to do things that your application needs to do.
The assumption of the architecture seems to be that each message is processed by each of your consuming applications totally in a vacuum. What this means is that you don't care that a message processed successfully by Billing_App ultimately failed with Inventory_App. Maybe this is true, but in my experience, it isn't.
If the end goal is to achieve some consistent state in the overall data, you're going to need a some supervisory component orchestrating and monitoring the various operations to ensure that the state is consistent. This means, in effect, that your statement about rejecting a message back to RabbitMQ means you have a bit more thought to put into what happens when something fails.
I would focus on identifying some UML activity diagrams that describe your behavior and how it achieves the end-state, and use that as a guide to determine how the orchestration of your application needs to be designed.
I have a scenario where I need to expose a bunch of event messages that have been created in NServiceBus to third parties over a simple authenticated REST API. The third party may or may not be using .NET (and they might even be JavaScript in the browser).
I understand that that pub/sub is a push mechanism, but I'm looking for a polling mechanism. Is this even possible in NServiceBus? Is this what an adapter is for, or is that for accepting inbound messages?
Typically you would not want to expose your service bus to third parties. You could manage to have some transport deliver to subscribers but then you would be sending an internal structure to the outside world. You also did mention that you need a pull mechanism via a REST interface.
What I would suggest is to have a subscriber within your service bus that listens to the relevant messages and then either saves them in a serialized form in a type of 'event store' or de-normalizes them into the resources that the REST interface would expose. These messages/resources would contain the relevant date/time stamp.
It would be up to the consumer of the REST API to specify some point in time to retrieve the resources from. So the third party would simply keep track of when last they retrieved the data. Of course they could retrieve as much as they need and new 'subscribers' would be able to retrieve the entire history if required. Each message/resource should also have a GUID of sorts to be able to aid idempotence.
We've created a web application that is an a e-book reader. So one thing to keep in mind is that the domain is not exactly that of reading a physical book. We are now trying to gather users' reading behavior by storing information about e-book pages accessed by our users. Since this information goes to a data warehouse we thought raising an event from the bookcontroller is the right way to do it.
bus.Publish()
But we are not sure if it should be a publish or a send since there is really only one consumer to this event and that is our business intelligence team. We've also read that it is not advisable to publish from the web app (http://www.make-awesome.com/2010/10/why-not-publish-nservicebus-messages-from-a-web-application/). So now the alternative is to use bus.Send(RecordPageAccessedCommand)
But the above command does not change our application state in anyway. So is it truly a command? I have a feeling that the mistake we are making is using NServiebus's features (Publish,Send) and trying to equate it with what a command or event is.
Please let me know what the solution to this is.
Based on the information you provided, I would recommend "sending" to your endpoint.
Sending a command implies that the endpoint handling the message should do something. In your case, recording that the page was accessed is the thing the endpoint should do.
Publishing an event implies that you are notifying 0..n subscribers that something occurred. You could publish an event from your command handler if some other service in your system was interested in the fact that a page was accessed. The key point here is that it's not a "fact" until you've recorded it.
I've found that consumers tend to grow once data is available. Having the ability to publish an event from your command handler will make it trivial to notify new consumers without changing/risking your existing code base.
The RecordPageAccessedCommand is a command as it is commanding the system to do something, in this case, record that a page has been accessed.
If I've understood your scenario correctly. A message should be sent from your controller to the "Business intelligence Team Service" telling the system to record that a page has been accessed. This service would store this information and would be the owner/technical authority of this information.
No other services should store or require this information in its pure form, they can however subscribe to events from this service, in highly contrived scenario for example, when a user reads 1000 pages the "Business intelligence Team Service" can publish an event that a 1000 pages have been read ie Bus.Publish(), which may be handled by a billing service that gives a discount for the user on their next purchase.
The data warehouse can have access to this information stored in your "Business intelligence Team Service" as it would fall under IT/OPS.
I've been reading through the documentation on the NServiceBus site but am struggling to piece it all together.
The goal is to provide a durable messaging solution between on-premise back office systems and a public facing web site in another data center.
I will need bidirectional (on-premise <> web site) pub-sub and request-response communication.
The documentation makes it clear that there isn't one central point that all communication goes through, but surely the subscriptions need to persisted somewhere (in a central location?).
The NServiceBus gateway does look like it would meet my requirements but I can't find any working examples of this.
Can someone provide a bit more detail on how the Gateway works and whether it will meet my requirements?
The subscriptions are persisted on each publisher endpoint. Say you have a service endpoint publishing web orders. All other services who are interested can subscribe by sending a subscription message to the publisher, who then stores the subscriptions locally. When a message is available the publisher evaluates the subscriptions and send a message to each of the subscribers.
This brings us onto your other requirement - that of request/response. Because NSB is based on msmq, everything is asynchronous. The most a publisher could do is send a response to a caller just saying that the request has been received and will be published. The nature of async messaging means that you cannot have a synchronous response from any downstream subscribers.
But this cost comes with benefits - namely reliability and availability.
Reliability - because you are using durable messaging the messages will eventually be delivered, at which point a response can be generated which will also eventually find it's way back to the caller. This is highly reliable when compared to request response.
Availability: because the publisher service is always able to send a message (whether a downstream subscriber is available or not), it never needs to block incoming requests. If you load balance your publisher somehow you can easily achieve availability at enterprise levels.
However you need to balance this against your latency requirements. Asynchrony usually equals latency. So if you have latency requirements in the sub-100 ms range NSB may not be your best bet.
Apologies for not answering your query about NSB Gateway - I have never used it.
Hope this helps.
The Gateway solves the communication problem between sites. It will ensure that messages get delivered from SiteA to SiteB. The messages are hashed and validated on the other end. Apparently there isn't an example of this in 2.5, so I'm thinking of throwing one together as this has come up a few times in the past month.
Is it possible to create end points dynamically at runtime. E.g. Send a message to a known endpoint with details of a new endpoint so that a network node can learn of new nodes on the fly.
NServiceBus does not support this out of the box, but if you really really want it (and you are sure that it is the right way to go), you are free to implement your own message routing and send messages explicitly to an endpoint with bus.Send(endpoint, message).
In a project I am currently involved with, we do this with great success, because it allows us to seamlessly sign services in and out of the system while it is running, resulting in zero downtime during upgrades.
It took a bit of work to get it working though, so I would only recommend this if you are certain that your requirements demand it.