I have a scenario where I need to expose a bunch of event messages that have been created in NServiceBus to third parties over a simple authenticated REST API. The third party may or may not be using .NET (and they might even be JavaScript in the browser).
I understand that that pub/sub is a push mechanism, but I'm looking for a polling mechanism. Is this even possible in NServiceBus? Is this what an adapter is for, or is that for accepting inbound messages?
Typically you would not want to expose your service bus to third parties. You could manage to have some transport deliver to subscribers but then you would be sending an internal structure to the outside world. You also did mention that you need a pull mechanism via a REST interface.
What I would suggest is to have a subscriber within your service bus that listens to the relevant messages and then either saves them in a serialized form in a type of 'event store' or de-normalizes them into the resources that the REST interface would expose. These messages/resources would contain the relevant date/time stamp.
It would be up to the consumer of the REST API to specify some point in time to retrieve the resources from. So the third party would simply keep track of when last they retrieved the data. Of course they could retrieve as much as they need and new 'subscribers' would be able to retrieve the entire history if required. Each message/resource should also have a GUID of sorts to be able to aid idempotence.
Related
After recently reading about event-based architecture, I wanted to change my architecture into one making use of such strengths.
I have two services that expose an API (crud, graphql), each based around a different entity and using a different database.
However, now whenever someone deletes a certain type of row in service A, i need to delete a coupled row in Service B.
So I added Kafka to my design, and whenever I delete the entity in service A, it publishes a notification message into Kafka.
In service B I am currently consuming the same topic so whenever a new message is received the Service will also handle the deletion of the matching entity, because it already has access that table because the same service already exposes the CRUD API to users.
What i'm not sure about is whether putting the Kafka Consumer and the API together in the same service is a good design. It contradicts the point of single responsibility in micro services, and whether there is an issue in one part of the service, it will likely affect the second.
However, creating a new service will also cause me issues - i will have 2 different services accessing the same table, and i will have to make sure i always maintain them together, whenever making changes to the table or database.
What is the best practice in a incident such as this? Is it inevitable to have different services have data coupling or is it not so bad to use the same service for two, similiar usages.
There is nothing wrong with using Kafka... You could do the same with point-to-point service communication, however (JSON-RPC / gRPC), however.
The real problem you seem to be asking about is dual-writes or race-conditions leading to data inconsistency.
While you could use a single consumer group and one topic-partition to preserve order and locking across consumers interested in those events, that does not lock out other consumer-groups from interacting with the database to perform the same action. Therefore, Kafka itself won't help with this problem.
You'll need external, distributed locks (e.g. Zookeeper can be used here) that fence off your database clients while you are performing actions against it.
To the original question, Kafka Connect offers an API and is also a Producer and Consumer client (and would be recommended for database interactions). So is Confluent Schema Registry, KSQLdb, etc.
I believe that the consumer of your service B would not be considered "a service" or part of the "service", as in that it is not called as part the code which services requests. Yet it does provide functionality that is required for the domain function of your microservice. So yes I would consider the consumer part of the Microservice in terms of team/domain responsibility.
There may be different opinions on if the consumer code should share the same code base/repo as the "service" code. Some people believe that it is better to limit the repo scope to a single "executable", others believe it is beneficial to keep the domain scope and have everything in a single repo. I probably belong to the latter group but do not have a very strong opinion on it. I would argue it is more important to have a central documentation / wiki for the domain that will point to the repos involved etc.
I work on a system where we have the same website across multiple countries. Each of these websites has it's own services. Everything works well, but I've always found myself having to send messages rather than publishing as the messages otherwise other services where I know before hand it's completely irrelevant. It sounds pointless to me publishing to many services and then filtering it's relevance.
Is there a practice I should be dealing with when wanting to publish messages to a certain subset of services, how have others dealt with this problem?
By default endpoints subscribe to all messages. If you want only certain endpoints to subscribe to specific sets, then you need to configure your endpoint to DoNotAutoSubscribe(). You then must explicitly subscribe to each message type the endpoint will be interested in using Bus.Subscribe().
Could you describe your logic of determining relevance for particular endpoint systems ? the purpose of publishing and subscribing is that there are events in a system that other endpoints can subscribe to.
you should not know something about your subscribers. so how do you determine relevance ?
if these messages are not relevant for a specific endpoint why do you want to subscribe to these messages ?
If it truly is an event message then you need to publish the message. If you need to publish to a subset you could have a separate subscription store that the endpoint in question would use.
Typically it should be up to the subscriber to determine whether the received event is relevant but if you do have the information up-front then could go with the separate subscription store.
In my FOSS ESB project (http://shuttle.codeplex.com/) a ISubscriptionManager implementation has to be provided to the ESB to determine the subscriber uris to send published messages to. Although it may be overkill one could provide a custom implementation that contains some logic to perform the filtering; otherwise the separate subscription store.
Can any developers/architects with experience with NServiceBus offer guidance and help on the following?
We have a requirement in the business (and not a lot of money) to create a robust interface between an externally hosted application and our internal ERP's (yup, more than one).
When certain activities take place in the third party application they will send us the message. i.e. call a web service passing various fields of information in the message etc. We are not in control nor can we change this third party application.
My responsibility is creating this web service and the processing of the messages into each ERP. The third party dictates how the web service will look, but not what its responsible for. We have to accept that if they get a response back of 'success' then we at this point have taken responsibility for that message! i.e. we need to ensure as close to perfect no data loss takes place.
This is where I'm interested in the use of NServiceBus. Use it to store/accept a message at first. At this point I get lost, I can't tell what should happen, i.e. what design follows. Does another machine (process) subscribe and grab the message to process it into an ERP, if so since each ERPs integration logic differs do I make a subscriber per ERP? A message may have two destination ERP targets however, so is it best the message is sent and not subscribed to.
Obviously in the whole design, I need to have some business rules which help determine the destination ERP's and then business rules that determine what actually takes place with in each ERP. So I also have a question on BRE's but this can wait although still may be a driver for what the message has to do.
so:
Third party > web service call > store message (& return success) > determine which ERP is target > process each into ERP > mark message complete
If anything fails along the lines making sure the message does not get lost. p.s. how does MSMQ prevent loss since the whole machine may die ? is this just disk resilience etc?
Many thanks if you've read and even more for any advice.
This sounds like a perfect application for NServiceBus.
Your web service should ONLY parse the request from the third and translate it into an NServiceBus message, which it should Bus.Send(). You don't respond with a 200 status code until that message is on the Bus, at which point, you are responsible for it, and NServiceBus's built-in error/retry and error queue facilities become your best friend.
This message should be received by another endpoint, but it needs to be able to account for duplicate messages or use idempotence so that duplicates aren't a problem. If the third party hits your web service, and the message is successfully placed on the bus, but then some error prevents them from receiving the 200 response code, you will get duplicates from them.
At this point, the endpoint receiving the MessageFromWebServiceCommand message could Bus.Publish() a SomeBusinessEventHappenedEvent that contains the command data.
For each ERP, create an additional endpoint that subscribes to the SomeBusinessEventHappenedEvent and uses your business logic to decide what to do respective to that ERP. In some cases, that "something" may be "nothing". Keep idempotence in mind here too, because if the message fails it will be retried.
All the other things you're worried about (preventing loss of messages, what happened if machines die) will be taken care of thanks to NServiceBus and MSMQ being naturally resilient to such problems.
Here is a blog post, including a sample project, that shows how to receive messages from an external partner via a web service and handle them with NServiceBus, and a link straight to the sample project on GitHub:
Robust 3rd Party Integrations with NServiceBus
Project Source Code on GitHub
My team are looking for ways to separate the various components of our system to decoupled services. What we'd like to achieve is an event-driven pattern where services subscribe to receive information sent when events occur in other systems.
Since we're already using WCF for our request-reply messaging, we'd like to continue using it for this particular pattern. Ideally, the messages would be managed via MSMQ to enable us to use reliable messaging to give us fault tolerance in the event of a service failure.
We're not looking for anything complicated like transactional support across service boundaries. Really, we just need a simple subscription-based message dispatch system. Are there any simple frameworks or services which can help us work to this pattern?
Probably the easiest is NServiceBus (http://www.nservicebus.com/PubSub.aspx) but this does not use WCF.
However from a integration perspective sending and receiving messages is far simpler than the messaging semantics on web services, so you don't need WCF to abstract that away.
Edit: In order to enable this using NetMsmqBinding you will have to implement the subscription infrastructure yourself. It would also be fairly easy to do.
Your publisher would need to have a database to store the subscriptions. When your subscribers start up, the first thing they do is send a subscription message to the publisher, who logs the subscription in it's subscription db.
The subscription messages should contain:
The message types I am interested in
My queue address
Then when your publisher wants to publish a message it retrieves the subscriptions and evaluates each one to see if the message matches the subscription and to retrieve the address to send to. Then it just sends the messages.
This is a standard pattern for implementing pub sub.
The scenario is a follow:
I have multiple clients in which they can register themselves on a workflow server, using WCF requests, to receive some kind of notifications. The information of the notifications will be received from an external system using another receive activity. The workflow then should get the notification information and callback all registered clients using send activity and callback correlations (the clients are exposing callback interfaces implemented in there and the end-point addresses passed initially with the registration requests). "Log-running workflow service" approach is used with a persistent storage.
Now, I'm looking for some way to correlate the incoming information of the notifications received from the external system with the persisted workflow instances created previously when the registration requests, so that all clients will be notified using end-points that already passed with the registration requests. Is WF 4.0 capable of resuming and executing multiple workflow instances when the information of the notification received without storing end-points somehow manually and go though them? If yes, how can I do that?
Also, if my approach of doing so is not correct, then please advice me about the best practice of doing such system using WCF services.
Your help is highly appreciated.
When you use request correlation with workflow services the correlation key must always match a single workflow instance, you can't have multiple workflow instances react to a single message. So you either need to multicast the message using all the different correlation keys or resume you workflow instances in some other way. That other way could be to store the request somewhere, like a SQL table, and have the workflows periodically check that location if they need to notify the client.