I have a service that should reply either to consumer or to supplier to the same request - to show stock. How do I model such a service that replies to the corresponding party? Current solution - duplicating same paths for client and supplier - looks weird and does not describe what is really happening at the service side (the service is one processing system that knows who to reply)
There are different ways to do this.
First approach could be to use an exclusive gateway and check the type of the message.
Second approach, which I personally prefer is to collapse the stock pool. An only model the client and supplier. See the following model.
You probably don't have to model (in this diagram) supplier and client as two separate entities. Just replace them with a single entity third party as this communication seems similar for both.
Related
After recently reading about event-based architecture, I wanted to change my architecture into one making use of such strengths.
I have two services that expose an API (crud, graphql), each based around a different entity and using a different database.
However, now whenever someone deletes a certain type of row in service A, i need to delete a coupled row in Service B.
So I added Kafka to my design, and whenever I delete the entity in service A, it publishes a notification message into Kafka.
In service B I am currently consuming the same topic so whenever a new message is received the Service will also handle the deletion of the matching entity, because it already has access that table because the same service already exposes the CRUD API to users.
What i'm not sure about is whether putting the Kafka Consumer and the API together in the same service is a good design. It contradicts the point of single responsibility in micro services, and whether there is an issue in one part of the service, it will likely affect the second.
However, creating a new service will also cause me issues - i will have 2 different services accessing the same table, and i will have to make sure i always maintain them together, whenever making changes to the table or database.
What is the best practice in a incident such as this? Is it inevitable to have different services have data coupling or is it not so bad to use the same service for two, similiar usages.
There is nothing wrong with using Kafka... You could do the same with point-to-point service communication, however (JSON-RPC / gRPC), however.
The real problem you seem to be asking about is dual-writes or race-conditions leading to data inconsistency.
While you could use a single consumer group and one topic-partition to preserve order and locking across consumers interested in those events, that does not lock out other consumer-groups from interacting with the database to perform the same action. Therefore, Kafka itself won't help with this problem.
You'll need external, distributed locks (e.g. Zookeeper can be used here) that fence off your database clients while you are performing actions against it.
To the original question, Kafka Connect offers an API and is also a Producer and Consumer client (and would be recommended for database interactions). So is Confluent Schema Registry, KSQLdb, etc.
I believe that the consumer of your service B would not be considered "a service" or part of the "service", as in that it is not called as part the code which services requests. Yet it does provide functionality that is required for the domain function of your microservice. So yes I would consider the consumer part of the Microservice in terms of team/domain responsibility.
There may be different opinions on if the consumer code should share the same code base/repo as the "service" code. Some people believe that it is better to limit the repo scope to a single "executable", others believe it is beneficial to keep the domain scope and have everything in a single repo. I probably belong to the latter group but do not have a very strong opinion on it. I would argue it is more important to have a central documentation / wiki for the domain that will point to the repos involved etc.
I've searched for a good explanation for the difference between these two,
but didn't really find one.
What I know till now is that:
correlation id is a string (Guid which was converted to string), and delivery tag is an int.
correlation id is unique for each message, and delivery tag is unique only in
the channel (The channel is the scope).
That's fine....but what is the difference in the purposes? why do we need two identifiers for a message?
The two identifiers exist at two different conceptual layers of communication, and have different properties that are useful in each case. While a protocol could probably be designed that had one identifier serving both purposes, keeping them separate makes both implementations simpler.
Delivery tags
Part of the AMQP communication layer, built into RabbitMQ itself.
Example use: a consumer process can acknowledge that a message has been processed and can be permanently discarded on the broker (RabbitMQ server).
Automatically assigned within the open channel for every message delivered.
Must be unique within that channel in order for the protocol to function correctly. Does not need to be unique across different channels, so a simple incrementing integer is simple to implement.
The same message may be delivered at different times with different delivery tags, or even exist on multiple queues and be delivered simultaneously to different consumers.
Correlation IDs
Part of the logic of the application that is using RabbitMQ, not the broker itself.
Example use: using a matching correlation ID and "reply to" on two separate messages, which the application wants to treat as a request and a response in an RPC pattern.
Needs to be manually added when the message is first created, and is optional.
Not guaranteed to be unique by the protocol, which just sees it as an arbitrary string. It is up to an application to generate in a way that is sufficiently unlikely to collide for its use case, such as an appropriate form of UUID.
Will stay the same every time a message is delivered, no matter how many times it is forwarded or duplicated into multiple queues.
Correlation ID is generally used in the context of RabbitMQ when I want to see a synchronous behavior in which a message is sent and in response to it another sender will send a response but will have the correlationID in the reply-to tag . The common pattern which is replicated in RabbitMQ is the RPC call which is more like a Synchronous messaging.
Delivery Tag is however an indicator of the delivery of the message per channel and generally comes in scope when Acknowledged Delivery model is being followed.
Both have completely different purpose and are not message identifier as such.
We have a situation where several of our services are shared across our system. For example one that tracks stock movements. Whenever the stock level of an article changes an event is raised.
The problem we run in to is that while sometimes another service may be interested in ALL stock change events (for example to do some aggregation), in most cases only stock changes that are the result of a specific action are interesting.
The problem we now face is this. Say have an IArticleStockChangedEvent event that contains the article number, the stock change and a ProcessId that requested the change. This event is raised for every change in the article stock.
Now some external service has a saga to change 10 articles and commands the stock service to make it so. It also implements IHandleMessages to keep track of the progress. This works well in theory, but in practise this means that the service containing this saga will be flooded with unrelated IArticleStockChangedEvent message for which it will be unable to find a corresponding saga instance. While not technically breaking anything it causes unnecessary delays in the system.
I'm not really looking forward to creating a new kind of IArticleStockChangedEvent for every saga that can possibly cause a stock change. What is the recommended approach to handle this issue?
Thanks
The knowledge about which IArticleStockChangedEvent events you need to be delivered to your service lives inside your "external" service and changes dynamically, so it's not possible (or is complex and non-scalable) to make a filter in either Stock service or at a transport level (Ex. Service Bus subscription filter).
To make an optimization, namely avoid deserialization of the IArticleStockChangedEvent, you might consider custom Behavior<IIncomingPhysicalMessageContext> where you read the Stock item's Id from message header and lookup db to see if there is any saga for that stock item and if not, short circuit the message processing.
Better solution might be to use Reply and reply with a message from Stock service.
We are into the lead business. We capture leads and pass it on to the clients based on some rules. integration to each client very in nature like nature of the API and in some cases, data mapping is also required. We perform the following steps in order to route leads to the client.
Select the client
Check if any client-specific mapping(master data) is required.
Send Lead to nearest available dealer(optional step)
Call client api to send lead
Update push status of the lead to database
Note that some of the steps can be optional.
Which design pattern would be suitable to solve this problem. The motive is to simplify integration to each client.
You'll want to isolate (and preferably externalize) the aspects that differ between clients, like the data mapping and API, and generalize as much as possible. One possible force to consider is how easily new clients and their APIs can be accommodated in the future.
I assume you have a lot of clients, and a database or other persistent mechanism that holds this client list, so data-driven routing logic that maps leads to clients shouldn't be a problem. The application itself should be as "dumb" as possible.
Data mapping is often easily described with meta-data, and also easily data-driven. Mapping meta-data is client specific, so it could easily be kept in your database associated with each client in XML or some other format. If the transformations to leads necessary to conform to specific APIs are very complex, the logic could be isolated through the use of a strategy pattern, with the specific strategy selected according to the target client. If an extremely large number of clients and APIs need to be accommodated, I'd bend over backwards to make the API data-driven as well. If you have just a few client types (say less than 20), I'd employ some distributed asynchronicity, and just have my application publish the lead and client info to a topic corresponding to client-type, and have subscribed external processors specific for each client-type do their thing and publish the results on another single queue. A consumer listing to the results queue would update the database.
I will divide your problem statement into three parts mentioned below:
1) Integration of API with different clients.
2) Perfom some steps in order to route leads to the client.
3) Update push status of the lead to database.
Design patterns involved in above three parts:
1) Integration of API with different clients - Integration to each client vary in nature like the nature of the API. It seems you have incompitable type of interface so, you should design this section by using "Adapter Design Pattern".
2) Perform some steps in order to route leads to the client- You have different steps of execution. Next step is based on the previous steps. So, you should design this section by using "State Design Pattern".
3) Update push status of the lead to database: This statement shows that you want to notify your database whenever push status of the lead happens so that information will be updated into database. So, you should design this section by using "Observer Design Pattern".
Sounds like this falls in the workflow realm.
If you're on Amazon Web Services, there's SWF, otherwise, there's a lot of workflow solutions out there for your favorite programming language.
Can any developers/architects with experience with NServiceBus offer guidance and help on the following?
We have a requirement in the business (and not a lot of money) to create a robust interface between an externally hosted application and our internal ERP's (yup, more than one).
When certain activities take place in the third party application they will send us the message. i.e. call a web service passing various fields of information in the message etc. We are not in control nor can we change this third party application.
My responsibility is creating this web service and the processing of the messages into each ERP. The third party dictates how the web service will look, but not what its responsible for. We have to accept that if they get a response back of 'success' then we at this point have taken responsibility for that message! i.e. we need to ensure as close to perfect no data loss takes place.
This is where I'm interested in the use of NServiceBus. Use it to store/accept a message at first. At this point I get lost, I can't tell what should happen, i.e. what design follows. Does another machine (process) subscribe and grab the message to process it into an ERP, if so since each ERPs integration logic differs do I make a subscriber per ERP? A message may have two destination ERP targets however, so is it best the message is sent and not subscribed to.
Obviously in the whole design, I need to have some business rules which help determine the destination ERP's and then business rules that determine what actually takes place with in each ERP. So I also have a question on BRE's but this can wait although still may be a driver for what the message has to do.
so:
Third party > web service call > store message (& return success) > determine which ERP is target > process each into ERP > mark message complete
If anything fails along the lines making sure the message does not get lost. p.s. how does MSMQ prevent loss since the whole machine may die ? is this just disk resilience etc?
Many thanks if you've read and even more for any advice.
This sounds like a perfect application for NServiceBus.
Your web service should ONLY parse the request from the third and translate it into an NServiceBus message, which it should Bus.Send(). You don't respond with a 200 status code until that message is on the Bus, at which point, you are responsible for it, and NServiceBus's built-in error/retry and error queue facilities become your best friend.
This message should be received by another endpoint, but it needs to be able to account for duplicate messages or use idempotence so that duplicates aren't a problem. If the third party hits your web service, and the message is successfully placed on the bus, but then some error prevents them from receiving the 200 response code, you will get duplicates from them.
At this point, the endpoint receiving the MessageFromWebServiceCommand message could Bus.Publish() a SomeBusinessEventHappenedEvent that contains the command data.
For each ERP, create an additional endpoint that subscribes to the SomeBusinessEventHappenedEvent and uses your business logic to decide what to do respective to that ERP. In some cases, that "something" may be "nothing". Keep idempotence in mind here too, because if the message fails it will be retried.
All the other things you're worried about (preventing loss of messages, what happened if machines die) will be taken care of thanks to NServiceBus and MSMQ being naturally resilient to such problems.
Here is a blog post, including a sample project, that shows how to receive messages from an external partner via a web service and handle them with NServiceBus, and a link straight to the sample project on GitHub:
Robust 3rd Party Integrations with NServiceBus
Project Source Code on GitHub