I am adding a new flow to my Mulesoft application whose source/starting point is a SalesForce connector that listens to SF Platform events.
I am trying to understand which layer should this message listener be in
The confusion is that,
I think as a connector to pull SF data, this listener should be in the system layer
On the other hand, should flows be started in the system layer; and then they should call process layer for processing? Shouldn't communication start from experience, then come to process and then system layer ?
TIA
https://www.youtube.com/watch?v=1WjfkyL6OhA : at 5 min 15 seconds
says listeners should be in experience layer
"System" layer describes APIs, not applications. The listener you're describing is a mule application event source, not an API.
Perhaps you have created an event driven API whose client is your salesforce platform... the consumer is the publisher of the platform events. That consumer needs to know what the events need to look like so they can publish them to your integration layer.
In this case, I'd describe that as an experience API, since the consumer is external to your integrations.
Related
I am reading through nestjs docs and there seems to be no way to use anything else then built in eventbus. Now let's say my cqrs event wants to communicate with microservices with rabbitmq. So 2 questions:
Is it possible to dispatch event that will send queue to external eventbus?
Is it okey from ddd point of view? Where should I dispatch it then? Dispatch from domain layer? And then parties (microservices) that are interested can listen and write to their own db parts of it or whatever.
And if you can please explain it on human level, I am new to ddd with cqrs. And sorry if it's frequently asked question but it's also related to nestjs so I need some clarifications.
Thanks 😊
Integration events should be used to notify external services. A good practice is to keep the external event bus (integration bus) independent from any micro service. And regarding the 2 questions:
It should be possible to send events to external bus with nestjs. Ex: write a domain event handler that create an integration event and dispatch it. You can create a new IntegrationBus interface and implement it in any technology that fits.
It is definitely OK from DDD point of view. Context mapping (how different contexts share data) is part of the strategic patterns of DDD. Usually the integration events are published in the application layer and in some case it can be more convinient to do it in the infrastructure layer.
Here are few articles with more details about domain and integration events:
https://learn.microsoft.com/en-us/dotnet/architecture/microservices/microservice-ddd-cqrs-patterns/domain-events-design-implementation
https://devblogs.microsoft.com/cesardelatorre/domain-events-vs-integration-events-in-domain-driven-design-and-microservices-architectures/
I have a task where I need to establish a connection between an ActiveMQ queue and a .Net application. I am using the AMQP.Net Lite plugin for this. But I have a need for the receiver of the .Net application to be called the moment the message goes into the queue.
Is there any solution where there is no need for the .Net application to stay from time to time by checking the MQ queue to see if there is any new message?
Any direct connection using socket? how should I proceed in this case?
Adan-
Sounds like you want to setup a standard consumer (or it may be called a receiver). Your use case is exactly the purpose of the consumer-side of the AMQP API.. see below
Note: Messaging systems often deploy a 'callback' or 'listener' model for asynchronous processing when receiving messages. That will still feel "instantaneous" from a data processing perspective. It is a different programming paradigm that is simpler to code as it does not require logic to break out of an infinite loop in the receiver/consumer pattern.
AMQP.NET lite receiver sample
I haven't found an existing post asking this but apologize if I missed it.
I'm trying to get my head round microservices and have come across articles where RabbitMQ is used. I'm confused why RabbitMQ is needed. Is the intention that the services will use a web api to communicate with the outside world and RabbitMQ to communicate with each other?
In Microservices architecture you have two ways to communicate between the microservices:
Synchronous - that is, each service calls directly the other microservice , which results in dependency between the services
Asynchronous - you have some central hub (or message queue) where you place all requests between the microservices and the corresponding service takes the request, process it and return the result to the caller. This is what RabbitMQ (or any other message queue - MSMQ and Apache Kafka are good alternatives) is used for. In this case all microservices know only about the existance of the hub.
microservices.io has some very nice articles about using microservices
A message queue provide an asynchronous communications protocol - You have the option to send a message from one service to another without having to know if another service is able to handle it immediately or not. Messages can wait until the responsible service is ready. A service publishing a message does not need know anything about the inner workings of the services that will process that message. This way of handling messages decouple the producer from the consumer.
A message queue will keep the processes in your application separated and independent of each other; this way of handling messages could create a system that is easy to maintain and easy to scale.
Simply put, two obvious cases can be used as examples of when message queues really shine:
For long-running processes and background jobs
As the middleman in between microservices
For long-running processes and background jobs:
When requests take a significant amount of time, it is the perfect scenario to incorporate a message queue.
Imagine a web service that handles multiple requests per second and cannot under any circumstances lose one. Plus the requests are handled through time-consuming processes, but the system cannot afford to be bogged down. Some real-life examples could include:
Images Scaling
Sending large/many emails (like newsletters)
Search engine indexing
File scanning
Video encoding
Delivering notifications
PDF processing
Calculations
The middleman in between microservices:
For communication and integration within and between applications, i.e. as the middleman between microservices, a message queue is also useful. Think of a system that needs to notify another part of the system to start to work on a task or when there are a lot of requests coming in at the same time, as in the following scenarios:
Order handling (Order placed, update order status, send an order, payment, etc.)
Food delivery service (Place an order, prepare an order, deliver food)
Any web service that needs to handle multiple requests
Here is a story explaining how Parkster (a digital parking service) are breaking down their system into multiple microservices by using RabbitMQ.
This guide follow a scenario where a web application allows users to upload information to a web site. The site will handle this information and generate a PDF and email it back to the user. Handling the information, generating the PDF and sending the email will in this example case take several seconds and that is one of the reasons of why a message queue will be used.
Here is a story about how and why CloudAMQP used message queues and RabbitMQ between microservices.
Here is a story about the usage of RabbitMQ in an event-based microservices architecture to support 100 million users a month.
And finally a link to Kontena, about why they chose RabbitMQ for their microservice architecture: "Because we needed a stable, manageable and highly-available solution for messaging.".
Please note that I work for the company behind CloudAMQP (hosting provider of RabbitMQ).
The same question can be why REST is necessary for microservices? Microservice concept is not something new under moon. A long time distribution of workflow was used for backend engineering and asynchronous request processing, Microservice is the same component in a separated jvm which matches with S(single responsibility) in SOLID. What makes it micro SERVICE - is that it is balanced. And that is the all! Particularly (!), it can be REST Service on Spring Cloud/REST base, which is registered by Eureka, has proxy gateway and load balancing over Zuul and Ribbon. But it is not the whole world of microservices!By the way, asynchronous distributed processing is one of tasks which microservices are used for. Long time ago services(components) in separated JVM was integrated over any messaging and the pattern is known as ESB. Microservices are the same subjects the pattern. Due to fashion for Spring Cloud REST seems like it is the only way of microservices. Nope! Message based asynchronous microservice architecture is supported by Vertx https://dzone.com/articles/asynchronous-microservices-with-vertx, for example. Why not to use RabbitMQ as message channel? In this case load balancing can be provided by building RabbitMQ cluster. For example:https://codeburst.io/using-rabbitmq-for-microservices-communication-on-docker-a43840401819. So, world is much wide more.
Background
We are trying to introduce a new architectural pattern in our company and are considering CQRS with Event Sourcing using a Service Bus. Technologies we are currently developing our POC with are NServiceBus, Event Store, and MSMQ. We would like to have a single endpoint in NServiceBus defined with two different transports, MSMQ for our commands and Event Store for our events. The current state of our enterprise does not permit us to easily switch everything to Event Store presently as we have significant investment in our legacy apps using MSMQ, which is a reason why we are considering the hybrid approach.
Question
Is it possible to create a single NServiceBus endpoint that uses different transports? If yes, how? If no, what alternatives are there?
Aaron,
I think the best option would be to use MSMQ as a transport in NServiceBus. Here's how it may look like:
send a command via MSMQ
in a command handler (re)create an aggregate which is the target of the command
invoke the operations
store the resulting events in the EventStore along with the command's message id to ensure idempotence. The aggregate itself will be responsible for knowing the commands it already processed
in a separate component (event processor) use EventStore persistent subscription APIs to hook to all events stream. Some of these processed events should cause sending a command. Such a command might be send via NServiceBus send-only endpoint hosted inside this event processor.
in that event processor you can also re-publish all events via NServiceBus and MSMQ. Such events should not be subscribed by other services (see note on autonomy below)
NServiceBus Sagas (process managers) should live inside your service boundary and react on commands and/or events sent or re-published by these event processors over MSMQ.
One remark regarding the service boundaries is that you have to decide what level of service autonomy suits you:
* Weak, where services can directly subscribe to other service event streams. In this design events that cross service boundary are obviously allowed to carry data.
* Strong, where services use higher-level events for communication and these events only carry the identity of things and no data. If you want something like this, you can use your event processors to map from ES events to these "higher level" events.
I have a question about an event driven design architecture.
I have a server application that contains many domain services. each service is a WCF service, and I want the services to communicate using raising events via a dedicated pub-sub infrastructure that I've created.
Now everything works well, but I have a question regarding the orchestration..
I have a manager service which accepts calls from the consumers and starts the process, sending a message to component A which in turn sends a message to component B and so on.
Eventually some of these services raise events for the manager service which keeps no state of incoming requests.
What is the best way to orchestrate or manage this flow?
Thanks
Take a look at NServiceBus and NServiceBus sagas in particular. A saga (the meaning in NSB is more akin to 'process manager') maintains state and correlates messages by some ID. In your case, the manager service is a sort of saga. It should maintain sate for each process that it manages. This state should be keyed by a correlation ID to be shared by all messages sent between components.