I am using MassTransit and RabbitMQ in both a "competing consumers" model and a Pub/Sub model.
3 tiers,
1st tier = UI, 2nd tier = gateway, 3rd tier = many distributed services
I have a working competing consumers model but I wish to do the following with Pub/Sub:
The gateway service publishes a message that all connected subscriber instances consume and then respond to to the gateway. The gateway doesn't respond to the UI until all its 3rd tier have responded, the gateway accumulates the response and finally passes back to the UI.
I cannot find a way to inspect MassTransit (whether I use SAGAs or not) in the 2nd tier to know how many subscribers i have in the 3rd tier (to work out if they've all responded). The overall goal is that the UI gets a single response with the accumulated results from the 3rd tier.
A similar question is here - no answers as yet.
UPDATE
Effectively I want to count the number of sinks on the inboundPipeline. Should I be doing this and is there a clean way to do it?
Pub/Sub in general doesn't allow you to know how many consumers for a given message exist. The whole idea is that you aren't coupled to that answer.
To do this, you need to build the solution into your application to keep track of that. When a consume comes up, publish a message for the gateway to register with it. When it shutdowns, do the same to remove that registration.
Related
I haven't found an existing post asking this but apologize if I missed it.
I'm trying to get my head round microservices and have come across articles where RabbitMQ is used. I'm confused why RabbitMQ is needed. Is the intention that the services will use a web api to communicate with the outside world and RabbitMQ to communicate with each other?
In Microservices architecture you have two ways to communicate between the microservices:
Synchronous - that is, each service calls directly the other microservice , which results in dependency between the services
Asynchronous - you have some central hub (or message queue) where you place all requests between the microservices and the corresponding service takes the request, process it and return the result to the caller. This is what RabbitMQ (or any other message queue - MSMQ and Apache Kafka are good alternatives) is used for. In this case all microservices know only about the existance of the hub.
microservices.io has some very nice articles about using microservices
A message queue provide an asynchronous communications protocol - You have the option to send a message from one service to another without having to know if another service is able to handle it immediately or not. Messages can wait until the responsible service is ready. A service publishing a message does not need know anything about the inner workings of the services that will process that message. This way of handling messages decouple the producer from the consumer.
A message queue will keep the processes in your application separated and independent of each other; this way of handling messages could create a system that is easy to maintain and easy to scale.
Simply put, two obvious cases can be used as examples of when message queues really shine:
For long-running processes and background jobs
As the middleman in between microservices
For long-running processes and background jobs:
When requests take a significant amount of time, it is the perfect scenario to incorporate a message queue.
Imagine a web service that handles multiple requests per second and cannot under any circumstances lose one. Plus the requests are handled through time-consuming processes, but the system cannot afford to be bogged down. Some real-life examples could include:
Images Scaling
Sending large/many emails (like newsletters)
Search engine indexing
File scanning
Video encoding
Delivering notifications
PDF processing
Calculations
The middleman in between microservices:
For communication and integration within and between applications, i.e. as the middleman between microservices, a message queue is also useful. Think of a system that needs to notify another part of the system to start to work on a task or when there are a lot of requests coming in at the same time, as in the following scenarios:
Order handling (Order placed, update order status, send an order, payment, etc.)
Food delivery service (Place an order, prepare an order, deliver food)
Any web service that needs to handle multiple requests
Here is a story explaining how Parkster (a digital parking service) are breaking down their system into multiple microservices by using RabbitMQ.
This guide follow a scenario where a web application allows users to upload information to a web site. The site will handle this information and generate a PDF and email it back to the user. Handling the information, generating the PDF and sending the email will in this example case take several seconds and that is one of the reasons of why a message queue will be used.
Here is a story about how and why CloudAMQP used message queues and RabbitMQ between microservices.
Here is a story about the usage of RabbitMQ in an event-based microservices architecture to support 100 million users a month.
And finally a link to Kontena, about why they chose RabbitMQ for their microservice architecture: "Because we needed a stable, manageable and highly-available solution for messaging.".
Please note that I work for the company behind CloudAMQP (hosting provider of RabbitMQ).
The same question can be why REST is necessary for microservices? Microservice concept is not something new under moon. A long time distribution of workflow was used for backend engineering and asynchronous request processing, Microservice is the same component in a separated jvm which matches with S(single responsibility) in SOLID. What makes it micro SERVICE - is that it is balanced. And that is the all! Particularly (!), it can be REST Service on Spring Cloud/REST base, which is registered by Eureka, has proxy gateway and load balancing over Zuul and Ribbon. But it is not the whole world of microservices!By the way, asynchronous distributed processing is one of tasks which microservices are used for. Long time ago services(components) in separated JVM was integrated over any messaging and the pattern is known as ESB. Microservices are the same subjects the pattern. Due to fashion for Spring Cloud REST seems like it is the only way of microservices. Nope! Message based asynchronous microservice architecture is supported by Vertx https://dzone.com/articles/asynchronous-microservices-with-vertx, for example. Why not to use RabbitMQ as message channel? In this case load balancing can be provided by building RabbitMQ cluster. For example:https://codeburst.io/using-rabbitmq-for-microservices-communication-on-docker-a43840401819. So, world is much wide more.
We have multiple web and windows applications which were deployed to different servers that we are planning to integrate using NservierBus to let all apps can pub/sub message between them, I think we using pub/sub pattern and using MSMQ transport will be good for it. but one thing I am not clear if it is a way to avoid hard code to set sub endpoint to MSMQ QueueName#ServerName which has server name in it directly if pub is on another server. on 6-pre I saw idea to set endpoint name then using routing to delegate to transport-level address, is that a solution to do that? or only gateway is the solution? is a broker a good idea? what is the best practice for this scenario?
When using pub/sub, the subscriber currently needs to know the location of the queue of the publisher. The subscriber then sends a subscription-message to that queue, every single time it starts up. It cannot know if it subscribed already and if it subscribed for all the messages, since you might have added/configured some new ones.
The publisher reads these subscriptions messages and stores the subscription in storage. NServiceBus does this for you, so there's no need to write code for this. The only thing you need is configuration in the subscriber as to where the (queue of the) publisher is.
I wrote a tutorial myself which you can find here : http://dennis.bloggingabout.net/2015/10/28/nservicebus-publish-subscribe-tutorial/
That being said, you should take special care related to issues regarding websites that publish messages. More information on that can be found here : http://docs.particular.net/nservicebus/hosting/publishing-from-web-applications
In a scale out situation with MSMQ, you can also use the distributor : http://docs.particular.net/nservicebus/scalability-and-ha/distributor/
As a final note: It depends on the situation, but I would not worry too much about knowing locations of endpoints (or their queues). I would most likely not use pub/sub just for this 'technical issue'. But again, it completely depends on the situation. I can understand that rich-clients which spawn randomly might want this. But there are other solutions as well, with a more centralized storage and an API that is accessed by all the rich clients.
I am getting my head around messaging. Currently we are spiking a few scenarios using Rebus. We are also considering NServiceBus.
The scenario we are trying to build is a proof of concept for a background task processing system. Today we have a handful of backend services hosted in different ways. (web, windows services, console apps) I am looking to hook them up to rebus and start consuming messages using competing consumer, some mesages will have one listener and some will share the load of messages. Elegant :)
I got a pretty good start from this other question How should I set rebus up for one producer and many consumers and it is working nicely in the proof of concept.
Now I want to start reporting progress. My intital approach is to set up pub/sub as well and spin up a service that listens to progress events from all the services. And if a service is interrested in a specific progress in the future it is easy to subscripe of interrest to the messages and start listening.
But how shall I approach setting up both competing consumer and pub/sub? it is dimply two separate things? (In the rebus case one adapter using UseSqlServerInOneWayClientMode / UseSqlServer and another adapter that is set up for the pub/sub using whatever protocol we want?)
Or is there a better solution then having two "buses" here?
I've built something like that myself a couple of times, and I've had pretty good results with using SignalR to report progress from this kind of backend worker processes.
Our setup had a bunch of WPF clients, one single SignalR hub, and a bunch of backend worker processes. All WPF clients and all backend workers would then establish a connection to the hub, allowing workers to send progress reports while doing their work.
SignalR has some nice properties that makes it very suitable for this exact kind of problem:
The published messages "escape" the Rebus unit of work, allowing progress report messages to be sent several times from within one single message handler even though it could take a long time to complete
It was easy to get the messages all the way to the clients because they subscribe directly
We could use the hub groups functionality to group users so we could target progress/status messages from the backend at either all users or a single user (could also be used for departments, etc.)
The most important point, I guess, is that this progress reporting thing (at least in our case) was not as important as our Rebus messages, i.e. it didn't require the same reliability etc, which we could use to our advantage and then pick a technology with some other nice properties that turned out to be cool.
I work on a system where we have the same website across multiple countries. Each of these websites has it's own services. Everything works well, but I've always found myself having to send messages rather than publishing as the messages otherwise other services where I know before hand it's completely irrelevant. It sounds pointless to me publishing to many services and then filtering it's relevance.
Is there a practice I should be dealing with when wanting to publish messages to a certain subset of services, how have others dealt with this problem?
By default endpoints subscribe to all messages. If you want only certain endpoints to subscribe to specific sets, then you need to configure your endpoint to DoNotAutoSubscribe(). You then must explicitly subscribe to each message type the endpoint will be interested in using Bus.Subscribe().
Could you describe your logic of determining relevance for particular endpoint systems ? the purpose of publishing and subscribing is that there are events in a system that other endpoints can subscribe to.
you should not know something about your subscribers. so how do you determine relevance ?
if these messages are not relevant for a specific endpoint why do you want to subscribe to these messages ?
If it truly is an event message then you need to publish the message. If you need to publish to a subset you could have a separate subscription store that the endpoint in question would use.
Typically it should be up to the subscriber to determine whether the received event is relevant but if you do have the information up-front then could go with the separate subscription store.
In my FOSS ESB project (http://shuttle.codeplex.com/) a ISubscriptionManager implementation has to be provided to the ESB to determine the subscriber uris to send published messages to. Although it may be overkill one could provide a custom implementation that contains some logic to perform the filtering; otherwise the separate subscription store.
I've been reading through the documentation on the NServiceBus site but am struggling to piece it all together.
The goal is to provide a durable messaging solution between on-premise back office systems and a public facing web site in another data center.
I will need bidirectional (on-premise <> web site) pub-sub and request-response communication.
The documentation makes it clear that there isn't one central point that all communication goes through, but surely the subscriptions need to persisted somewhere (in a central location?).
The NServiceBus gateway does look like it would meet my requirements but I can't find any working examples of this.
Can someone provide a bit more detail on how the Gateway works and whether it will meet my requirements?
The subscriptions are persisted on each publisher endpoint. Say you have a service endpoint publishing web orders. All other services who are interested can subscribe by sending a subscription message to the publisher, who then stores the subscriptions locally. When a message is available the publisher evaluates the subscriptions and send a message to each of the subscribers.
This brings us onto your other requirement - that of request/response. Because NSB is based on msmq, everything is asynchronous. The most a publisher could do is send a response to a caller just saying that the request has been received and will be published. The nature of async messaging means that you cannot have a synchronous response from any downstream subscribers.
But this cost comes with benefits - namely reliability and availability.
Reliability - because you are using durable messaging the messages will eventually be delivered, at which point a response can be generated which will also eventually find it's way back to the caller. This is highly reliable when compared to request response.
Availability: because the publisher service is always able to send a message (whether a downstream subscriber is available or not), it never needs to block incoming requests. If you load balance your publisher somehow you can easily achieve availability at enterprise levels.
However you need to balance this against your latency requirements. Asynchrony usually equals latency. So if you have latency requirements in the sub-100 ms range NSB may not be your best bet.
Apologies for not answering your query about NSB Gateway - I have never used it.
Hope this helps.
The Gateway solves the communication problem between sites. It will ensure that messages get delivered from SiteA to SiteB. The messages are hashed and validated on the other end. Apparently there isn't an example of this in 2.5, so I'm thinking of throwing one together as this has come up a few times in the past month.