I have a microservice architecture and now I need to introduce a notification center. Requirements are: any service is able to send a notification, any service is able to subscribe to any kind of notifications, UI (web) is able to subscribe to notifications (websockets are preferred). Of course I can write such service by myself but maybe there is ready-made robust solution for that.
UPD: I'm not looking for pub/sub messaging system as it is too low-level for notification center
What you are looking for is publish-subscriber messaging. If you are using AWS stack, then I can recommend Amazon SNS or Amazon SQS. I think Amazon SNS is more suitable because its push based.
Amazon SNS allows applications to send time-critical messages to multiple subscribers through a “push” mechanism, eliminating the need
to periodically check or “poll” for updates.
Amazon SQS is a message queue service used by distributed applications to exchange messages through a polling model, and can be
used to decouple sending and receiving components—without requiring
each component to be concurrently available.
Out of Amazon web services stack, there are a bunch of free messaging solutions:
RabbitMQ is one of the leading implementation of the AMQP protocol (along with Apache Qpid). Therefore, it implements a broker
architecture, meaning that messages are queued on a central node
before being sent to clients. This approach makes RabbitMQ very easy
to use and deploy, because advanced scenarios like routing, load
balancing or persistent message queuing are supported in just a few
lines of code. However, it also makes it less scalable and “slower”
because the central node adds latency and message envelopes are quite
big.
ZeroMq is a very lightweight messaging system specially designed for high throughput/low latency scenarios like the one you can find in
the financial world. Zmq supports many advanced messaging scenarios
but contrary to RabbitMQ, you’ll have to implement most of them
yourself by combining various pieces of the framework (e.g : sockets
and devices).
ActiveMQ is in the middle ground. Like Zmq, it can be deployed with both broker and P2P topologies. Like RabbitMQ, it’s easier to
implement advanced scenarios but usually at the cost of raw
performance.
Now you know what you need, I would recommend to read through each technology for a while and decide which one serves your goal more accurately. If that doesn't worth our time and your requirement is more specific & relatively small, then you can go for writing something on your own.
Related
My team wants to move to microservices architecture. Currently we are using Redis Pub/Sub as message broker for some legacy parts of our system. My colleagues think that it is naturally to continue use redis as service bus as they don't want spend their time on studying new product. But in my opinion RabbitMQ (especially with MassTransit) is a better approach for microservices. Could you please compare Redis Pub/Sub with Rabbit MQ and give me some arguments for Rabbit?
Redis is a fast in-memory key-value store with optional persistence. The pub/sub feature of Redis is a marginal case for Redis as a product.
RabbitMQ is the message broker that does nothing else. It is optimized for reliable delivery of messages, both in command style (send to an endpoint exchange/queue) and publish-subscribe. RabbitMQ also includes the management plugin that delivers a helpful API to monitor the broker status, check the queues and so on.
Dealing with Redis pub/sub on a low level of Redis client can be a very painful experience. You could use a library like ServiceStack that has a higher level abstraction to make it more manageable.
However, MassTransit adds a lot of value compared to raw messaging over RMQ. As soon as you start doing stuff for real, no matter what transport you decide to use, you will hit typical issues that are associated with messaging like handling replies, scheduling, long-running processes, re-delivery, dead-letter queues, and poison queues. MassTransit does it all for you. Neither Redis or RMQ client would deliver any of those. If your team wants to spend time dealing with those concerns in their own code - that's more like reinventing the wheel. Using the argument of "not willing to learn a new product" in this context sounds a bit weird, since, instead of delivering value for the product, developers want to spend their time dealing with infrastructure concerns.
RabbitMQ is far more stable and robust than Redis for passing messages.
RabbitMQ is able to hold and store a message if there is no consumer for it (e.g. your listener crashed , etc).
RabbitMQ has different methods for communication: Pub/Sub , Queue. That you can use for load balancing , etc
Redis is convenient for simple cases. If you can afford losing a message and you don't need queues then I think Redis is also a good option.
If you however can not afford losing a message then Redis is not a good option.
I have a set of RESTful services that my Angular 5 client uses to perform CRUD and business operations for the application. These are a set of micro services and they use pub/sub message queues to communicate between them, e.g. when a user is created the user server publishes a UserCreated event to the message queue and subscribers can listen for this event and act upon it as required.
Now, this is all good but i was thinking that wouldn’t it be better if the Angular 5 application itself published the event to the message queue rather than making HTTP POST/PUT or DELETE and only make GET requests against the API?
So repeating the example above the Angular 5 client would publish a CreateUserEvent to the message bus (in my case cloud pub/sub), I could then have services subscribe to these events and act upon them. My RESTful services would then only expose GET /users and GET /user/:id for example.
I know that this is doable and I guess what I am describing is CQRS, but I am keen to understand if publishing events to a message bus from the UI is good practice?
The concept of Messaging Bus is very different than Microservices. Probably, the answer to your question lies in the way you look at these two, from architectural perspective.
A messaging bus(whether it is backend specific or frontend specific) is designed in such a way, that it serves the purpose of communication of entities within the confined boundary of an environment, i.e. backend or frontend.
Whereas on the other hand, microservices architecture is designed in such a way that, two different environments that may be backend-frontend or backend-backend, can "effectively" communicate.
So there is a clear separation of motivation behind both the concepts. Now, from your viewpoint, you may use a hybrid approach which might work, and it may also lead to interesting findings related to performance, architectural design or overheads as well.
Publishing directly from the client is possible, but the caveat is that it means that the client needs to have the proper credentials to publish. For this reason, it may be preferable to have the service do the publishing in response to requests sent from the clients.
I haven't found an existing post asking this but apologize if I missed it.
I'm trying to get my head round microservices and have come across articles where RabbitMQ is used. I'm confused why RabbitMQ is needed. Is the intention that the services will use a web api to communicate with the outside world and RabbitMQ to communicate with each other?
In Microservices architecture you have two ways to communicate between the microservices:
Synchronous - that is, each service calls directly the other microservice , which results in dependency between the services
Asynchronous - you have some central hub (or message queue) where you place all requests between the microservices and the corresponding service takes the request, process it and return the result to the caller. This is what RabbitMQ (or any other message queue - MSMQ and Apache Kafka are good alternatives) is used for. In this case all microservices know only about the existance of the hub.
microservices.io has some very nice articles about using microservices
A message queue provide an asynchronous communications protocol - You have the option to send a message from one service to another without having to know if another service is able to handle it immediately or not. Messages can wait until the responsible service is ready. A service publishing a message does not need know anything about the inner workings of the services that will process that message. This way of handling messages decouple the producer from the consumer.
A message queue will keep the processes in your application separated and independent of each other; this way of handling messages could create a system that is easy to maintain and easy to scale.
Simply put, two obvious cases can be used as examples of when message queues really shine:
For long-running processes and background jobs
As the middleman in between microservices
For long-running processes and background jobs:
When requests take a significant amount of time, it is the perfect scenario to incorporate a message queue.
Imagine a web service that handles multiple requests per second and cannot under any circumstances lose one. Plus the requests are handled through time-consuming processes, but the system cannot afford to be bogged down. Some real-life examples could include:
Images Scaling
Sending large/many emails (like newsletters)
Search engine indexing
File scanning
Video encoding
Delivering notifications
PDF processing
Calculations
The middleman in between microservices:
For communication and integration within and between applications, i.e. as the middleman between microservices, a message queue is also useful. Think of a system that needs to notify another part of the system to start to work on a task or when there are a lot of requests coming in at the same time, as in the following scenarios:
Order handling (Order placed, update order status, send an order, payment, etc.)
Food delivery service (Place an order, prepare an order, deliver food)
Any web service that needs to handle multiple requests
Here is a story explaining how Parkster (a digital parking service) are breaking down their system into multiple microservices by using RabbitMQ.
This guide follow a scenario where a web application allows users to upload information to a web site. The site will handle this information and generate a PDF and email it back to the user. Handling the information, generating the PDF and sending the email will in this example case take several seconds and that is one of the reasons of why a message queue will be used.
Here is a story about how and why CloudAMQP used message queues and RabbitMQ between microservices.
Here is a story about the usage of RabbitMQ in an event-based microservices architecture to support 100 million users a month.
And finally a link to Kontena, about why they chose RabbitMQ for their microservice architecture: "Because we needed a stable, manageable and highly-available solution for messaging.".
Please note that I work for the company behind CloudAMQP (hosting provider of RabbitMQ).
The same question can be why REST is necessary for microservices? Microservice concept is not something new under moon. A long time distribution of workflow was used for backend engineering and asynchronous request processing, Microservice is the same component in a separated jvm which matches with S(single responsibility) in SOLID. What makes it micro SERVICE - is that it is balanced. And that is the all! Particularly (!), it can be REST Service on Spring Cloud/REST base, which is registered by Eureka, has proxy gateway and load balancing over Zuul and Ribbon. But it is not the whole world of microservices!By the way, asynchronous distributed processing is one of tasks which microservices are used for. Long time ago services(components) in separated JVM was integrated over any messaging and the pattern is known as ESB. Microservices are the same subjects the pattern. Due to fashion for Spring Cloud REST seems like it is the only way of microservices. Nope! Message based asynchronous microservice architecture is supported by Vertx https://dzone.com/articles/asynchronous-microservices-with-vertx, for example. Why not to use RabbitMQ as message channel? In this case load balancing can be provided by building RabbitMQ cluster. For example:https://codeburst.io/using-rabbitmq-for-microservices-communication-on-docker-a43840401819. So, world is much wide more.
We have multiple web and windows applications which were deployed to different servers that we are planning to integrate using NservierBus to let all apps can pub/sub message between them, I think we using pub/sub pattern and using MSMQ transport will be good for it. but one thing I am not clear if it is a way to avoid hard code to set sub endpoint to MSMQ QueueName#ServerName which has server name in it directly if pub is on another server. on 6-pre I saw idea to set endpoint name then using routing to delegate to transport-level address, is that a solution to do that? or only gateway is the solution? is a broker a good idea? what is the best practice for this scenario?
When using pub/sub, the subscriber currently needs to know the location of the queue of the publisher. The subscriber then sends a subscription-message to that queue, every single time it starts up. It cannot know if it subscribed already and if it subscribed for all the messages, since you might have added/configured some new ones.
The publisher reads these subscriptions messages and stores the subscription in storage. NServiceBus does this for you, so there's no need to write code for this. The only thing you need is configuration in the subscriber as to where the (queue of the) publisher is.
I wrote a tutorial myself which you can find here : http://dennis.bloggingabout.net/2015/10/28/nservicebus-publish-subscribe-tutorial/
That being said, you should take special care related to issues regarding websites that publish messages. More information on that can be found here : http://docs.particular.net/nservicebus/hosting/publishing-from-web-applications
In a scale out situation with MSMQ, you can also use the distributor : http://docs.particular.net/nservicebus/scalability-and-ha/distributor/
As a final note: It depends on the situation, but I would not worry too much about knowing locations of endpoints (or their queues). I would most likely not use pub/sub just for this 'technical issue'. But again, it completely depends on the situation. I can understand that rich-clients which spawn randomly might want this. But there are other solutions as well, with a more centralized storage and an API that is accessed by all the rich clients.
I'm a little confused as to what the difference is between a message broker e.g. RabbitMQ and Message-orientated Middleware. I can't find much info apart from what's on Wikipedia. When searching MOM I find info on AMQP which states is a protocol for MOM.. what does this mean? What is MOM then? I also have read that RabbitMQ implements the AMPQ protocol.. so why does that make a RabbitMQ a messsage broker? Are a message broker and MOM the same thing?
Hope some can unravel my confusion. thanks
An overview -
A protocol - A set of rules.
AMQP - AMQP is an open internet protocol for reliably sending and receiving messages.
MOM (message-oriented-middleware) - is an approach, an architecture for distributed system i.e. a middle layer for the whole distributed system, where there's lot of internal communication (a component is querying data, and then needs to send it to the other component, which will be doing some processing on the data) so components have to share info/data among them.
Message broker - is any system (in MOM) which handles messages (sending as well as receiving), or to be more precise which routes messages to the specific consumer/recipient. A Message Broker is typically built upon a MOM. The MOM provides the base communication among the applications, and things like message persistence and guaranteed delivery. "Message brokers are a building block of Message oriented middleware."
Rabbitmq - a message broker; a MOM implementation; an open-source implementation of AMQP; as per Wikipedia:
RabbitMQ is an open source message broker software (sometimes
called message-oriented middleware) that implements the Advanced
Message Queuing Protocol (AMQP).
As you asked:
When searching MOM I find info on AMQP which states is a protocol for MOM.. what does this mean?
MOM is about having a messaging middleware (middle layer) between (distributed) system components, and AMQP is protocol (set of rules) for reliably sending and receiving messages. So, a MOM implementation (i.e. Rabbitmq) may use AMQP.
What is MOM then?
Message-Oriented-Middleware - is an approach, an architecture for distributed system i.e. a middle layer for the whole distributed system, where there's lot of internal communication (a component is querying data, and then needs to send it to the other component, which will be doing some processing on the data) so components have to share info/data among them.
In short it's a way to design a system, for example: depending upon the overall requirements we need to develop a distributed system, with some internal communication. The biggest advantage of MOM architecture/decision is decoupling of the components i.e. if we're going to change the data query component it'll have no effect on the data processing components, as they're communicating via MOM (e.g. Rabbitmq Cluster) - the data processing component is getting the data in form messages, which then parses and processes them.
MOM at the end is just a design decision, that we use a middleware for gluing our system (distributed) components, a middleware for handling communication between them, in the form of messages (i.e. JSON). To implement a message-oriented-middleware we need more - set of specific rules i.e. how the messages will be published, consumed, how the acknowledgement will work, the lifetime of a message is until it is consumed, the persistence of a message, etc. AMQP is basically these set of rules i.e. a standard/protocol for implementing a MOM i.e. a messaging system using AMQP, means it confines itself by the stated rules. From Wikipedia:
AMQP mandates the behavior of the messaging provider and client to the
extent that implementations from different vendors are inter-operable,
in the same way as SMTP, HTTP, FTP, etc. have created inter-operable
systems.
I also have read that RabbitMQ implements the AMPQ protocol.. so why does that make a RabbitMQ a message broker?
Yes, Rabbitmq is a message broker (publisher -> exchange -> queue -> consumer). It's an open source AMQP implementation i.e. a messaging system/broker which confines to AMQP (the AMQP rules) - one can use Rabbitmq as the middleware, hence MOM.
AMQP - is just a set of rules i.e .how messages will be published, kept (in queues), consumed, delivery acknowledgement, etc.
Are a message broker and MOM the same thing?
In simple words, Yes. If we need to go with MOM design for our distributed system, we can simply use Rabbitmq (a message broker; an AMQP implementation) as the middleware.
"MOM" broadly means any technology that can deliver "messages" from one user-space application to another. A message is usually understood to be a discrete piece of information, as compared to a stream.
MOM products used to be quite large and complex: CORBA, JMS, TIBCO, WebsphereMQ, etc. and tried to do a lot more than simply deliver messages.
A broker is a particular set of routing and queuing patterns, and we usually use the term "broker" specifically in MOM (as compared to HTTP, email, XMPP, etc.) Routing means, one message goes to one peer, to one of many peers, to all of many peers, etc. Queuing means messages are held in memory or disk until they can be delivered (and in some cases, acknowledged).
AMQP used to specific those broker patters, so an application could rely on consistent behavior from any AMQP-compatible broker (thus RabbitMQ and OpenAMQ looked much the same to a client app, like two HTTP or two XMPP servers would look the same). AMQP/1.0 specifies just the connection between nodes, so you don't have guarantees of behavior. This makes AMQP/1.0 much easier for firms to implement, but doesn't deliver interoperability.
ZeroMQ is message-oriented middleware that defines, like AMQP/1.0, the connections between pieces rather than the behaviour of a central broker. However it's relatively easy to write MOM brokers using 0MQ, and we've done a few of these (like Majordomo).
Message brokers are one (quite popular) kind of MOM. Another kind of MOM would be brokerless MOM, like ZeroMQ. With broker based MOM, all messages go to one central place: broker, and get distributed from there. Broker less MOM usually allows for peer to peer messaging (but does not exclude option of central server as well) .
AMQP is broker based MOM protocol definition (at least all versions prior to 1.0, which drifts into more general MOM), and there are several different Message brokers implementing that protocol, RabbitMQ is just one of them.