I am trying to use rabbitmq and express to build a web application. I am very confused about the producer and receiver.
The tutorial said
Producing means nothing more than sending. A program that sends messages is a producer, Consuming has a similar meaning to receiving. A consumer is a program that mostly waits to receive message.
Is that mean the server should be both producer and receiver? Because the server needs to get the client's request and send a response to the client. If that's the case, why we need to run the 2 scripts (producer.js and receiver.js) separately?
Thank you
Your question appears to be mixing two different conceptual models.
"the server needs to get the client's request and send a response to the client" - That, in a nutshell, is the "client-server" paradigm/architecture, as typically used by web applications, where the client is the browser ("show me this web page, please") and the web app is the server ("OK, here is that page you requested").
Rabbit supports a very different paradigm/architecture (messaging), where (typically) the producer generates messages as they naturally occur (not based on any request). Those messages sit in one or more queues, waiting to be consumed by whoever has subscribed to those queues (the consumers). The consumer does not typically send requests to the server; instead it periodically checks to see if there are any new messages on the queue - and then consumes them.
Which paradigm you use depends on the purpose of your web application. Your app may even use both paradigms.
Update
"Could you please give an example that uses messaging paradigm? I couldn't imagine how a website works without request and response."
When you say "a web site" then I will assume you mean web pages displayed in a browser, provided by a web application running on a server. In that specific case, then, yes, the standard approach is the "client-server" model. There is typically no messaging involved. Messaging is not a good fit for that type of model. Client-server is a great model.
But imagine that "web application" which is sending those web pages to the browser. It is quite possible that the web application may also provide additional services, not accessed directly via web page requests. One such service may be messaging-based: The application produces messages - for example, notifications of events relevant to the application. And users can consume those messages.
A very crude example: Imagine an auction web site. You can log on to the site and see web pages where you can place bids to buy items.
But you may also want to receive messages (e.g. via texts) telling you when there have been new bids from other people, placed on specific items that you are interested in. You don't want to have to constantly refresh a web page to get the latest data; and you don't want to receive updates about every bid on every product. This is a better fit for a different type of client - not one which requests web pages, but one which consumes messages.
So, I also can't imagine a web site which uses messaging to provide web pages (although I bet someone somewhere has built that). But I can imagine a web-based application which serves web pages (client-server) and which maybe also produces messages. Or a web-based application which produces messages, and may not even have any web pages at all.
Your producer.js is a tiny example of a program which is only a message producer. Your consumer.js program is a tiny example of a program which is only a message consumer. And sitting in between them is the message broker program (Rabbit) which is where the queues exist, and where messages are sent (and retrieved).
Related
Problem description
i am working on a Xamarin application that consumes a REST API written in Python flask.
The Xamarin application offers virtual shopping lists where user can collaborate on buying stuff they have on a shared list.
To improve the user experience, i want to be able to actively notify the user about finished items on the list.
Possible solutions:
Synchronous API polling from client side
Notifications are stored by the API in a relational database and have a flag indicating if the user received the notification already.
The API has an endpoint GET /users/:user_id/notifications/ that queries the database for notifications and returns a JSON response with those.
Advantages
fairly simple to implement
Problems
synchronous polling creates a huge amount of http requests
API service remains stateless, making a horizontal scaling with a loadbalancer easier
Websocket endpoint on the API
The API has an endpoint POST /users/:user_id/notifications/register which creates a websocket connection between client and API.
The connection is stored to a global array in which each entry maps a client id to a websocket connection.
When a new notification is created, the endpoint makes a lookup in the connection dictionary by comparing the owner id of the notification with the dictionary entries. The notification is sent to appropriate user through the websocket.
Notifications are stored in the database like in the first approach.
When a user calls the endpoint, a new websocket connection will be established first and upon success the API sends all unseen notifications from the database to the user.
Advantages
API can push notifications to clients asynchronously
Problems
When a user terminates the websocket connection his dictionary entry will persis
Retaining one websocket connection per user permanently adds additional overhead to the API
Horizontal scalability of the API is more difficult because the service is not stateless anymore (Websocket connection information saved in
RabbitMQ
The API uses a RabbitMQ service to send notifications to the client. Every client uses subscribes to his own notification queue to prevent the broadcasting of messages.
Advantages
API remains stateless
Problems
Notifications needs to be resend to the exchange when a user is offline
Amount of queues grows drastically
Additional costs for RabbitMQ service
High temporary load on the RabbitMQ service when many users come online in the same time
Final words
It would be interesting to hear the opinion of others.
I believe the active distribution of notifications from backen services to clients i a very common use case.
best,
D
I would use RabbitMQ and consume events forwarding them as push notifications. This will work while the user is not actively connected to the website and enhance the engagement with each user experience that will return to the website when notified for more information see How to setup basic web push notification functionality using a Flask backend or How to send push notifications to a browser in ASP.NET Core or Sending Notifications with Spring Boot, Angular, and Firebase Cloud Messaging this way the RabbitMQ will not wait until the user is back online. If the user is online you can forward the notification directly to the Xamarin application via WebSockets and a load balancer like NGINX that can handle many WebSockets in an optimized way.
Synchronous API polling from the client-side is the less preferred way since it overloads the webserver with requests while nothing was changed.
I don't think the scalability of WebSocket is a problem. You can scale up easily with pub/sub. The hotspot of long connections is a kind of serious problem.
For one-way communication, I would suggest Server sent event. In the end, it usually depends on what your team is confident with.
I can recommend on a different approach for API that provides JSON which is called GraphQL
It supports subscriptions capabilities that are pushed by the GraphQL API Server (using web sockets)
GraphQL is considered today to be better than RESTful API since its very flexible and you can get exactly the data you need with one query.
I have read Why not publish NServiceBus messages from a web application and another similar question about this but I am not clear if this applies to service layer as well. For example, if the service layer is composed of web services or REST services built using WCF or Web API or any other way, should those services publish events or send commands? If those services are hosted in load balanced web servers, the problems outlined in the articles apply to this layer as well. How would the recommendation change or not change?
If I look from the definition of Event vs Command, the messages I am talking about are Events e.g. "a user was created" and so an event should be published. As a matter of fact, the service that created the user doesn't even know what else to do i.e. may be another application is supposed to create a customized portal for it and yet another application is supposed to send a welcome kit to the user. This would be an event and not a command. I guess I am hung up on the definition of a web application and application service when application service itself is composed of one or many web applications.
The definition of Web Application
A web application is an application that is accessed by users over a
network such as the Internet or an intranet.
However, to me, the users can be computers and thus web services are web applications and that is the reason for this question.
EDIT:
Let's consider a concrete example. An ASP.NET website (MVC or Web form - doesn't matter) displays the form to the operator, gets a post with data about user creation (Name, UserName, Password) and invokes a WCF service to create the user. In between website and WCF service we can put ServiceBus and send command to create the user (Request/Response) so that we get all the benefits described in the first article. WCF service is the actual business processing layer i.e. it would create the user. That is where I have the question. After the user is created, it should announce that a user has been created and other systems can react to it and do whatever they are supposed to do. So it fits perfectly the pattern of publish the message. However, the WCF service itself is a web application and thus has most of the traits of the web applications and thus the confusion.
As mentioned in the answer to the SO question you linked to, publishing event has more to do with where the actual processing takes places. Just as a side-note: it is not a matter of Send instead of Publish since that would imply that the two are interchangeable whereas they have rather different intentions. When you want to publish, you want to publish.
The same questions should arise if you find yourself publishing from your web-exposed integration layer: should you be performing the business processing in that code or rather sending it off to another endpoint for processing? Typically you should just send it off to another endpoint. You may even consider how you would perform the relevant action should anyone wish to invoke it. For instance, if you are publishing a UserCreatedEvent message it implies that you created a user. How would a user be created? Would I be forced to use the WCF / Web-Api layer or can I send a CreateUserCommand message on the bus that is processed by some application endpoint? If it is the former then you may need to rethink your design. However, if the latter you should be sending the command from your WCF / Web-Api anyway and the processing endpoint will perform the Publish bit :)
update:
My take on it is that it is more about cohesion / concerns. You would typically interact with your domain, from within your business, via a service bus for commands and events, and a simple query layer for reads. If you need to expose anything to a third-party (or simply via the web) then you use WCF / WS / Web-APi. The point is that you should try to avoid business processing in an integration endpoint (or in a front-end like a website). Business processing is better suited to application servers. There are usually exceptions to the rule but if you are in a position to influence the structure then you are in a better space.
The fact is, whatever code is truly responsible for performing the action should be the same which publishes the event. If you've got a MVC app and in the controller itself you're using Entity Framework to insert the User record, then that is exactly where the Publish should be, right after the SaveChanges call. If however, the controller calls a referenced binary or service which does the actions involved in the "add user" call, then the Publish should be there. My thought is the event should be right alongside the code that does the action whose event you are trying to publish.
So I'm looking into implementing NServiceBus in our current setup and just trying to get a better understanding of how things should be setup.
Our current setup consists of multiple clients (websites, scheduled tasks, etc..) calling a WCF service we have set up for handling the sending of emails. Of course, if the service goes down then our clients start getting errors and all of those messages are then lost (one of the reasons we want an ESB).
I've seen how you can configure your WCF service to handle nservicebus messages in a pub/sub setup. What I'm not sure on is what is the best way to set it up.
Setup 1:
Client (Publisher) -> NServiceBus handler (Subscriber) -> WCF Service
In this case, to scale you'd increase the number of handlers (hosted nservicebus services?), keeping just the one WCF service.
Setup 2:
Client (Publisher) -> WCF Service (Subscriber)
This one you just increase the number of WCF services to scale (updates would be a nightmare).
I just started looking into the ESB architecture in general so if I'm completely off let me know. I'm essentially just wanting to know what is working for you, and what the "best practice" tends to be.
Thanks!
I'm not completely clear on what you need WCF for anymore if you implement this via NServiceBus. Is the WCF component required for anything besides receiving messages (to send an email) from the multiple clients? If not, you could remove WCF from the equation.
From the sound of it, you will also want the Service to act as a single logical endpoint that handle requests to send emails. If that's the case, you will want to use Send (a command) instead of Publish (an event). Publish is used to broadcast an event, which means that something happened already; Send is used to instruct another component to do something. It sounds like you want the latter.
Scaling of an endpoint can be done via the Distributor. This may or may not be useful depending on where you expect the bottleneck to be.
Edit: Based on your comment, I would simply go with the second setup, and just add the handler to the WCF service. If you are hosting WCF in IIS, make sure you have something that wakes the process up if the app pool recycles (the incoming message won't wake it up the same way an incoming request to WCF will).
We do something similar internally where one NSB endpoint handles all the sending of email. The clients can either use NSB directly to Bus.Send() the command to send a message to the email endpoint or you can expose that endpoint via WCF as well (only to get the commands over to the endpoint). Once the endpoint has the commands, they would just call your existing service to maintain compatibility with your existing clients.
We have an asp.net mvc3 application which publishes a number of events. Recently, someone pointed out that we should be sending messages rather than publishing events from the web application and referenced this excellent blog post which makes perfect sense. However, I want to confirm that the points made in the post are still valid for NServiceBus v3.x?
Thanks!
I would imagine that the reasoning behind the post would always hold true, irrespective of the version of NServiceBus.
That being said, there is nothing stopping you from publishing a message from a web-site. The idea behind publishing a message is that the message represents an event that is typically produced by some processing endpoint. Since a web application should not really be processing anything but rather be sending commands off to a processing endpoint it would stand to reason that a web application should not be publishing events.
So if you find yourself in a situation where it seems to make sense to publish from your web application it indicates that you need to make a design decision: either the design is not optimal (so the web application is performing processing) or you are constrained in some way the prohibits the implementation of a processing endpoint (maybe a shared hosting environment).
I have load balanced web servers, that with the existing code base, handles when a user logs into the site. I would like to send a broadcast message to any applications that have subscribed saying 'hey x logged in' ? So have many web servers and many applications subscribing.
How does discovery work/configuration work with nservicebus ? Should each application know about each web server and subscribe individually or is this where the distributer comes in, so web servers all send to the 1 distributer and all application subscribe to single distributer and the distributer relays the message ?
I've tried to research this, but having troubles.
Thanks
MrT
First, see the guidance about publishing a message from a web application (or more specifically, about NOT publishing a message from a web application).
Taking all that in consideration, I would recommend your webservers Send() a message to a central event aggregator, which could then Publish() events that other applications (or your web applications) could subscribe to.
More specifically:
MyWeb on Webserver1 (with input queue MyWeb#Webserver1) Send()s a UserChangeMessage to queue WebEventBroker#CentralServer
WebEventBroker app, running on CentralServer, with input queue WebEventBroker#CentralServer, receives UserChangeMessage, and publishes UserChangedEvent
The MyWeb application subscribes to events of type UserChangedEvent, so when it is published, it is received by queues MyWeb#Webserver1 and MyWeb#Webserver2 where both web applications can process the message and take appropriate action.