I need to create a RabbitMQ-based microservices solution in Nestjs. To scale the system, I need to deploy several instances (e.g., 3 dockers) of the same microservice (server) listening to the same RabbitMQ topic, processing requests coming from the consumer (client) (see the following figure). I assume these 3 instances use a kind of Round Robin to pop up messages/requests from the queue/topic and process them and then return the results in another queue (i.e., 'callback' queue) to the consumer. I am wondering if
Nestjs already handles the correlation_id (to correlate responses with requests | to tackle the issue, having received a response in that queue it's not clear to which request the response belongs.) under the hood, or do I need to extend the current library?
Does NestJs create an exclusive callback queue (when several instances of the same microservices are deployed) or do I need to take care of that myself in my code (i.e., The worker (aka: server | microservice) is waiting for requests on that queue. When a request appears, it does the job and sends a message with the result back to the Client/Consumer, using the queue from the reply_to field.). In other words, The client (consumer) waits for data on the callback queue. When a message appears, it checks the correlation_id property. If it matches the value from the request it returns the response to the application.
Related
I am using a RabbitMQ Server (v3.8.9) with Java clients.
Use case is:
Our Backend creates messages for different clients. We send them out to their respective Endpoints.
1 Producer -> Outbound Queue -> 1 Consumer
The producer creates messages for n clients
Which the consumer should send out to the clients' endpoints
Messages must be kept in the correct order regarding each client
Works fine, unless all clients are up and running. Problem: If one client becomes unavailable, we need to have a bulletproof retry mechanism for that.
Say:
Wait 1 Minute and try again
All following messages must NOT be delivered before the first failed one and kept in the correct order
If a retry works, then ALL other messages should be send to the client immediately
As you can see, it is not a solution to just "supsend" the consumer, because it should still deliver msg to the other (alive) clients. Due to application limitations and a dynamic number of clients, we cannot spawn one consumer per client queue.
My best approach right now is to dynamically create one queue per client, which are then routed to a single outbound queue. If one msg to one client cannot be delivered by the consumer, I would like to "pause" the clients queue for x minutes. An API call like "queue_pause('client_q1', '5 Minutes')" would help. But even then I have to deal with the other, already routed messages to that particular client and keep them in the correct order...
Any better ideas?
I think the key here is that a single consumer script can consume from multiple queues. So if I'm understanding correctly, you could model this as:
Each client has its own queue. These could be created by the consumer script when it starts up, or by a back-end process when a new client is created.
The consumer script subscribes to each queue separately
When a message is received, the consumer tries to send it immediately to the client; if it succeeds, it is manually acknowledged with basic.ack, and the consumer is ready to send the next message to that client.
When a message cannot be delivered to the client, it is requeued (basic.nack or basic.reject with requeue=1), retaining its position in the client's queue.
The consumer then needs to pause consuming from that particular queue. Depending on how its written, that could be as simple as a sleep in that particular thread, but if that's not practical, you can effectively "pause" the subscription to the queue:
Cancel the subscription to that queue, leaving other subscriptions in tact
Store the queue name and the retry time in an appropriate variable
If the consumer script is implemented with an event/polling loop, check the list of "paused" subscriptions each time around that loop; if the retry time has been reached, re-subscribe.
Alternatively, if the library / framework supports it, register a delayed event that will fire at the appropriate time and re-subscribe the queue. The exact mechanics of this depend on the technologies you're using.
All the other subscriptions will continue, so messages to other clients will be delivered. The queue with no subscribers will retain the messages for the offline client in order until the consumer script starts consuming them again.
We are building an application with a microservice architecture.
The microservice architecture will follow a message-oriented pattern, with AWS SQS.
We would like to return completion results from the consumer service back to the producer service.
This is the algorithm we are considering:
Producer creates a message with a unique id
Producer subscribes to a Redis channel that is named with the message id
Producer places the message onto the SQS queue
Consumer removes the message from the SQS queue and performs an operation
Consumer publishes the results of the operation to the Redis channel that is named with the message id
Producer recieves the completion results and resumes execution
Is this a reasonable way to pass message-completion results from a consumer back to a producer?
After continued research, it became apparent that message queues are not part of the solution. Point #5 in this article, "...or doesn’t even care about the result..." suggests (by implication) that we are simply using the wrong approach.
We changed our design so that request ordering is not important, and will make direct calls to AWS Lambda functions using the invoke api.
The issue is we have some modern web applications that are integrated with a legacy system that was never designed to support multiple concurrent requests from a single user. Basically there are certain types of requests that the legacy system can only handle one-at-a-time from a single user. It can handle multiple concurrent requests coming from different users, but for technical reasons cannot handle multiple from a single user. In these situations, the user's first request will complete successfully, but any subsequent requests from that same user that come in while the first request is still executing will fail.
Because our apps are ajax enabled, multi-tab/multi-browser friendly, and just the fact that there are multiple apps - there are certain scenarios where a user could wind up having more than one of these types of requests being sent to the legacy system at the same time.
I'm trying to determine if something like RabbitMQ could be positioned in front of the legacy system and leveraged to single-thread requests per user/IP. The thinking being that the web apps would send all requests to MQ, and they'd stack into per-user queues and pass on to the legacy system one at a time.
I don't know if there would be concerns about the potential number of queues this could create - we have a user-base of approx 4,000.
And I know we could somewhat address this in the web apps individually, but since there are multiple apps it'd be duplicating logic across them, and you'd still have the potential for two different apps to fire off concurrent requests.
Any feedback would be appreciated. Thanks-
I'm not sure a unique queue per user will work as you would need to have a backend worker process listening for messages on that queue that would need to be dynamically created.
Below is one option but it does have a performance bottleneck potential as a single backend process would be handling all requests sequentially. You could use multiple worker processes but you wouldn't know if one had completed before the other causing a race condition if your app requires a specific sequence of actions.
You could simply put all transactions (from all users) into a single queue and have a backend process pull off of that queue and service the request. If there needs to be a response back to the user once the request was serviced, then the worker process could respond back to a separate queue with a correlationID that could be used to send the response date back to the correct user.
I've done this before with ExpressJS apps where the following flow would happen:
The user/process/ajax makes a request
Express takes the payload from the request object and sends it to a RabbitMQ queue with a unique correlationId (e.g. UUID).
Express then takes the response object and stores it in a responseStore object with the key being the correlationId
Meanwhile, a backend worker process pulls the item from the queue, does some work and then sends a message to a different response queue with the same correlationId
The ExpressJS application has a connection to the response queue and when it receives a message, it takes the correlationId from the response and looks for a response object stored with same correlationId in the responseStore. If it finds it, it takes the payload from the message and does something like response.send(payload) or response.json(payload)
To do this, you should also have a mechanism that stores the creation time of the response object in the responseStore along with the response object. Then have a separate process that will check the responseStore and clean up old response objects after a certain timeout in case there are issues with the backend process completing.
Look here for more info on RPC with RabbitMQ:
https://www.rabbitmq.com/tutorials/tutorial-six-javascript.html
Hope this helps.
TL;DR - Whats the best way to expose RabbitMQ to a consumer via REST API?
I'm creating an API to publish and consume message from RabbitMQ. In my current design, the publisher is going to make a POST request. My API will route the POST request to the exchange. In this way, the publisher doesn't have to know the server address, exchange name etc. while publishing.
Now the consumer part is where I'm not sure how to proceed.
At the beginning there will be no queues. When a new consumer wants to subscribe to a TOPIC, then I will create a queue and bind it to the exchange. I need help with answers to few questions -
Once I create a queue for the consumer, what's the next step to let the consumer get messages from that queue?
I make the consumer ask for a batch of messages(say 50 messages) from the queue. Then once I receive an ack from the consumer I will send the next 50 messages from queue. If I don't receive an ack I will requeue the 50 messages back into the queue. Isn't this expensive in terms of opening and closing connection between the consumer and my API?
If there is a better approach then please suggest
In general, your idea of putting RMQ behind a REST API is a good one. You don't want to expose RMQ to the world, directly.
For the specific questions:
Once I create a queue for the consumer, what's the next step to let the consumer get messages from that queue?
Have you read the tutorials? I would start there, for the language you are working with: http://www.rabbitmq.com/getstarted.html
Isn't this expensive in terms of opening and closing connection between the consumer and my API?
Don't open and close connections for each batch of messages.
Your application instance (the "consumer" app) should have a single connection. That connection stays open as long as you need it - across as many calls to RabbitMQ as you want.
I typically open my RMQ connection as soon as the app starts, and I leave it open until the app shuts down.
Within the consumer app, using that one single connection, you will create multiple channels through the connection. A channel is where the actual work is done.
Depending on your language, you will have a single channel per thread; a single channel per queue being consumed; etc
You can create and destroy channels very quickly, unlike connections.
More specifically with your idea of batch processing, this will be handled by putting a consumer prefetch limit on your consumer and then requiring messages to be acknowledged after processing it.
I have a requirement to Queue web service requests and then process each request based on priority and request time. And then send response back.
The approach I'm thinking is as follows
1 Create a web service method to submit requests and enqueue requests.
2 Create two queues (high priority requests and lower priority requests)
3 Create a Processing method to process each request one at a time(dequeue the high priority queues first if it exists) process and then store the response
4 Create a dictionary to store response for the respective request.
5 create a web service method to get the response
I'm thinking to use in memory queue since I expect few number of requests queued at a time.
The problem I'm having is in step 3. I want the processor method to continuously run as long as there are requests in the queue.
How can I accomplish step 3 using WCF web service ?
I'm using .NET 4.0 environment.
I really appreciate any ideas or suggestions.
Thanks
I would create my service contract to make it clear that the operations will be queued. Something like:
[OperationContract]
string EnqueueRequest(int priority, RequestDetails details);
[OperationContract]
bool IsRequestComplete(string requestId);
I would have EnqueueRequest place each request into an MSMQ queue. I'd have a Windows Service processing the requests in the queue. That service would be the only process that has access to the SDLC device.
Have you coded the service in a plane jane, meat and potatoes way and profiled to see if it's necessary to queue requests? There is overhead involved in queuing. It's a good idea to do some measurement and see if just servicing requests is adequate.
Another way to approach it would be to use Microsoft Message Queue. Here is even some tight integration between message queues and WCF. The thought is, if you do actually need a queue, why not use something that is already built and tested.