I have a use case where I need my controller action to wait for the reception of a specific rabbitmq message so I can return the result to the client, this message would come from a separate worker performing a certain task.
My api project and the worker project are separated and rabbitmq bus is the only intermediary between them.
EDIT: This is the current Scenario:
Client sends request to the web api to ask for let's call it 'DATA'
The web api publishes a Message-A through rabbitmq
A separate service project handles the published Message-A, does some work, and publishes a new Message-B that contains the result of that work which we called 'DATA'
Here is the problem: My web api controller have to return the results contained in Message-B, so the controller action should wait for that message before returning to the client
You need to use a TaskCompletionSource<T>.
You need to subscribe to the reply messages and, if it's the reply you're waiting for, set the result of the task completion source.
Then await the task of the task completion source.
Related
I'm working on a library for interacting with the Discord API. My current setup is:
A gateway, each handling x amounts of shards - so that I can spin up as many of these as I like to scale well. These gateways publish events received to a redis message queue.
A client, which subscribes to the message queue, and responds to events received.
However, there are some scenarios - working with message components - where I want a specific client to handle events related to that message. This client will then use the node.js event emitter to emit an event in itself which is then received by a 'collector' in my code.
Does anyone have any recommendations how I might stop other clients from picking up the event from the message queue, so that only this specific client picks it up? Is it possible for a subscriber to 'read' an event before it like accepts it? As then all clients could read an event to see if it like matches a list of events its waiting for?
I need to create a RabbitMQ-based microservices solution in Nestjs. To scale the system, I need to deploy several instances (e.g., 3 dockers) of the same microservice (server) listening to the same RabbitMQ topic, processing requests coming from the consumer (client) (see the following figure). I assume these 3 instances use a kind of Round Robin to pop up messages/requests from the queue/topic and process them and then return the results in another queue (i.e., 'callback' queue) to the consumer. I am wondering if
Nestjs already handles the correlation_id (to correlate responses with requests | to tackle the issue, having received a response in that queue it's not clear to which request the response belongs.) under the hood, or do I need to extend the current library?
Does NestJs create an exclusive callback queue (when several instances of the same microservices are deployed) or do I need to take care of that myself in my code (i.e., The worker (aka: server | microservice) is waiting for requests on that queue. When a request appears, it does the job and sends a message with the result back to the Client/Consumer, using the queue from the reply_to field.). In other words, The client (consumer) waits for data on the callback queue. When a message appears, it checks the correlation_id property. If it matches the value from the request it returns the response to the application.
I need to implement a logic on Retry. Inbound endpoint pushes the messages to Rest (Outbound). If the REST is unavailable, I need to retry for 1 time and put it in the queue. But the second upcoming messages should not do any retry, it has to directly put the messages in to queue until the REST service is available.
Once the service is available, I need to pushes all the messages from QUEUE to REST Service (in ordering) via batch job.
Questions:
How do I know the service is unavailable for my second message? If I use until Successful, for every message it do retry and put in queue. Plm is 2nd message shouldn't do retry.
For batch, I thought of using poll, but how to tell to poll, when the service becomes available to begin the batch process. (bcz,Poll is more of with configuring timings to run batch)?
Other ticky confuses me is - Here ordering has to be preserved. once the service is available. Queue messages ( i,e Batch) has to move first to REST Services then with real time. I doubt whether Is it applicable.
It will be very helpful for the quick response to implement the logic.
Using Mule: 3.5.1
I could try something like below: using flow controls
process a message; if exception or bad response code, set a variable/property like serviceAvailable=false.
subsequent message processing will first check the property serviceAvailable to process the messages. if property is false, en-queue the messages to a DB table with status=new/unprocessed
create a flow/scheduler to process the messages from DB sequentially, but it will not check the property serviceAvailable and call the rest service.
If service throws exception it will not store the messages in db again but if processes successfully change the property serviceAvailable=true and de-queue the messages or change the status. Add another property and set it to true if there are more messages in db table like moreDBMsg=true.
New messages should not be processed/consumed until moreDBMsg=false
once moreDBMsg=false and serviceAvailable=true start processing the messages from queue.
For the timeout I would still look at the response code and catch time-outs to determine if the call was successful or requires a retry. Practically you normally do multi threading anyway, so you have multiple calls in parallel anyway. Or simply one call starts before the other ends.
That is just quite normal.
But you can simply retry calls in a queue that time out. And after x amounts of time-outs you "skip" or defer the retry.
But all of this has been done using actual Mule flow components like either:
MEL http://www.mulesoft.org/documentation/display/current/Mule+Expression+Language+Reference
Or flow controls: http://www.mulesoft.org/documentation/display/current/Choice+Flow+Control+Reference
Or for example you reference a Spring Bean and do it in native Java code.
One possibility for the queue would be to persist it in a database. Mule has database connector that has a "poll" feature, see: http://www.mulesoft.org/documentation/display/current/JDBC+Transport+Reference#JDBCTransportReference-PollingTransport
Most of the examples that I've seen on Workflow Services put activities between the Receive and SendReply activities. However, if the activities take a long time to complete the service timesout. I could increase the timeout or put the activities after the SendReply. Is there a best practice on where to run these activities?
There is no need to keep all activities between Receive and send reply. Your activities will be executed after completion SendReply activity. For a log running process send reply can send its client message related that service is started or any exception. Workflow will be executing after sendreply completion.
You can follow this approach..
1. Put receive activity as first activity on the workflow.
2. Apply validation on Data contract used as argument.
3. Put a code activity that can set WorkflowinstanceID in out parameter that can return as response from send reply. This is can
be used to control Workflow.
4. Add another send reply by right click on Receive activity, return response if any validation faults occur.
5. Put rest of activity below of send reply configure service behavior for any unhandled exception.
My WCF service(hosted as Windows Service), has some 'SendEmail' methods, which sends out emails after doing some processing.
Now, I have got another requirement where client wants to preview emails before they are being sent out, so my WCF service needs to return whole email object to calling web app.
If client is happy with emails object, they can simply click 'Send out' which will then again call WCF service to send the emails.
Because at times it can take a bit longer for emails object processingy, I do not want calling application to wait until emails object is ready.
Can anyone please guide what changes I need to make to my WCF service (which currently has all one way operation)?
Also, please guide me whether I need to go for Asynch operation or message queuing or may be a duplex contract?
Thank you!
Based on your description I think you will have to:
Change current operation from sending email to storing email (probably in database).
Add additional operation for retrieving prepared emails for current user
Add additional method to confirm sending one or more emails and removing them from storage.
The process will be:
User will trigger some http request which will result in calling your WCF service for processing (first operation)
WCF service will initiate some processing (asynchronously or firt operation will be one-way so that client doesn't have to wait).
Processing will save email somehow
Depend on duration of processing you can either use AJAX to poll WebApp which will in turn poll WCF service for prepared emails or you will create separate page which will user have to access to see prepared emails. Both methods are using second operation.
User will check prepared email(s) and trigger http request which will result in calling third operation to send those emails.
You have multiple options:
Use Ladislav's approach. Only to add that service returns a token and then client uses the token to poll until a time out or a successful response. Also server keeps these temp emails for a while and after a timeout purges them.
Use duplex communication so that server also gets a way to callback the client and does so when it has finished processing. But don't do this - and here is my view why not.
Use an Asynchronous approach. You can find nice info here.