In mule I have a CSV file containing 50 rows of records like product id, product name, quantity, price, offer, expire date. I want to convert the CSV format to a JSON format and I use an external API using the above data. While in mid calling, the API network goes down. How can I retry with same data.
Can anybody shed light on this
You would have to create an exception catcher that would log the failure requests, then run through those again.
Do not keep "looping" the same record until the API comes up. This will block all others from being processed and your queue will not decrease.
See this post which explains how to setup batch jobs and this post on how to create error handlers; both from the mulesoft blog.
Take a look at the until-successful router: https://developer.mulesoft.com/docs/display/current/Until+Successful+Scope
NOTE: This is by default asynchronous, but can be configured to be synchronous.
Related
I'm looking for best practices or the recommended approach to test async code execution with Karate.
Our use cases are all pretty similar but a basic one is:
Client makes HTTP request to API
API accepts request and creates a messages which is added to a queue
API replies with ACCEPTED / 202
Worker picks up message from queue processes it and updates database
Eventually after the work is finished another endpoint delivers updated data
How can I check with Karate that after processing has finished other endpoints return the correct result?
Concrete real life example:
Client requests a processing intensive data export to API e.g. via HTTP POST /api/export
API create message with information for creating the export and puts it on AWS SQS queue
API replies with 202
Worker receives message and creates export, uploads result as ZIP to S3 and finally creates and entry in the database symbolizing this export
Client can now query list exports endpoint e.g. via HTTP GET /api/exports
API returns 200 with the list of exports incl. the newly created entry
Generally I have two ideas on how to approach this:
Use karate retry until on the endpoint that returns the list of exports
In the API response (step #3) return the message ID and use the HTTP API of SQS to poll that endpoint until the message has been processed and then query the list endpoint to check the result
Is either of those approach recommended or should I choose an entirely different solution?
The moment queuing comes into the picture, I would not recommend retry until. It would work if you are in a hurry, but if you are ok to write a little bit of Java code, please read on. Note that this Java "glue code" needs to be written only once, and then the team responsible for writing the functional flows will be up and running.
I personally would prefer option (2) just because when a test fails, you will have a lot more diagnostic information and traces to look at.
Pretty sure you won't have a problem using AWS Java libs to do things such as polling SQS.
I think this example will answer all your questions: https://twitter.com/getkarate/status/1417023536082812935
I am building an application which uses the Amazon MWS API.
The API has limits for how frequently you can hit it.
I am looking for a tool that can act as a reverse-proxy, save the MWS API responses, and eventually masquerade as the MWS API without ever hitting it, returning only responses from the cache.
Some tools do this, but what I need is a bit more complicated.
Say I request a report from Amazon MWS:
I'll call RequestReport
I'll get a ReportRequestId back
I'll start server polling GetReportRequestList to find out what the current status of the report request is. The report request will go likely go through the statuses SUBMITTED then DONE, but it could also be set to ERROR or CANCELLED
When the report request status returned by GetReportRequestList is DONE, I can finally call GetReport and get the data.
The behavior from step 3 is what I'm trying to replicate.
This external API cache should be able to produce different results for the same request: the first response should yield SUBMITTED and then the second response should yield DONE.
I should be able to easily configure these flows as I wish, setting the responses I want for the 1st, 2nd, nth request.
I would like this tool to necessitate minimal configuration, I do not want to configure routes or anything, I want it to automatically cache everything and then return everything from the cache, never flushing it.
Also, I need this level of control over what's returned in a response, depending on the count of requests done up to that point.
I'm trying to copy data from a database and place it in S3 using nifi. I'm able to copy the data from database and place it in S3. Now I'm trying to add error handling for this flow. I just added the PutEmail processor for error notification. I just gave a wrong bucket name to validate the Email. This PutEmail processor is getting triggered for each and every flow file(As there are 100 flow files mail is triggering 100 times). I just want to trigger this PutEmail(notification) only once whenever there is a error in the flow. Any suggestions on this please.
Below is the Flow:
Any suggestions on better(Generic) error handling will be helpful for me.
For your use case, MergeContent would allow you to batch several FlowFiles over a given duration to be rolled up into a singular email.
You could additionally do some additional transforms to only get the key parts of the content and/or attributes to provide source FlowFiles to MergeContent that would give a summary listing in the message sent.
You can implement custom ReportingTasks which will periodically sends reports based on Need
I am new to Laravel and Api development, i am facing a problem, the workflow of my api is, a user sends post data to api, then api takes that data and processes the data to databases, now there is a process in which php waits for 30 min. while inserting data into two different tables.
The problem is as far as i know after that process is complete then only i can send json response back to user. but this way user has to wait for 30 minute.
Is there a way that process that takes 30 min do work in background and send the response json immediately when that process started ?
1) I studied about queues but the web server i will be hosting will not give me access to server as a whole to install something, it will only give me space for my files.
I am confused how to achieve this functionality, so that user do not have to wait much for Response.
I will really appreciate.
Thanks,
You can use the queue without any server installation. All your configuration goes in the config/queue.php file.
You can use
Amazon SQS: aws/aws-sdk-php ~3.0
Beanstalkd: pda/pheanstalk ~3.0
Redis: predis/predis ~1.0
Read more here: https://laravel.com/docs/5.2/queues#introduction
Can anyone give me examples of how in production a correlation id can be used?
I have read it is used in request/response type messages but I don't understand where I would use it?
One example (which maybe wrong) I can think off is in a publish subscribe scenario where I could have 5 subscribers and if I get 5 replies with the same correlation id then I could say all my subscribers have received it. Not sure if this would the be correct usage of it.
Or if I send a simple message, the I can use the correlation to guarantee that the client received it.
Any other examples?
A web application that is providing HTTP API for outsiders for performing a processing task and you want to give the results for the caller as a response to the HTTP request they made.
A request comes in, message describing the task is pushed to queue by the frontend server. After that the frontend server blocks to wait for response message with the same correlation id. A pool of worker machines are listening on queue and one of them picks up the task, performs it and returns the result as message. Once a message with right correlation id comes in, frontend server continues to return the response to the caller.
In the context of CQRS and EventSourcing a command message correlation id will most likely get stored togehter with the corresponding events from the domain. This information can later be used to form an audit trail.
Streaming engines like Apache Flink use correlation ids, much like you said, to guarantee exactness of processing.