I have a list of objects, which right now I am processing in foreach. The list is nothing but a string of ids that kicks off other stuff internally.
<flow name="flow1" processingStrategy="synchronous">
<quartz:inbound-endpoint jobName="integration" repeatInterval="86400000" responseTimeout="10000" doc:name="Quartz" >
<quartz:event-generator-job/>
</quartz:inbound-endpoint>
<component class="RequestFeeder" doc:name="RequestFeeder"/>
<foreach collection="#[payload]" doc:name="For Each">
<flow-ref name="createFlow" doc:name="createFlow"/>
<flow-ref name="queueFlow" doc:name="queueFlow"/>
<flow-ref name="statusCheckFlow" doc:name="statusCheckFlow"/>
<flow-ref name="resultsFlow" doc:name="resultsFlow"/>
<flow-ref name="sftpFlow" doc:name="sftpFlow"/>
<logger message="RequestType #[flowVars['rqstType']] complete" level="INFO" doc:name="Done"/>
</foreach>
<logger message="ALL 15 REQUESTS HAVE BEEN PROCESSED" level="INFO" doc:name="Logger"/>
</flow>
I want to process them in parallel. ie execute the same 4 flow-refs in parallel for all 15 requests coming in the list. This seems simple, but I havent been able to figure it out yet. Any help appreciated.
An alternative to the scatter-gather approach is to simply split the collection and use a VM queue for the items in the list. This method can be simpler if you don't need to wait and collect all 15 results, and will still work if you do.
Try something like this. Mule automatically uses a thread pool (more info) to run your flow, so the requestProcessor flow below will process your requests in parallel.
<flow name="scheduleRequests">
<quartz:inbound-endpoint jobName="integration" repeatInterval="86400000" responseTimeout="10000" doc:name="Quartz" >
<quartz:event-generator-job/>
</quartz:inbound-endpoint>
<component class="RequestFeeder" doc:name="RequestFeeder"/>
<collection-splitter />
<vm:outbound-endpoint path="requests" />
</flow>
<flow name="requestProcessor">
<vm:inbound-endpoint path="requests" />
<flow-ref name="createFlow" doc:name="createFlow"/>
<flow-ref name="queueFlow" doc:name="queueFlow"/>
<flow-ref name="statusCheckFlow" doc:name="statusCheckFlow"/>
<flow-ref name="resultsFlow" doc:name="resultsFlow"/>
<flow-ref name="sftpFlow" doc:name="sftpFlow"/>
</flow>
I reckon you still want those four flows to run sequentially, right?
If that were not the case you could always change the threading profile.
Another thing you could do is to wrap the four flows in an async scope although you may need a processor change.
In any event I think you'll be better of using the scatter gather component:
https://developer.mulesoft.com/docs/display/current/Scatter-Gather
https://www.mulesoft.com/exchange#!/scatter-gather-flow-control
Which without needing the for each scope will split the list and execute each item in a different thread. You could define how many threads you want to run in parallel (so you don't just spin of a new thread you use a pool).
One final note though, is meant to aggregate the result of all the processed items. I reckon you could change that with a custom aggregation strategy but not sure really, please take a look at the docs for that.
HTH
You say 4 flows, but the list contains 5 flows. If you want all flows executed in sequence, but each item in the collection executed in parallel, you will want a splitter followed by a separate vm flow containing all (4/5) flows, as explained here: https://support.mulesoft.com/s/article/Concurrently-processing-Collection-and-getting-the-results.
If you want the flows inside the loop to execute in parallel then you choose a Scatter-Gather component.
It is important to be clear which of the two things you are wanting to achieve as the solution would be very different. So the basic difference is, in Scatter-Gather a single message is sent to multiple recipients for processing in parallel, but in Splitter-Aggregator a single message is split into multiple sub messages and processed individually and then aggregated. See: http://muthurajud.blogspot.com/2016/07/eai-patterns-scattergather-versus.html
Scatter- gather of Mule component is one of the component to make easy for parallel processing, A simple example will be following :-
<scatter-gather >
<flow-ref name="flow1" />
<flow-ref name="flow2" />
<flow-ref name="flow3" />
</scatter-gather>
So, the flows you want to execute in parallel can be kept inside the
Related
In my application i am making a HTTP call to read data from external system. I wanted to make repeated trails until 3 times with an interval until i get the success response. I am using Mule's component to do this. below is the code.
<until-successful maxRetries="3" millisBetweenRetries="10000">
<http:request method="GET"></http:request>
</until-successful>
This code is making 3 trails with an interval of 10 seconds.
However, i wanted to increase wait time for each iteration.
i.e. I want component to wait for 10s on first iteration, 20s on second iteration and 30s on third iteration.
Is there any option to do this with component.
Please suggest. Thanks.
I don't think there is a way to do this out of the box. The wait time can be an expression but I don't think you can change the values in each iteration.
You can make this happen by nesting <until-successful> scopes with different delays. Not the most elegant but at least it uses out of the box components!
<flow name="pavanFlow">
<http:listener doc:name="go" config-ref="HTTP_Listener_config" path="go"/>
<until-successful maxRetries="1" doc:name="Third Retry" millisBetweenRetries="30000">
<until-successful maxRetries="1" doc:name="Second Retry" millisBetweenRetries="20000">
<until-successful maxRetries="1" doc:name="First Retry" millisBetweenRetries="10000">
<flow-ref doc:name="attemptFlow" name="attemptFlow" />
</until-successful>
</until-successful>
</until-successful>
</flow>
<flow name="attemptFlow">
<logger level="INFO" doc:name="Trying" message="Trying"/>
<raise-error doc:name="SOMEFAILURE" type="X:SOMEFAILURE"/>
</flow>
Below is a part of my mule flow
<until-successful objectStore-ref="ObjStreuntil" maxRetries="60" secondsBetweenRetries="60" doc:name="Until Successful" failureExpression="# [payload.state == 'Queued' || payload.state == 'InProgress']">
<processor-chain doc:name="Processor Chain">
<sfdc:batch-info config-ref="Salesforce" doc:name="Salesforce">
<sfdc:batch-info ref="#[payload]"/>
</sfdc:batch-info>
<logger message="#[payload]" level="INFO" doc:name="Logger"/>
</processor-chain>
</until-successful>
I would like my flow to wait until my batch is completed and then proceed to the next processor. I believed using processing chain will get the outcome.
But the flow doesn't work. I'm aware that until sucessfull is made synchronos in 3.5 is there any method to acheive this on 3.4.0
Any suggestions would be of great help
Thank you in advance
To achieve your goal in 3.4, add a flow-ref or vm:outbound-endpoint after the batch call so the subsequent logic can be executed when the batch is done.
This is preferable to blocking the main flow thread anyways, since batch processing can take a while.
Note that you may need to add a filter after sfdc:batch-info if you want to process the subsequent logic only for certain return codes.
As of now ,when I poll the database with a query,It will fetch me multiple records during a specified duration.Problem is ,these multiple records will be pushed as a single message into the queue.How do I push every record from the set of records as an individual message?
As you have explained the JDBC endpoint is fetching a collection of records and sending them as one single message to the queue. Solution for this is two options.
Using Mule's For-Each message processor. This helps in iterating through the collection object and processes each item as one message.
Using Mule's collection splitter to iterate through the collection of records.
Solution for option 1 looks like as shown in the image below.
The code for this flow loks like this.
<flow name="JDBC-For-Each-JMS-Flow" >
<jdbc-ee:inbound-endpoint queryKey="SelectAll" mimeType="text/javascript" queryTimeout="500000" pollingFrequency="1000" doc:name="Database">
<jdbc-ee:query key="SelectAll" value="select * from users"/>
</jdbc-ee:inbound-endpoint>
<foreach doc:name="For Each" collection="#[payload]" >
<jms:outbound-endpoint doc:name="JMS"/>
</foreach>
<catch-exception-strategy doc:name="Catch Exception Strategy"/>
</flow>
Note: This is a sample flow.
Hope this helps.
I have a list of json objects containing about 200 objects. I want to split that list into smaller lists where each list contains max 20 objects each. I would like to POST each sublist to HTTP based endpoint.
<flow name="send-to-next-step" doc:name="send-to-vm-flow">
<vm:inbound-endpoint exchange-pattern="one-way"
path="send-to-next-step-vm" doc:name="VM" />
<!-- received the JSON List payload with 200 objects-->
<!-- TODO do processing here to split the list into sub-lists and call sub-flow for each sub-list
<flow-ref name="send-to-aggregator-sf" doc:name="Flow Reference" />
</flow>
One possible way is that I write a java component which iterates over the list and after iterating over each 20 objects, call sub-flow. Is there any better way of accomplishing this?
If your payload is a Java Collection, the Mule foreach scope has batching built in: http://www.mulesoft.org/documentation/display/current/Foreach
Example:
<foreach batchSize="20">
<json:object-to-json-transformer/>
<http:outbound-endpoint ... />
</foreach>
You could use the Groovy collate method for the batching, and then foreach or collection-splitter, depending on your needs:
<json:json-to-object-transformer returnClass="java.util.List"/>
<set-payload value="#[groovy:payload.collate(20)]"/>
<foreach>
<json:object-to-json-transformer/>
<http:outbound-endpoint exchange-pattern="request-response" host="0.0.0.0" port="8082" path="xx"/>
</foreach>
<set-payload value="#[groovy:payload.flatten()]"/>
This will send each batch of 20 objects to the http endpoint and then flatten back to the original list.
I have a flow something like this
A 'Database inbound endpoint' which polls(for every 5 mins) to mySQL Database-Server and get result-set by a select-query (automatically this becomes the current payload i.e #[message.payload])
'For each' component and a 'Logger' component in it using a expression as #[message.payload]
Now flow has one more 'Database-out-bound-endpoint' component which executes another select-query and obtains result-set.
'For each' component with a 'Logger' component in it using a expression as #[message.payload]
Note: in the loggers result-set of first DB is printing. I mean second logger is also showing result-set of first query itself.Because the result-set is storing as payload
so, my questions are
what is the MEL to read the result-set of second database-query in the above scenario.
is there any another way to read result-set in the flow
Here is the configuration XML
<jdbc-ee:connector name="oracle_database" dataSource-ref="Oracle_Data_Source" validateConnections="true" queryTimeout="-1" pollingFrequency="0" doc:name="Database"/>
<flow name="testFileSaveFlow3" doc:name="testFileSaveFlow3">
<poll frequency="1000" doc:name="Poll">
<jdbc-ee:outbound-endpoint exchange-pattern="one-way" queryKey="selectTable1" queryTimeout="-1" connector-ref="oracle_database" doc:name="get data from table 1">
<jdbc-ee:query key="selectTable1" value="SELECT * FROM TABLE1"/>
</jdbc-ee:outbound-endpoint>
</poll>
<foreach doc:name="For Each">
<logger message="#[message.payload]" level="INFO" doc:name="prints result-set of table1"/>
</foreach>
<jdbc-ee:outbound-endpoint exchange-pattern="one-way" queryKey="selectTable2" queryTimeout="-1" connector-ref="oracle_database" doc:name="get data from table 2">
<jdbc-ee:query key="selectTable2" value="SELECT * FROM TABLE2"/>
</jdbc-ee:outbound-endpoint>
<foreach doc:name="For Each">
<logger message="#[message.payload]" level="INFO" doc:name="prints result-set of table2"/>
</foreach>
</flow>
thanks in advance.
This is not the issue with the MEL. It is the issue with your flow logic.
The second result set is not available in the message.
The JDBC Outbound Endpoint is one-way. So Mule flow will not wait for the reply (result set) from the second JDBC (outbound ) in the middle of the flow. So the second time also it is printing the first result set.
Type 1:
Try making your JBDC outbound request-response instead of one-way.
Type 2:
Try Mule Enricher to call the JDBC outbound to call the DB and store the result set into a varaible and try looping the varaible.
Hope this helps.