In my application i am making a HTTP call to read data from external system. I wanted to make repeated trails until 3 times with an interval until i get the success response. I am using Mule's component to do this. below is the code.
<until-successful maxRetries="3" millisBetweenRetries="10000">
<http:request method="GET"></http:request>
</until-successful>
This code is making 3 trails with an interval of 10 seconds.
However, i wanted to increase wait time for each iteration.
i.e. I want component to wait for 10s on first iteration, 20s on second iteration and 30s on third iteration.
Is there any option to do this with component.
Please suggest. Thanks.
I don't think there is a way to do this out of the box. The wait time can be an expression but I don't think you can change the values in each iteration.
You can make this happen by nesting <until-successful> scopes with different delays. Not the most elegant but at least it uses out of the box components!
<flow name="pavanFlow">
<http:listener doc:name="go" config-ref="HTTP_Listener_config" path="go"/>
<until-successful maxRetries="1" doc:name="Third Retry" millisBetweenRetries="30000">
<until-successful maxRetries="1" doc:name="Second Retry" millisBetweenRetries="20000">
<until-successful maxRetries="1" doc:name="First Retry" millisBetweenRetries="10000">
<flow-ref doc:name="attemptFlow" name="attemptFlow" />
</until-successful>
</until-successful>
</until-successful>
</flow>
<flow name="attemptFlow">
<logger level="INFO" doc:name="Trying" message="Trying"/>
<raise-error doc:name="SOMEFAILURE" type="X:SOMEFAILURE"/>
</flow>
Related
I have a list of objects, which right now I am processing in foreach. The list is nothing but a string of ids that kicks off other stuff internally.
<flow name="flow1" processingStrategy="synchronous">
<quartz:inbound-endpoint jobName="integration" repeatInterval="86400000" responseTimeout="10000" doc:name="Quartz" >
<quartz:event-generator-job/>
</quartz:inbound-endpoint>
<component class="RequestFeeder" doc:name="RequestFeeder"/>
<foreach collection="#[payload]" doc:name="For Each">
<flow-ref name="createFlow" doc:name="createFlow"/>
<flow-ref name="queueFlow" doc:name="queueFlow"/>
<flow-ref name="statusCheckFlow" doc:name="statusCheckFlow"/>
<flow-ref name="resultsFlow" doc:name="resultsFlow"/>
<flow-ref name="sftpFlow" doc:name="sftpFlow"/>
<logger message="RequestType #[flowVars['rqstType']] complete" level="INFO" doc:name="Done"/>
</foreach>
<logger message="ALL 15 REQUESTS HAVE BEEN PROCESSED" level="INFO" doc:name="Logger"/>
</flow>
I want to process them in parallel. ie execute the same 4 flow-refs in parallel for all 15 requests coming in the list. This seems simple, but I havent been able to figure it out yet. Any help appreciated.
An alternative to the scatter-gather approach is to simply split the collection and use a VM queue for the items in the list. This method can be simpler if you don't need to wait and collect all 15 results, and will still work if you do.
Try something like this. Mule automatically uses a thread pool (more info) to run your flow, so the requestProcessor flow below will process your requests in parallel.
<flow name="scheduleRequests">
<quartz:inbound-endpoint jobName="integration" repeatInterval="86400000" responseTimeout="10000" doc:name="Quartz" >
<quartz:event-generator-job/>
</quartz:inbound-endpoint>
<component class="RequestFeeder" doc:name="RequestFeeder"/>
<collection-splitter />
<vm:outbound-endpoint path="requests" />
</flow>
<flow name="requestProcessor">
<vm:inbound-endpoint path="requests" />
<flow-ref name="createFlow" doc:name="createFlow"/>
<flow-ref name="queueFlow" doc:name="queueFlow"/>
<flow-ref name="statusCheckFlow" doc:name="statusCheckFlow"/>
<flow-ref name="resultsFlow" doc:name="resultsFlow"/>
<flow-ref name="sftpFlow" doc:name="sftpFlow"/>
</flow>
I reckon you still want those four flows to run sequentially, right?
If that were not the case you could always change the threading profile.
Another thing you could do is to wrap the four flows in an async scope although you may need a processor change.
In any event I think you'll be better of using the scatter gather component:
https://developer.mulesoft.com/docs/display/current/Scatter-Gather
https://www.mulesoft.com/exchange#!/scatter-gather-flow-control
Which without needing the for each scope will split the list and execute each item in a different thread. You could define how many threads you want to run in parallel (so you don't just spin of a new thread you use a pool).
One final note though, is meant to aggregate the result of all the processed items. I reckon you could change that with a custom aggregation strategy but not sure really, please take a look at the docs for that.
HTH
You say 4 flows, but the list contains 5 flows. If you want all flows executed in sequence, but each item in the collection executed in parallel, you will want a splitter followed by a separate vm flow containing all (4/5) flows, as explained here: https://support.mulesoft.com/s/article/Concurrently-processing-Collection-and-getting-the-results.
If you want the flows inside the loop to execute in parallel then you choose a Scatter-Gather component.
It is important to be clear which of the two things you are wanting to achieve as the solution would be very different. So the basic difference is, in Scatter-Gather a single message is sent to multiple recipients for processing in parallel, but in Splitter-Aggregator a single message is split into multiple sub messages and processed individually and then aggregated. See: http://muthurajud.blogspot.com/2016/07/eai-patterns-scattergather-versus.html
Scatter- gather of Mule component is one of the component to make easy for parallel processing, A simple example will be following :-
<scatter-gather >
<flow-ref name="flow1" />
<flow-ref name="flow2" />
<flow-ref name="flow3" />
</scatter-gather>
So, the flows you want to execute in parallel can be kept inside the
I am trying to do a test on how to limit the number of concurrent incoming HTTP requests.
So I tried to simulate the following scenario.
I have created a simple flow with
1. Http Listener as Msg source
2. Using Groovy Script, Sleeping for 15 seconds to introduce delay
3. Set the Payload with Hello world.
So any single request will have Min 15 sec of response time. Then in order to limit number of active threads (i.e assuming to control concurrent requests / processing threads) , I have set the maxActiveThreads to 1. So ideally I would allow 1 concurrent thread to process the flow.
Now when I fire using apache benchmarking, as simple get with 1 request, I receive with response time as 15 seconds, which is fine. Now when I increase the number of concurrent requests to 2, I still receive the response time as 15 seconds . I am expecting it be 30 seconds
I see the behaviour until 9 concurrent requests. Beyond 9, then from 10th request onwards it is placed in the waiting queue.
So can an expert please explain how Can I limit the number of active threads to 1. And how is the number of concurrent requests set to 9 (I see in threads using JConsole there are 9 SelectorRunner threads, which I assume there are linked to it).
Below is the simple flow.
<http:listener-config name="HTTP_Listener_Configuration" host="localhost" port="8082" doc:name="HTTP Listener Configuration">
<http:worker-threading-profile maxThreadsActive="1" />
</http:listener-config>
<flow name="getting-sartedFlow">
<http:listener config-ref="HTTP_Listener_Configuration" path="/" doc:name="HTTP" />
<scripting:component doc:name="Groovy">
<scripting:script engine="Groovy">
<![CDATA[Thread.currentThread().sleep((long)(15000));]]>
</scripting:script>
</scripting:component>
<set-payload value="Hello World" doc:name="Set Payload" />
</flow>
As answered in the forum, you need to define a poolExhaustedAction for the worker-threading-profile. If you don't then the default one will be used which is RUN, which explains the behavior you're seeing. From what I understand you should use WAIT.
<http:worker-threading-profile maxThreadsActive="1" poolExhaustedAction="WAIT"/>
I'm facing issues in Process Record Phase of Batch, Kindly suggest- I'm trying to load the some KB file ( which has about 5000 record). For the success scenario it works.
If suppose error happened in input phase for the first hit and the flows stops, when the second time when it try to hit the same record. Mule stops executing in Process Record step.It is not running After loading Phase. Please find the run time logs below
11:55:33 INFO info.org.mule.module.logging.DispatchingLogger - Starting loading phase for instance 'ae67601a-5fbe-11e4-bc4d-f0def1ed6871' of job 'test'
11:55:33 INFO info.org.mule.module.logging.DispatchingLogger - Finished loading phase for instance ae67601a-5fbe-11e4-bc4d-f0def1ed6871 of job order. 5000 records were loaded
11:55:33 INFO info.org.mule.module.logging.DispatchingLogger - **Started execution of instance 'ae67601a-5fbe-11e4-bc4d-f0def1ed6871' of job 'test**
It stopped processing after instance starts- I'm not sure what is happening here.
When i stop the flow and delete the .mule folder from the workspace. It then works.
I hope in loading phase mule using temporary queue it is not being deleted automatically when exception happens in input phase, but not sure this could be the real cause.
I cant go and delete each time the .muleFolder in a real time.
Could you please anyone suggest what makes the strange behavior here. How to i get rid of this issue. Please find config xml
<batch:job name="test">
<batch:threading-profile poolExhaustedAction="WAIT"/>
<batch:input>
<component class="com.ReadFile" doc:name="File Reader"/>
<mulexml:jaxb-xml-to-object-transformer returnClass="com.dto" jaxbContext-ref="JAXB_Context" doc:name="XML to JAXB Object"/>
<component class="com.Transformer" doc:name="Java"/>
</batch:input>
<batch:process-records>
<batch:step name="Batch_Step" accept-policy="ALL">
<batch:commit doc:name="Batch Commit" streaming="true">
<logger message="************after Data mapper" level="INFO" doc:name="Logger"/>
<data-mapper:transform config-ref="Orders_Pojo_To_XML" stream="true" doc:name="Transform_CanonicalToHybris"/>
<file:outbound-endpoint responseTimeout="10000" doc:name="File" path="#[sessionVars.uploadFilepath]"">
</file:outbound-endpoint>
</batch:commit>
</batch:step>
</batch:process-records>
<batch:on-complete>
<set-payload value="BatchJobInstanceId:#[payload.batchJobInstanceId+'\n'], Number of TotalRecords: #[payload.totalRecords+'\n'], Number of loadedRecord: #[payload.loadedRecords+'\n'], ProcessedRecords: #[payload.processedRecords+'\n'], Number of sucessfull Records: #[payload.successfulRecords+'\n'], Number of failed Records: #[payload.failedRecords+'\n'], ElapsedTime: #[payload.elapsedTimeInMillis+'\n'], InpuPhaseException #[payload.inputPhaseException+'\n'], LoadingPhaseException: #[payload.loadingPhaseException+'\n'], CompletePhaseException: #[payload.onCompletePhaseException+'\n'] " doc:name="Set Batch Result"/>
<logger message="afterSetPayload: #[payload]" level="INFO" doc:name="Logger"/>
<flow-ref name="log" doc:name="Logger" />
</batch:on-complete>
I'm in struck with this behavior quite a long days. Your help will be much appreciated.
Version:3.5.1
Thanks in advance.
Set max-failed-records to -1 so that batch job will continue even though there an exception
<batch:job name="test" max-failed-records="-1">
in the real time environment you don't have the situation to clean .mule folder
this happens only when you are working with Anypoint Studio
Below is a part of my mule flow
<until-successful objectStore-ref="ObjStreuntil" maxRetries="60" secondsBetweenRetries="60" doc:name="Until Successful" failureExpression="# [payload.state == 'Queued' || payload.state == 'InProgress']">
<processor-chain doc:name="Processor Chain">
<sfdc:batch-info config-ref="Salesforce" doc:name="Salesforce">
<sfdc:batch-info ref="#[payload]"/>
</sfdc:batch-info>
<logger message="#[payload]" level="INFO" doc:name="Logger"/>
</processor-chain>
</until-successful>
I would like my flow to wait until my batch is completed and then proceed to the next processor. I believed using processing chain will get the outcome.
But the flow doesn't work. I'm aware that until sucessfull is made synchronos in 3.5 is there any method to acheive this on 3.4.0
Any suggestions would be of great help
Thank you in advance
To achieve your goal in 3.4, add a flow-ref or vm:outbound-endpoint after the batch call so the subsequent logic can be executed when the batch is done.
This is preferable to blocking the main flow thread anyways, since batch processing can take a while.
Note that you may need to add a filter after sfdc:batch-info if you want to process the subsequent logic only for certain return codes.
I have a flow something like this
A 'Database inbound endpoint' which polls(for every 5 mins) to mySQL Database-Server and get result-set by a select-query (automatically this becomes the current payload i.e #[message.payload])
'For each' component and a 'Logger' component in it using a expression as #[message.payload]
Now flow has one more 'Database-out-bound-endpoint' component which executes another select-query and obtains result-set.
'For each' component with a 'Logger' component in it using a expression as #[message.payload]
Note: in the loggers result-set of first DB is printing. I mean second logger is also showing result-set of first query itself.Because the result-set is storing as payload
so, my questions are
what is the MEL to read the result-set of second database-query in the above scenario.
is there any another way to read result-set in the flow
Here is the configuration XML
<jdbc-ee:connector name="oracle_database" dataSource-ref="Oracle_Data_Source" validateConnections="true" queryTimeout="-1" pollingFrequency="0" doc:name="Database"/>
<flow name="testFileSaveFlow3" doc:name="testFileSaveFlow3">
<poll frequency="1000" doc:name="Poll">
<jdbc-ee:outbound-endpoint exchange-pattern="one-way" queryKey="selectTable1" queryTimeout="-1" connector-ref="oracle_database" doc:name="get data from table 1">
<jdbc-ee:query key="selectTable1" value="SELECT * FROM TABLE1"/>
</jdbc-ee:outbound-endpoint>
</poll>
<foreach doc:name="For Each">
<logger message="#[message.payload]" level="INFO" doc:name="prints result-set of table1"/>
</foreach>
<jdbc-ee:outbound-endpoint exchange-pattern="one-way" queryKey="selectTable2" queryTimeout="-1" connector-ref="oracle_database" doc:name="get data from table 2">
<jdbc-ee:query key="selectTable2" value="SELECT * FROM TABLE2"/>
</jdbc-ee:outbound-endpoint>
<foreach doc:name="For Each">
<logger message="#[message.payload]" level="INFO" doc:name="prints result-set of table2"/>
</foreach>
</flow>
thanks in advance.
This is not the issue with the MEL. It is the issue with your flow logic.
The second result set is not available in the message.
The JDBC Outbound Endpoint is one-way. So Mule flow will not wait for the reply (result set) from the second JDBC (outbound ) in the middle of the flow. So the second time also it is printing the first result set.
Type 1:
Try making your JBDC outbound request-response instead of one-way.
Type 2:
Try Mule Enricher to call the JDBC outbound to call the DB and store the result set into a varaible and try looping the varaible.
Hope this helps.