I have a mule application that generates individual xml files and places them in a folder on the basis of a query, now I want to create aggregate reports which will consist data from various individual reports. Since services are run randomly, I want to make sure that I delay the generation of the aggregate report so that all the individual files exist before the service for aggregate report is called. Is it possible to set a timer on a service?
Seems like a candidate for aggregation:
http://blogs.mulesoft.org/asynchronous-message-processing-with-mule/
You can use quartz for scheduling execution of flow in mule. User cron expression to customize the schedule for your needs. Here is an example of flow with quartz scheduler -
<flow name="resendFailedMessages">
<description>
"*/15 07-18 * * ?" run every 15 minutes from 7 am to 6 pm every day -->
</description>
<quartz:inbound-endpoint jobName="hostRedeliveryJob" cronExpression="*/15 07-18 * * ?">
<quartz:endpoint-polling-job>
<quartz:job-endpoint ref="redeliverToHost" />
</quartz:endpoint-polling-job>
</quartz:inbound-endpoint>
<set-variable variableName="hostXML" value="#[payload]" />
<logger message="QUARTZ found message for host" level="INFO" />
<flow-ref name="webServiceCall" />
<flow-ref name="inspectWSResponse" />
<exception-strategy ref="retryExceptionStrategy" />
Also check this out - https://github.com/ddossot/mule-in-action-2e/blob/master/chapter07/src/main/app/quartz-config.xml
Related
I am configuring transaction and response timeout for API in Mule 4, Is there anyway that I set different timeout for different methods(GET,POST,DELETE) for single API in Mule Soft because the API has different SLA for different operations ?
As the http timeout is set at the connector level that does not allow you to set timeouts per method.
One way you could try to achieve this is via separating your interface flow from your logic. Then have your interface flow call your logic flow via something like vm where you can set a timeout individually. You can then catch the timeout error and do what you want with it.
Here is an example that has a flow for a POST method. All this flow does is offload the logic to another floe and invokes that logic using v:publish-consume and awaits the response. It sets a timeout of 2 seconds(configurable with properties etc.) and catches VM:QUEUE-TIMEOUT errors and sets an 'SLA exceeded'error message:
<flow name="myPOSTInterface">
<vm:publish-consume queueName="postQueue" config-ref="vm" timeout="2" timeoutUnit="SECONDS" />
<logger level="INFO" message="Result from logic flow: #[payload]" />
<error-handler>
<on-error-continue type="VM:QUEUE_TIMEOUT">
<set-payload value="#[{error: 'SLA exceeded'}]" />
</on-error-continue>
</error-handler>
</flow>
<flow name="myPOSTLogic">
<vm:listener config-ref="vm" queueName="postQueue" />
<set-payload value="#[{result: 'Result from my logic'}]" />
</flow>
I have a requirement in which the start of the flow is triggered to load customer data from a SQL database into a Mongo DB database. All the current trades need to be moved to the Mongo DB. Once the migration of current trades to MongoDB is complete, it is intended that new trades would start to be monitored and copied to MongoDB. The goal is to trigger the flow that monitors new customer trades to start after all the current customer trades has been moved to MongoDB. The problem is that moving the current customer trades is to be triggered from a property in a properties file.
<flow name="migrateCurrentTrades">
<quartz:inbound-endpoint responseTimeout="10000" doc:name="Task"
cronExpression="0 */2 * * * ?" jobName="mailJob"
repeatInterval="1000000"
repeatCount="0">
<quartz:event-generator-job/>
</quartz:inbound-endpoint>
<flow-ref name="readAndSave">
<!-- This actually is a reference to a groovy script that starts the
monitorCustomerActivity flow below in the 'stopped'
initialState
-->
<script-ref name="startMonitorCustomerActivityFlow"/>
</flow>
<!-- long running job----->
<sub-flow name="readAndSave">
<db:select config-ref="mySQLConfig">
<db:parameterized-query><![CDATA[
SELECT * FROM CUSTOMER c WHERE status='ACTIVE'
]]>
</db:parameterized-query>
</db:select>
<custom-transformer class="com.gdc.CustomerTransformer" />
<!-- Save to Mongo DB -- via queue -->
<vm:outbound-endpoint path="mongodb-queue" exchange-pattern="one-way"/>
</sub-flow>
<flow name="monitorCustomerActivity" initialState="stopped">
<quartz:inbound-endpoint responseTimeout="10000" doc:name="Task"
cronExpression="0 */45 * * * ?" jobName="mailJob" repeatInterval="0"
repeatCount="0">
<quartz:event-generator-job/>
</quartz:inbound-endpoint>
<db:parameterized-query><![CDATA[
SELECT * FROM CUSTOMER_TRADES c WHERE trade_status='NEW'
]]>
</db:parameterized-query>
</db:select>
<custom-transformer class="com.gdc.TradesTransformer" />
<!-- Save to Mongo DB -- via queue -->
<vm:outbound-endpoint path="mongodb-queue" exchange-pattern="one-way"/>
</flow>
Unfortunately, the readAndSave flow keeps running repeatedly because it takes too long to complete. I have set the repeatInterval to a very high value and repeatCount to 0. I want the readAndSave flow to only be triggered once and for it to complete. Although it starts the second flow, monitorCustomerActivity, it starts to interleave with it causing errors. How can I solve this problem, making sure the readAndSave flow be called only once and complete saving the current customer trades before calling the second flow monitorCustomerActivity? I have battled with the problem for days
Try out the below steps, Increased timeout, repeatCount = 0 + startdelay ( on stop and start your anypoint/server it will wait for 20 secs and then the schedule starts).
Since it is Async, you can use flow instead of sub-flow.
<quartz:inbound-endpoint responseTimeout="60000" doc:name="Task"
jobName="mailJob" repeatInterval="1000000" repeatCount="0" startDelay="20000"> <quartz:event-generator-job/>
</quartz:inbound-endpoint>
etc ....
make readAndsave subflow to `flow`
<flow name="readAndSave">
<db:select config-ref="mySQLConfig" doc:name="Database">
<db:parameterized-query><![CDATA[
SELECT * FROM CUSTOMER c WHERE status='ACTIVE'
]]>
</db:parameterized-query>
</db:select>
<logger message="readAndSave:Check how many times flow gets trigger by looking at this log entry" />
etc...
I have a list of objects, which right now I am processing in foreach. The list is nothing but a string of ids that kicks off other stuff internally.
<flow name="flow1" processingStrategy="synchronous">
<quartz:inbound-endpoint jobName="integration" repeatInterval="86400000" responseTimeout="10000" doc:name="Quartz" >
<quartz:event-generator-job/>
</quartz:inbound-endpoint>
<component class="RequestFeeder" doc:name="RequestFeeder"/>
<foreach collection="#[payload]" doc:name="For Each">
<flow-ref name="createFlow" doc:name="createFlow"/>
<flow-ref name="queueFlow" doc:name="queueFlow"/>
<flow-ref name="statusCheckFlow" doc:name="statusCheckFlow"/>
<flow-ref name="resultsFlow" doc:name="resultsFlow"/>
<flow-ref name="sftpFlow" doc:name="sftpFlow"/>
<logger message="RequestType #[flowVars['rqstType']] complete" level="INFO" doc:name="Done"/>
</foreach>
<logger message="ALL 15 REQUESTS HAVE BEEN PROCESSED" level="INFO" doc:name="Logger"/>
</flow>
I want to process them in parallel. ie execute the same 4 flow-refs in parallel for all 15 requests coming in the list. This seems simple, but I havent been able to figure it out yet. Any help appreciated.
An alternative to the scatter-gather approach is to simply split the collection and use a VM queue for the items in the list. This method can be simpler if you don't need to wait and collect all 15 results, and will still work if you do.
Try something like this. Mule automatically uses a thread pool (more info) to run your flow, so the requestProcessor flow below will process your requests in parallel.
<flow name="scheduleRequests">
<quartz:inbound-endpoint jobName="integration" repeatInterval="86400000" responseTimeout="10000" doc:name="Quartz" >
<quartz:event-generator-job/>
</quartz:inbound-endpoint>
<component class="RequestFeeder" doc:name="RequestFeeder"/>
<collection-splitter />
<vm:outbound-endpoint path="requests" />
</flow>
<flow name="requestProcessor">
<vm:inbound-endpoint path="requests" />
<flow-ref name="createFlow" doc:name="createFlow"/>
<flow-ref name="queueFlow" doc:name="queueFlow"/>
<flow-ref name="statusCheckFlow" doc:name="statusCheckFlow"/>
<flow-ref name="resultsFlow" doc:name="resultsFlow"/>
<flow-ref name="sftpFlow" doc:name="sftpFlow"/>
</flow>
I reckon you still want those four flows to run sequentially, right?
If that were not the case you could always change the threading profile.
Another thing you could do is to wrap the four flows in an async scope although you may need a processor change.
In any event I think you'll be better of using the scatter gather component:
https://developer.mulesoft.com/docs/display/current/Scatter-Gather
https://www.mulesoft.com/exchange#!/scatter-gather-flow-control
Which without needing the for each scope will split the list and execute each item in a different thread. You could define how many threads you want to run in parallel (so you don't just spin of a new thread you use a pool).
One final note though, is meant to aggregate the result of all the processed items. I reckon you could change that with a custom aggregation strategy but not sure really, please take a look at the docs for that.
HTH
You say 4 flows, but the list contains 5 flows. If you want all flows executed in sequence, but each item in the collection executed in parallel, you will want a splitter followed by a separate vm flow containing all (4/5) flows, as explained here: https://support.mulesoft.com/s/article/Concurrently-processing-Collection-and-getting-the-results.
If you want the flows inside the loop to execute in parallel then you choose a Scatter-Gather component.
It is important to be clear which of the two things you are wanting to achieve as the solution would be very different. So the basic difference is, in Scatter-Gather a single message is sent to multiple recipients for processing in parallel, but in Splitter-Aggregator a single message is split into multiple sub messages and processed individually and then aggregated. See: http://muthurajud.blogspot.com/2016/07/eai-patterns-scattergather-versus.html
Scatter- gather of Mule component is one of the component to make easy for parallel processing, A simple example will be following :-
<scatter-gather >
<flow-ref name="flow1" />
<flow-ref name="flow2" />
<flow-ref name="flow3" />
</scatter-gather>
So, the flows you want to execute in parallel can be kept inside the
I have 2 flows, A.flow and B.flow, eventually both flows execute the same java class.
A & B read from a separate Queue.
I want to synchronize the flows so that if both flows get input simultaneously then one flow at a time process and after it finishes, the other flow will start processing.
Any ideas?
thanks
Use a pooled component and configure it to use one thread at a time:
<flow name="A">
<jms:inbound-endpoint...>
...
<vm:outbound-endpoint path="process"/>
...
</flow>
<flow name="B">
<jms:inbound-endpoint...>
...
<vm:outbound-endpoint path="process"/>
...
</flow>
<flow name="process">
<vm:inbound-endpoint path="process"/>
<pooled-component class="org.my.PrototypeObject">
<pooling-profile exhaustedAction="WHEN_EXHAUSTED_WAIT" initialisationPolicy="INITIALISE_ALL" maxActive="1" maxIdle="1" maxWait="1000" /> </pooled-component>
</pooled-component>
</flow>
Source: http://www.mulesoft.org/documentation/display/current/Configuring+Java+Components#ConfiguringJavaComponents-ConfiguringaPooledJavaComponent
Is there a way to configure a Quartz inbound endpoint in Mule to have multiple triggers? Say I want an event every day at 9:00, plus one at 1:00 a.m. on the first day of the month.
Here is what you might work for you --
<flow name="MultipleIBEndpoints" doc:name="MultipleIBEndpoints">
<composite-source doc:name="Composite Source">
<quartz:inbound-endpoint jobName="QuartzDaily" doc:name="Quartz Daily"
cronExpression="0 0 9 1/1 * ? *">
<quartz:event-generator-job>
<quartz:payload>dummy</quartz:payload>
</quartz:event-generator-job>
</quartz:inbound-endpoint>
<quartz:inbound-endpoint jobName="QuartzMonthly" doc:name="Quartz Monthly"
cronExpression="0 0 1 1 1/1 ? *">
<quartz:event-generator-job>
<quartz:payload>dummy</quartz:payload>
</quartz:event-generator-job>
</quartz:inbound-endpoint>
</composite-source>
<logger level="INFO" doc:name="Logger" />
</flow>
The above flow uses composite source scope which allows you to embed into a single message source two or more inbound endpoints.
In the case of Composite, the embedded building blocks are actually message sources (i.e. inbound endpoints) that listen in parallel on different channels for incoming messages. Whenever any of these receivers accepts a message, the Composite scope passes it to the first message processor in the flow, thus triggering that flow.
You can do you requirement just by using one quartz endpoint with the required quartz endpoint
CRON Expression 0 0 1,21 1 * *
Please refer to the below link for more tweaks.
Mulesoft quartz reference
wikipedia reference
List of Cron Expression examples
In that case you need to configure two crontrigger and add them to the scheduler.
Please go through the below link where i have described the whole thing.
Configure multiple cron trigger
Hope this will help.