I have a mule flow(mock-flow) that currently makes http calls to a couple of microservices. If one of the service calls fails due to a connection exception, i have a rollback exception strategy configured to reprocess the message (sending to kafka which in turn invokes the mock flow)but the retry seems to be happening indefinitely inspite of specifying a maxRedeliveryAttempts attribute. How do I limit the no of retries? Any help would be greatly appreciated
<flow name="mock-flow">
<logger level="INFO" message="CRD::: Calling Selldown settle Micro Service***"/>
<logger message="CRD::: Response received : #[message.payload]" level="INFO" />
<http:request config-ref="tp-ins-selldown-msConfig" path="/settle"
method="POST" doc:name="HTTP">
<http:success-status-code-validator
values="200,201" />
</http:request>
<http:request parseResponse="false" config-ref="tp-ins-limits-msConfig" path="booksdlimit"
method="POST" doc:name="HTTP">
<http:success-status-code-validator values="200,201"/>
</http:request>
<choice-exception-strategy doc:name="Choice Exception Strategy">
<rollback-exception-strategy when="exception.causedBy(java.net.ConnectException)" maxRedeliveryAttempts="2" doc:name="Rollback Exception Strategy">
<logger message="Will attempt redelivery" level="INFO" doc:name="Logger" />
<vm:outbound-endpoint exchange-pattern="one-way" path="kafka.inpath" doc:name="VM" />
<on-redelivery-attempts-exceeded>
<logger message="redelivery attempt exceeded" level="INFO" doc:name="Logger" />
<logger message="Retry exhausted" level="INFO" doc:name="Logger" />
</on-redelivery-attempts-exceeded>
</rollback-exception-strategy>
</choice-exception-strategy>
Since the maxReDeliveryAttempts attrtibute wasn't working(for whatever weird reason), I had to do it on my own(manually) by configuring a mule object store with the message request as the key and retry counter as the value.
<rollback-exception-strategy when="exception.causedBy(java.net.ConnectException)" doc:name="Rollback Exception Strategy">
<objectstore:retrieve config-ref="ObjectStore" key="#[uuid]" targetProperty="retryCounter" doc:name="Get value from ObjectStore" />
<logger level="INFO" doc:name="Logger" message="Retry counter --------------> #[retryCounter]"/>
<choice doc:name="Choice">
<when expression="#[retryCounter > 3]">
<logger message="Retry exhausted" level="INFO" doc:name="Logger" />
<objectstore:remove key="#[uuid]" config-ref="ObjectStore" ignoreNotExists="true" doc:name="Remove the key after retry exhaust" />
</when>
<otherwise>
<expression-component>
Thread.sleep(3000);
</expression-component>
<logger message="Will attempt redelivery" level="INFO" doc:name="Logger" />
<transformer ref="ObjectToString"/>
<vm:outbound-endpoint exchange-pattern="one-way" path="kafka.inpath" doc:name="VM" />
<objectstore:remove key="#[uuid]" config-ref="ObjectStore" ignoreNotExists="true" doc:name="Remove if exists" />
<objectstore:store config-ref="ObjectStore" key="#[uuid]" value-ref="#[retryCounter + 1]" doc:name="Store new value" />
</otherwise>
</choice>
</rollback-exception-strategy>
Seems even if maxRedeliveryAttempts variable had worked, it might not have helped my use case since when multiple requests are triggered, there is no way we would ensure that every request was retriggered only 3 times by simply relying on this attribute(unless mule internally uses a hashcode for every req to determine if it has already been retriggered or not)
Also flow variables and session variables didn't work in my case since the retrigger point was a kafka queue (which wasn't manually invoked by flow-ref). Hence had to use an object store
Related
I want to develop a flow that could allow me to make queries to an external system that could take a long time to return. I may have to make queries for multiple values in a list. I am using an until-successful scope in solving the problem. Unfortunately, the even though the request is run several times, the failed records never get put in the dead letter queue. Here is my attempt at solving the problem:
<!-- Dead Letter Queue for exhausted attempts-->
<vm:endpoint name="DLQ" path="DLQ_VM" doc:name="VM"/>
<flow name="StartFlow" processingStrategy="synchronous">
<!--Place a list of String errors to query for on this vm -->
<vm:inbound-endpoint path="request-processing-queue" "
exchange-pattern="one-way" doc:name="VM"/>
<vm:outbound-endpoint path="reprocessing-queue"
exchange-pattern="request-response" doc:name="VM"/>
<logger level="INFO" message="Data returned is #[payload]"/>
<catch-exception-strategy>
<logger level="ERROR" message="Failure During Processing"/>
</catch-exception-strategy>
</flow>
<flow name="RetryingProcess">
<vm:inbound-endpoint name="reprocessing-vm" exchange-
pattern="request-response"
path="reprocessing-queue" doc:name="VM"/>
<foreach collection="#[payload]" doc:name="For Each">
<vm:outbound-endpoint path="by-singles-vm" exchange-
pattern="request-response"/>
</foreach>
</flow>
<flow name="query-retry">
<vm:inbound-endpoint path="by-singles-vm" exchange-
pattern="request-response" doc:name="VM"/>
<until-successful objectStore-ref="objectStore"
failureExpression="#[groovy:(exception &&
exception in com.trion.CustomException)
||!(payload instanceof
com.trion.QueryResult])]"
maxRetries="5"
millisBetweenRetries="300000"
deadLetterQueue-ref="DLQ_VM" doc:name="Until
Successful">
<vm:outbound-endpoint path="try-again-vm" exchange-
pattern="request-response" doc:name="VM"/>
</until-successful>
</flow>
<flow name="GetQueryValue" >
<vm:inbound-endpoint path="try-again-vm" exchange-
pattern="request-response" doc:name="VM"/>
<flow-ref name="QueryRequest" />
</flow>
<!-- This never happens, i.e. the results are not put here... after retying
-->
<flow name="AttemptsExceededProcessing">
<inbound-endpoint ref="DLQ_VM" doc:name="Generic"/>
<logger level="DEBUG" message="Entering Final Destination Queue with
payload is #[payload]"/>
</flow>
<!-- Here I have a query to the external system... >
<flow name="QueryRequest">
...... Makes the long running query Here..
//returns com.trion.QueryResult
</flow>
</mule>
Please help!
There was no problem with the configuration. I had a millisSecondsBetweenRetry value set so small I wasn't seeing the log messages and assumed it wasn't working.
I am using collection-splitter to split my List. Now how should I set the payload to SFTP outbound-endpoint.
<sftp:inbound-endpoint connector-ref="sftp-inbound" host="${SFTP_HOST}" port="${SFTP_PORT}"
path="/files/" user="${SFTP_USER}" password="${SFTP_PASS}"
responseTimeout="10000" pollingFrequency="30000" fileAge="20000" sizeCheckWaitTime="5000"
archiveDir="/files/archive/" doc:name="SFTP" >
<file:filename-regex-filter pattern="Test(.*).zip" caseSensitive="true"/>
</sftp:inbound-endpoint>
<set-variable variableName="regexVal" value="${REGEX}" doc:name="Variable"/>
<set-variable variableName="sourceFileName" value="#[flowVars.originalFilename]" doc:name="Variable"/>
<custom-transformer name="zipTxt" class="com.mst.transform.UnzipTransformer" doc:name="Java" mimeType="image/gif">
<spring:property name="filenamePattern" value="*.csv,*.txt" />
</custom-transformer>
<set-variable variableName="fileContents" value="#[payload]" />
<collection-splitter enableCorrelation="IF_NOT_SET" />
<logger message="#[payload]" level="INFO" doc:name="Logger"/>
<sftp:outbound-endpoint connector-ref="sftp-inbound"
host="${SFTP_HOST}" port="${SFTP_PORT}"
path="/files/" user="${SFTP_USER}" password="${SFTP_PASS}"
responseTimeout="10000" doc:name="SFTP"
exchange-pattern="one-way"/>
</flow>
If your payload before collection splitter is list of objects that can be consumed by SFTP outbound endpoint like InputStream, then after splitter, you can wrap logger, sftp inside processor-chain. Splitter will send each object one-by-one to processor chain. SFTP should be able to write it if its an InputSream.
<collection-splitter enableCorrelation="IF_NOT_SET" />
<processor-chain doc:name="Processor Chain">
<logger message="#[payload]" level="INFO" doc:name="Logger"/>
<sftp:outbound-endpoint connector-ref="sftp-inbound"
host="${SFTP_HOST}" port="${SFTP_PORT}"
path="/files/" user="${SFTP_USER}" password="${SFTP_PASS}"
responseTimeout="10000" doc:name="SFTP"
exchange-pattern="one-way"/>
</processor-chain>
You wouldn't need processor-chain if you just want to put one processor (eg. SFTP) after splitter.
If this doesn't work, then please add error details to question.
I am new to Mule ESB. I have created a simple flow that loops through a list of orders and calls the Magento API to update the order statuses one by one. My problem is that if there are any exception occurs in the foreach scope, the whole process tops. I tried to use the Exception Strategy to capture the exception and it did capture the exception. But how to resume the process? I didn't find much info with google search. Maybe I was doing something wrong with the flow. How do we normally handle this in Mule?
Here is my flow in xml.
<flow name="Update_Magento_Order_Status_AU" doc:name="Update_Magento_Order_Status_AU" initialState="started">
<poll doc:name="Poll">
<fixed-frequency-scheduler frequency="10" timeUnit="MINUTES"/>
<jdbc-ee:outbound-endpoint exchange-pattern="request-response" queryKey="GET_ORDERS_BY_STATUS_QUERY" queryTimeout="-1" connector-ref="DSEDatabase" doc:name="Get Orders By Status"/>
</poll>
<flow-ref name="ProcessOrderStastusUpdate" doc:name="Process Order Status Update"/>
</flow>
<flow name="ProcessOrderStastusUpdate" doc:name="ProcessOrderStastusUpdate">
<foreach collection="#[payload]" doc:name="For Each">
<component doc:name="Set Magento Order Status for Update">
<singleton-object class="com.dse.esb.component.OrderStatusMapperComp">
<property key="as400OrderStatuses" value="${as400.orderstatuses}"/>
<property key="magentoOrderStatuses" value="${magento.orderStatuses}"/>
</singleton-object>
</component>
<logger message="About to update Magento Order Status" level="INFO" doc:name="Logger"/>
<magento:add-order-comment config-ref="Magento" comment="Updated by Mule ESB with AS400 order status: #[payload.TRNSTS]" orderId="#[payload.EPGORDNBR]" status="#[flowVars['magentoOrderStatus']]" doc:name="Update Magento Order Status"/>
</foreach>
<choice-exception-strategy doc:name="Choice Exception Strategy">
<catch-exception-strategy doc:name="default">
<logger message="Handle default exception" level="INFO" category="==============>>>>>>>>>>>>" doc:name="Logger"/>
</catch-exception-strategy>
</choice-exception-strategy>
</flow>
Use a private flow for the content of the for-each with its own exception strategy. THe exception will be handled in the private flow and the parent flow should be able to continue. Something like:
<flow name="ProcessOrderStastusUpdate">
<foreach collection="#[payload]" doc:name="For Each">
<flow-ref name="privateFlow" />
</foreach>
</flow>
<flow name="privateFlow">
<component doc:name="Set Magento Order Status for Update">
<singleton-object class="com.dse.esb.component.OrderStatusMapperComp">
<property key="as400OrderStatuses" value="${as400.orderstatuses}"/>
<property key="magentoOrderStatuses" value="${magento.orderStatuses}"/>
</singleton-object>
</component>
<logger message="About to update Magento Order Status" level="INFO" doc:name="Logger"/>
<magento:add-order-comment config-ref="Magento" comment="Updated by Mule ESB with AS400 order status: #[payload.TRNSTS]" orderId="#[payload.EPGORDNBR]" status="#[flowVars['magentoOrderStatus']]" doc:name="Update Magento Order Status"/>
<choice-exception-strategy doc:name="Choice Exception Strategy">
<catch-exception-strategy doc:name="default">
<logger message="Handle default exception" level="INFO" category="==============>>>>>>>>>>>>" doc:name="Logger"/>
</catch-exception-strategy>
</choice-exception-strategy>
</flow>
Please help me on this logic. I have to call the service and get the response back.If the service is down or something have to retry 3 times and once exhausted need to log in DLQ.I'm using until successful processor.For success scenario i'm getting only null payload in logger.But i tried keeping the HTTP outbound end point without until successfull able to get the response from the service. Please find my xml config.
<flow name="Flow" doc:name="eFlow" tracking:enable-default-events="true">
<wmq:inbound-endpoint queue="InputQ" tracking:enable-default-events="true" connector ref="WMQ_Connector" doc:name="connector">
<wmq:transaction action="NONE"/>
</wmq:inbound-endpoint>
<mulexml:dom-to-xml-transformer></mulexml:dom-to-xml-transformer>
<set-session-variable variableName="originalPayload" value="#[payload]" doc:name="Store_Payload"/>
<choice tracking:enable-default-events="true" doc:name="Choice">
<when expression="#[xpath('fn:local-name(/root/*[2])') == 'Master']">
<data-mapper:transform config-ref="Master_grf" doc:name="Master"/>
</when>
<when expression="#[xpath('fn:local-name(/root/*[2])') == 'Request']">
<data-mapper:transform config-ref="Request_grf" doc:name="Bulk"/>
</when>
<otherwise>
<scripting:component doc:name="Throw_Exception">
<scripting:script engine="Groovy"><![CDATA[throw new IllegalArgumentException ('requests invalid') ]]>
</otherwise>
</choice>
<mulexml:dom-to-xml-transformer></mulexml:dom-to-xml-transformer>
<flow-ref name="Invoke_Service" doc:name="Flow Reference"/>
</flow>
<flow name="Invoke_Service" doc:name="Invoke_Service" tracking:enable-default-events="true">
<cxf:jaxws-client enableMuleSoapHeaders="true" doc:name="SOAP" operation="Request" serviceClass="com.valid.ICase"/>
<until-successful objectStore-ref="objectStore" maxRetries="3" secondsBetweenRetries="2" deadLetterQueue-ref="VM" doc:name="UntilSuccessful_SymboticService">
<http:outbound-endpoint exchange-pattern="request-response" method="POST" doc:name="HTTP" address="http://localhost:1112/symbotic"/>
</until-successful>
<byte-array-to-string-transformer doc:name="Byte Array to String"/>
<logger message="**********success***Payload: #[payload]*****" level="INFO" doc:name="Logger"/>
But incase of failure scenario until success retry 3 times ( works fine). I have tried request reply processor inside until also transactional processor. It is not working. Using Mule version 3.4.kindly suggest me.
Retried the same with request -reply processor as per the suggestion
<flow name="Flow" doc:name="eFlow" tracking:enable-default-events="true">
<wmq:inbound-endpoint queue="InputQ" tracking:enable-default-events="true" connector- ref="WMQ_Connector" doc:name="connector">
<wmq:transaction action="NONE"/>
</wmq:inbound-endpoint>
<mulexml:dom-to-xml-transformer></mulexml:dom-to-xml-transformer>
<set-session-variable variableName="originalPayload" value="#[payload]" doc:name="Store_Payload"/>
<choice tracking:enable-default-events="true" doc:name="Choice">
<when expression="#[xpath('fn:local-name(/root/*[2])') == 'Master']">
<data-mapper:transform config-ref="Master_grf" doc:name="Master"/>
</when>
<when expression="#[xpath('fn:local-name(/root/*[2])') == 'Request']">
<data-mapper:transform config-ref="Request_grf" doc:name="Bulk"/>
</when>
<otherwise>
<scripting:component doc:name="Throw_Exception">
<scripting:script engine="Groovy"><![CDATA[throw new IllegalArgumentException('requests other then Master andRequest') ]]>
</otherwise>
</choice>
<mulexml:dom-to-xml-transformer></mulexml:dom-to-xml-transformer>
<request-reply timeout="10000">
<vm:outbound-endpoint path="request"/>
<vm:inbound-endpoint path="reply"/>
</request-reply>
<byte-array-to-string-transformer doc:name="Byte Array to String"/>
<logger message="**********success***Payload: #[payload]*****" level="INFO" doc:name="Logger"/>
</flow>
<flow name="Invoke_Service" doc:name="Invoke_Service" tracking:enable-default-events="true">
<vm:inbound-endpoint path="request" doc:name="VM"/>
<cxf:jaxws-client enableMuleSoapHeaders="true" doc:name="SOAP" operation="Request" serviceClass="com.valid.ICase"/>
<until-successful objectStore-ref="objectStore" maxRetries="3" secondsBetweenRetries="2" deadLetterQueue-ref="VM" doc:name="UntilSuccessful_Service">
<http:outbound-endpoint exchange-pattern="request-response" method="POST" doc:name="HTTP" address="http://localhost:1112/symbotic"/>
</until-successful>
</flow>
Even then getting the null payload. Kindly suggest is this is the way your were directing me. Please correct me if im wrong.
until-successful is an asynchronous processor. It does not return response from service call.
Success or failure are defined as:
If the child message processor throws an exception, this is a failure.
If the child message processor does not return a message (e.g. is a one-way endpoint), this is a success.
If a 'failure expression' (see below) has been configured, the return message is evaluated against this expression to determine failure or not.
Otherwise:
If the child message processor returns a message that contains an exception payload, this is a failure.
If the child message processor returns a message that does not contain an exception payload, this is a success.
http://www.mulesoft.org/documentation/display/current/Routing+Message+Processors#RoutingMessageProcessors-UntilSuccessful
I am fairly new to Mule ,using 3.3.0, but I am trying what I think should be a fairly stock example.
I have a mule config which will read a csv file and attempt to process the lines and columns of in different flows async. However, we are seeing ConcurrentModificationException when the message is being "handed off" to one of the async flows. I was wondering if anyone else has seen this issue and what they may have done to work around the problem.
java.util.ConcurrentModificationException
at org.apache.commons.collections.map.AbstractHashedMap$HashIterator.nextEntry(AbstractHashedMap.java:1113)
at org.apache.commons.collections.map.AbstractHashedMap$KeySetIterator.next(AbstractHashedMap.java:938)
at org.mule.DefaultMuleEvent.setMessage(DefaultMuleEvent.java:933)
at org.mule.DefaultMuleEvent.(DefaultMuleEvent.java:318)
at org.mule.DefaultMuleEvent.(DefaultMuleEvent.java:290)
at org.mule.DefaultMuleEvent.copy(DefaultMuleEvent.java:948)
<queued-asynchronous-processing-strategy poolExhaustedAction="RUN" name="commonProcessingStrategy" maxQueueSize="1000" doc:name="Queued Asynchronous Processing Strategy"/>
<file:connector name="inboundFileConnector" fileAge="1000" autoDelete="true" pollingFrequency="1000" workDirectory="C:/mule/orca/dataprovider/work"/>
<file:endpoint name="dataProviderInbound" path="C:\mule\orca\dataprovider\inbound" moveToPattern="#[function:datestamp]-#[header:originalFilename]" moveToDirectory="C:\mule\orca\dataprovider\history" connector-ref="inboundFileConnector" doc:name="Data Feed File" doc:description="new files are processed in 'work' folder, then moved to 'archive' folder"/>
<flow name="dataProviderFeedFlow">
<inbound-endpoint ref="dataProviderInbound"/>
<file:file-to-string-transformer />
<flow-ref name="dataSub"/>
</flow>
<sub-flow name="dataSub" >
<splitter expression="#[rows=org.mule.util.StringUtils.split(message.payload, '\n\r')]" />
<expression-transformer expression="#[org.mule.util.StringUtils.split(message.payload, ',')]" />
<foreach>
<flow-ref name="storageFlow" />
<flow-ref name="id" />
</foreach>
</sub-flow>
<flow name="storageFlow" processingStrategy="commonProcessingStrategy">
<logger level="INFO" message="calling the 'storageFlow' sub flow."/>
</flow>
<flow name="id" processingStrategy="commonProcessingStrategy">
<logger level="INFO" message="calling the 'id' sub flow."/>
</flow>
Here is a fixed version of the dataSub sub-flow that works fine:
<sub-flow name="dataSub">
<splitter expression="#[org.mule.util.StringUtils.split(message.payload, '\n\r')]" />
<splitter expression="#[org.mule.util.StringUtils.split(message.payload, ',')]" />
<flow-ref name="storageFlow" />
<all>
<async>
<flow-ref name="storageFlow" />
</async>
<async>
<flow-ref name="id" />
</async>
</all>
</sub-flow>
Notice that:
I use two splitter expressions,
I use an all message processor to ensure the same payload is sent to both private flows,
I have to wrap the flow-refs with an async message processor otherwise the invocation fails because the private flows are asynchronous but all forces synchronous.