Parallel Processing in Mule : Issue with getting right response - mule

Requirement is to develop mule flow which calls 3 different sync services in parallel and then aggregate the response of each of these and send it back to caller.
I have followed fork join approach as mentioned in docs and How to make parallel outbound calls.
My config file looks like below :
<flow name="fork">
<http:inbound-endpoint host="localhost" port="8090" path="mainPath" exchange-pattern="request-response">
<set-property propertyName="MULE_CORRELATION_GROUP_SIZE"
value="2" />
<all enableCorrelation="IF_NOT_SET">
<async>
<set-property propertyName="MULE_CORRELATION_SEQUENCE"
value="1" />
<flow-ref name="parallel1" />
</async>
<async>
<set-property propertyName="MULE_CORRELATION_SEQUENCE"
value="2" />
<flow-ref name="parallel2" />
</async>
</all>
</flow>
<sub-flow name="parallel1">
<logger level="INFO" message="parallel1: processing started" />
<!- Transformation payload -->
<http:outbound-endpoint address="..."
exchange-pattern="request-response" />
<logger level="INFO" message="parallel1: processing finished" />
<flow-ref name="join" />
</sub-flow>
<sub-flow name="parallel2">
<logger level="INFO" message="parallel2: processing started" />
<!- Transformation payload -->
<http:outbound-endpoint address="..."
exchange-pattern="request-response" />
<logger level="INFO" message="parallel2: processing finished" />
<flow-ref name="join" />
</sub-flow>
<sub-flow name="join">
<collection-aggregator timeout="6000"
failOnTimeout="true" />
<combine-collections-transformer />
<logger level="INFO" message="Continuing processing of: #[message.payloadAs(java.lang.String)]" />
<set-payload value="Soap XML Response"/>
</sub-flow>
I am able to verify that till "join" subflow is working fine but the response is not coming back as "Soap XML Response".
The response is the same initial SOAP Request.
How can I make this thread wait till sub-flow processing is complete and it sends back response whatever "join" sub-flow returns ??

The fork join in the above post looks good. The issue here is there is no way to capture the payload after the join and bring it back to the main flow.
As the calls to parallel made async the main flow continues without waiting for the join output.
I have modified the flow to address this issue. Now the flow will have a processor to wait for the reply and read the joined output to be written onto the http transformer.
<flow name="fork">
<http:inbound-endpoint host="localhost" port="8090" path="mainPath" exchange-pattern="request-response">
<!-- To get back the response after the fork-join -->
<request-reply timeout="60000">
<jms:outbound-endpoint queue="parallel.processor.queue">
<message-properties-transformer scope="outbound">
<delete-message-property key="MULE_REPLYTO" />
</message-properties-transformer>
</jms:outbound-endpoint>
<jms:inbound-endpoint queue="join.queue" >
</jms:inbound-endpoint>
</request-reply>
</flow>
<flow name="fork_join_flow" >
<jms:inbound-endpoint queue="parallel.processor.queue" exchange-pattern="one-way" />
<set-property propertyName="MULE_CORRELATION_GROUP_SIZE"
value="2" />
<all enableCorrelation="IF_NOT_SET">
<async>
<set-property propertyName="MULE_CORRELATION_SEQUENCE"
value="1" />
<flow-ref name="parallel1" />
</async>
<async>
<set-property propertyName="MULE_CORRELATION_SEQUENCE"
value="2" />
<flow-ref name="parallel2" />
</async>
</all>
</flow>
<sub-flow name="parallel1">
<logger level="INFO" message="parallel1: processing started" />
<!- Transformation payload -->
<http:outbound-endpoint address="..."
exchange-pattern="request-response" />
<logger level="INFO" message="parallel1: processing finished" />
<flow-ref name="join" />
</sub-flow>
<sub-flow name="parallel2">
<logger level="INFO" message="parallel2: processing started" />
<!- Transformation payload -->
<http:outbound-endpoint address="..."
exchange-pattern="request-response" />
<logger level="INFO" message="parallel2: processing finished" />
<flow-ref name="join" />
</sub-flow>
<sub-flow name="join">
<collection-aggregator timeout="6000"
failOnTimeout="true" />
<combine-collections-transformer />
<logger level="INFO" message="Continuing processing of: #[message.payloadAs(java.lang.String)]" />
<set-payload value="Soap XML Response"/>
<jms:outbound-endpoint queue="join.queue">
</jms:outbound-endpoint>
</sub-flow>
Hope this helps.

Related

Mule Request Reply and Rollback Exception Strategy

I've a mule flow where I configured Request Reply scope. The flow is like below:
<flow name="RequestReplyFlow">
<http:listener config-ref="HTTP_Listener_Configuration" path="/test" doc:name="HTTP" />
<set-payload value="#['Sample Payload']" doc:name="Set Payload" />
<request-reply doc:name="Request-Reply">
<vm:outbound-endpoint exchange-pattern="one-way" path="request" doc:name="VM" />
<vm:inbound-endpoint exchange-pattern="one-way" path="reply" doc:name="VM" />
</request-reply>
</flow>
and
<flow name="RequestReplyFlow1">
<vm:inbound-endpoint exchange-pattern="one-way" path="request" doc:name="VM" />
<logger message="Executing with payload: #[payload]" level="INFO" doc:name="Logger" />
<component class="org.ram.BusinessComponent" doc:name="Java" />
<rollback-exception-strategy maxRedeliveryAttempts="5" doc:name="Rollback Exception Strategy">
<logger message="Will attempt redelivery" level="INFO" doc:name="Logger" />
</rollback-exception-strategy>
</flow>
I configure rollback-exception-strategy on the request VM inbound endpoint.
When the component org.ram.BusinessComponent throws an exception, what I expected is the message is redelivered to the inbound VM endpoint but it did not happen. Why?
Can anyone please solve the issue?
If you are using rollback exception strategy instead of default exception strategy for unhandled exceptions, then you just have to specify the redirection manually in the "rollback-exception-strategy" by adding the "vm:outbound-endpoint" with the "reply" path, note that the maxRedeliveryAttempts was changed to "0" because your vm is not transactional:
<flow name="RequestReplyFlow1">
<vm:inbound-endpoint exchange-pattern="one-way" path="request" doc:name="VM" />
<logger message="Executing with payload: #[payload]" level="INFO" doc:name="Logger" />
<component class="org.ram.BusinessComponent" doc:name="Java" />
<rollback-exception-strategy maxRedeliveryAttempts="0" doc:name="Rollback Exception Strategy">
<logger message="Will attempt redelivery" level="INFO" doc:name="Logger" />
<vm:outbound-endpoint exchange-pattern="one-way" path="reply" doc:name="VM" />
</rollback-exception-strategy>
</flow>
When the vm is transactional then the value maxRedeliveryAttempts=5 makes sense because in that case Mule attempts message redelivery five (5) times. Then you can specify the redirection manually in the "rollback-exception-strategy" by adding the "vm:outbound-endpoint" with the "reply" path inside the "on-redelivery-attempts-exceeded" child element:
<flow name="RequestReplyFlow1">
<vm:inbound-endpoint exchange-pattern="one-way" path="request" doc:name="VM" />
<logger message="Executing with payload: #[payload]" level="INFO" doc:name="Logger" />
<component class="org.ram.BusinessComponent" doc:name="Java" />
<rollback-exception-strategy maxRedeliveryAttempts="0" doc:name="Rollback Exception Strategy">
<logger message="Will attempt redelivery" level="INFO" doc:name="Logger" />
<on-redelivery-attempts-exceeded>
<logger message="redelivery attempt exceeded" level="INFO" doc:name="Logger" />
<vm:outbound-endpoint exchange-pattern="one-way" path="reply" doc:name="VM" />
</on-redelivery-attempts-exceeded>
</rollback-exception-strategy>
</flow>

Counter not work properly while using collection splitter mule esb

I read data from salesforce and split it using collection splitter. I want to aggregate all data into single list. First I count all records and I set that counter with MULE_CORRELATION_GROUP_SIZE. But my counter gives me wrong count..And my data didn't aggregate..
How can I solve this problem
Is my following flow right?
Following is my code
<flow name="davesalesforceFlow1Tmp" doc:name="davesalesforceFlow1Tmp">
<http:inbound-endpoint exchange-pattern="request-response" host="localhost" port="8080" doc:name="HTTP" path="test" contentType="text/plain" />
<sfdc:query config-ref="salesforce" doc:name="Salesforce" query="My query"/>
<collection-splitter doc:name="Collection Splitter"/>
<set-session-variable variableName="cnt"
value="#[org.mule.util.StringUtils.countMatches(message.payload, '\n') + org.mule.util.StringUtils.countMatches(message.payload, ',') - 1]" doc:name="Session Variable"/>
<set-property propertyName="MULE_CORRELATION_GROUP_SIZE"
value="#[sessionVars.cnt]" doc:name="Property"/>
<collection-aggregator failOnTimeout="true" doc:name="Collection Aggregator"/>
I also tried following code But in output got only single record
<flow name="davesalesforceFlow1Tmp" doc:name="davesalesforceFlow1Tmp">
<http:inbound-endpoint exchange-pattern="request-response" host="localhost" port="8080" doc:name="HTTP" path="test" contentType="text/plain" />
<sfdc:query config-ref="salesforce" doc:name="Salesforce" query="My query"/>
<collection-splitter doc:name="Collection Splitter"/>
<set-property propertyName="MULE_CORRELATION_GROUP_SIZE"
value="#[message.outboundProperties['MULE_CORRELATION_SEQUENCE']]" doc:name="Property"/>
<collection-aggregator failOnTimeout="true" doc:name="Collection Aggregator"/>
<logger message="MY PAYLOAD IS #[payload]" level="INFO" doc:name="Logger"/>
</flow>
</flow>
If you use the collection splitter and aggregator together theres no need to keep track of the count or anything.
Mule will take care of this via the MULE_CORRELATION_* header behind the scenes.
Only if you are dong for complex splits and aggregates may you need to modify these headers yourself.
<logger level="INFO" message="size before: #[payload.size()]" />
<collection-splitter />
<logger level="INFO" message="Individual item: #[payload]" />
<collection-aggregator />
<logger level="INFO" message="size after: #[payload.size()]" />

Vm end point in mule pros and cons

I want to use VM endpoints for achieving parallel processing in mule flows. Being a beginner with mule, I am not quite sure about the implications of doing so. I read about private flows in mule 3 , but not sure, if I can replace vm endpoints with private flows in this case and if at all there would be any advantage I can get with that. Can someone, please let me know about the pros and cons of using VM . Here is the example I wanted to use for parallel processing.
<flow name="forkAndJoinFlow">
<http:inbound-endpoint exchange-pattern="request-response" host="localhost" port="81" path="lowestprice" />
<not-filter>
<wildcard-filter pattern="*favicon*" />
</not-filter>
<request-reply>
<all>
<vm:outbound-endpoint path="shop1"/>
<vm:outbound-endpoint path="shop2"/>
</all>
<vm:inbound-endpoint path="response">
<message-properties-transformer>
<add-message-property key="MULE_CORRELATION_GROUP_SIZE" value="2" />
</message-properties-transformer>
<collection-aggregator />
</vm:inbound-endpoint>
</request-reply>
<expression-transformer evaluator="groovy" expression="java.util.Collections.min(payload)" />
<object-to-string-transformer/>
<logger level="WARN" message="#[string:Lowest price: #[payload]]" />
</flow>
<flow name="shop1Flow">
<vm:inbound-endpoint path="shop1"/>
<logger level="INFO" message="SHOP1 Flow..." />
<expression-transformer evaluator="groovy" expression="new java.lang.Double(1000.0 * Math.random()).intValue()" />
<logger level="WARN" message="#[string:Price from shop 1: #[payload]]" />
</flow>
<flow name="shop2Flow">
<vm:inbound-endpoint path="shop2" />
<logger level="INFO" message="SHOP2 Flow..." />
<expression-transformer evaluator="groovy" expression="new java.lang.Double(1000.0 * Math.random()).intValue()" />
<logger level="WARN" message="#[string:Price from shop 2: #[payload]]" />
</flow>
vm:endpoints are one-way and asynchronous, that means you'll not receive a response back from them.
Private flows can be synchronous as well as asynchronous, means they can return a response also if the exchange-pattern is request-response.
Also, in private flow you get all variables and headers etc versus when you push your message in vm or any other JMS, variables and inbound properties are not propagated and outbound properties of the caller flow becomes inbound in called flow
Private flows can be used as a replacement for your VM.
As explained private flows can be Synchronus or Asynchronous based on the processingStrategy of the flow.
I case you use a private flow in asynchronous way to acheive parallel processing, then make sure that response is posted back onto the response VM at the end of each of the private flows.
I have implemented the fork-join pattern of parallel processing using sub-flows as well. Try this.
<set-property propertyName="MULE_CORRELATION_GROUP_SIZE" value="1" />
<all enableCorrelation="IF_NOT_SET">
<async>
<set-property propertyName="MULE_CORRELATION_SEQUENCE" value="1" />
<flow-ref name="parallel-flow1"></flow-ref>
</async>
<async>
<set-property propertyName="MULE_CORRELATION_SEQUENCE" value="2" />
<flow-ref name="parallel-flow2"></flow-ref>
</async>
</all>
And the sub-flows as follows.
<sub-flow name="parallel-flow1">
....
....
<flow-ref name="join-flow" />
</sub-flow>
<sub-flow name="parallel-flow2">
....
....
<flow-ref name="join-flow" />
</sub-flow>
<sub-flow name="join-flow">
<collection-aggregator timeout="10000" failOnTimeout="true" />
<combine-collections-transformer />
....
....
</sub-flow>
You can try this with private flows.
Hope this helps.

Why is Mule payload getting lost in <catch-exception-strategy> for java.net.ConnectException

I'm testing my Mule(3.3.1) flow which sends a web service call to external vendor. My aim is to catch java.net.ConnectException and apply appropriate XSLT to original payload and send it to caller.
But the payload received in <catch-exception-strategy> is of type org.apache.commons.httpclient.methods.PostMethod#12b13004 and not original XML. Tried using <objexct-to-string-transformer> but didn't help.
Any suggestions how to retrieve the original payload in catch block?
Part of mule-config.xml is below:
<flow name="orderRequirementsToVendor">
<jms:inbound-endpoint queue="order.vendor" />
<set-property propertyName="SOAPAction" value="http://vendor.com/services/InterfacePoint/Call" />
<cxf:proxy-client payload="body" enableMuleSoapHeaders="false">
<cxf:inInterceptors>
<spring:bean class="org.apache.cxf.interceptor.LoggingInInterceptor" />
</cxf:inInterceptors>
<cxf:outInterceptors>
<spring:bean class="org.apache.cxf.interceptor.LoggingOutInterceptor" />
</cxf:outInterceptors>
</cxf:proxy-client>
<outbound-endpoint address="${vendor.ws.url}" mimeType="text/xml" connector-ref="https.connector" />
<byte-array-to-string-transformer />
<choice-exception-strategy>
<catch-exception-strategy when="#[exception.causedBy(java.net.ConnectException)]">
<logger message="#[exception.causeException]" level="ERROR" />
<object-to-string-transformer/>
<transformer ref="vendorConnectExceptionTransformer" />
</catch-exception-strategy>
<catch-exception-strategy>
<logger message="#[exception.causeException]" level="ERROR" />
<transformer ref="generalErrorTransformer" />
</catch-exception-strategy>
</choice-exception-strategy>
</flow>
Store the original payload in a flow variable right after jms:inbound-endpoint with
<set-variable variableName="originalPayload" value="#[message.payload]" />
Access it back in your exception strategy with a MEL expression like: #[flowVars.originalPayload].

Mule - Unable to find an StreamCloser for the stream type: class java.lang.String

After the JMS message is delivered to the queue I see a log statement related to Stream Closer. It doesn't look right to me .. why do I see this message?
2013-04-22 19:08:29,385 [DEBUG] org.mule.transport.jms.activemq.ActiveMQJmsConnector - Returning dispatcher for endpoint: jms://retry.queue = EeJmsMessageDispatcher{this=5c5801d7, endpoint=jms://retry.queue, disposed=false}
2013-04-22 19:08:29,433 [DEBUG] org.mule.util.DefaultStreamCloserService - Unable to find an StreamCloser for the stream type: class java.lang.String, the stream: <?xml version="1.0" encoding="UTF-8"?> < ....... rest of the XML ....... /> will not be closed.
What does it mean by - "the stream: will not be closed."?
What should I do to fix this?
====EDIT =====
There is an error happening. The JMS message has XML as payload. Mule version: 3.3.2
Here's my flow
<flow name="sendToHost">
<jms:inbound-endpoint queue="host.queue" exchange-pattern="one-way" />
<copy-properties propertyName="*" />
<file:outbound-endpoint path="/hostmessages" outputPattern="outgoing-xml-[function:dateStamp].log" />
<set-variable variableName="hostXML" value="#[payload]" />
<flow-ref name="webServiceCall" />
<flow-ref name="inspectWSResponse" />
<exception-strategy ref="retryExceptionStrategy" />
</flow>
<flow name="resendFailedMessages">
<description>
"*/15 07-18 * * ?" run every 15 minutes from 7 am to 6 pm every day -->
</description>
<quartz:inbound-endpoint jobName="hostRedeliveryJob" cronExpression="0 0/1 * * * ?">
<quartz:endpoint-polling-job>
<quartz:job-endpoint ref="redeliverToHost" />
</quartz:endpoint-polling-job>
</quartz:inbound-endpoint>
<set-variable variableName="hostXML" value="#[payload]" />
<logger message="QUARTZ found message for host" level="INFO" />
<flow-ref name="webServiceCall" />
<flow-ref name="inspectWSResponse" />
<exception-strategy ref="retryExceptionStrategy" />
</flow>
<choice-exception-strategy name="retryExceptionStrategy">
<catch-exception-strategy when="#[exception.causedBy(java.io.IOException)]">
<logger message="In retryExceptionStrategy IO exception strategy. " level="ERROR" />
<logger message="retryExceptionStrategy exception is #[exception.causeException]" level="ERROR" />
<set-property propertyName="exception" value="#[exception.summaryMessage]" />
<set-payload value="#[hostXML]" />
<logger message="retryExceptionStrategy payload is #[payload]" level="ERROR" />
<jms:outbound-endpoint queue="retry.queue" />
</catch-exception-strategy>
<catch-exception-strategy>
<logger message="Other error in sending result to host in retryExceptionStrategy flow." level="INFO" />
<set-property propertyName="exception" value="#[exception.summaryMessage]" />
<set-payload value="#[hostXML]" />
<jms:outbound-endpoint queue="declined.queue" />
</catch-exception-strategy>
</choice-exception-strategy>
<sub-flow name="webServiceCall">
<cxf:proxy-client payload="body" enableMuleSoapHeaders="false">
<cxf:inInterceptors>
<spring:bean class="org.apache.cxf.interceptor.LoggingInInterceptor" />
</cxf:inInterceptors>
<cxf:outInterceptors>
<spring:bean class="org.apache.cxf.interceptor.LoggingOutInterceptor" />
</cxf:outInterceptors>
</cxf:proxy-client>
<outbound-endpoint address="${host.ws.url}" mimeType="text/xml" connector-ref="http.connector" />
<byte-array-to-string-transformer />
</sub-flow>
<sub-flow name="inspectWSResponse">
<choice>
<when expression="#[xpath('//acord:TestResult/acord:TestCode/acord:Name/#tc').value == '1']">
<logger message="Message Delivered Successfully to host" level="INFO" />
</when>
<otherwise>
<set-payload value="#[hostXML]" />
<jms:outbound-endpoint queue="declined.queue" />
</otherwise>
</choice>
</sub-flow>
Log entries at DEBUG level can typically be safely ignored.
In this particular case, it seems Mule is using the StreamCloserService on a message's payload that is not a stream but a string.
Looking at the source code this can seem to possibly happen only when an exception is processed and Mule attempts to forcefully close a streaming payload without first checking if it is actually streaming. This is benign and can't trigger any side-effect so you can safely ignore this DEBUG statement.