Scatter-Gather: merge into one map - mule

I need to query several tables and at the end of the scatter-gather flow I want to have a payload of this form
{table_1: [ list of results ], table_2: [ list of results ]}
I tried this:
<scatter-gather doc:name="Scatter-Gather">
<processor-chain>
<db:select config-ref="dataSrouce" doc:name="Table 1">
<db:parameterized-query><![CDATA[// my query
</db:select>
<set-payload value="#[['table_1': #[payload]]]" doc:name="Set Payload"/>
</processor-chain>
<processor-chain>
<db:select config-ref="dataSrouce" doc:name="Table 2">
<db:parameterized-query><![CDATA[// my query
</db:select>
<set-payload value="#[['table_2': #[payload]]]" doc:name="Set Payload"/>
</processor-chain>
</scatter-gather>
<set-payload value="#[groovy:payload.inject([:]) {result, entry -> result += entry; result}]" doc:name="Set Payload"/>
But getting an exception:
Exception was found for route(s): 1. Message payload is of type: CaseInsensitiveHashMap (org.mule.routing.CompositeRouting
Exception)
org.mule.routing.CollectAllAggregationStrategy:51 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/routing/Comp
ositeRoutingException.html)

In each branch of your Scatter-Gather place an ObjectToString and an ObjectToJSON Transformer.
Then after the Scatter-Gather place a SetPayload with the following value:
{
"table_1": #[payload[0]],
"table_2": #[payload[1]]
}
Example:
<scatter-gather doc:name="Scatter-Gather">
<processor-chain>
<flow-ref name="Subflow_1" doc:name="Subflow_1"/>
<object-to-string-transformer doc:name="Object to String"/>
<json:object-to-json-transformer doc:name="Object to JSON"/>
</processor-chain>
<processor-chain>
<flow-ref name="Subflow_2" doc:name="Subflow_2"/>
<object-to-string-transformer doc:name="Object to String"/>
<json:object-to-json-transformer doc:name="Object to JSON"/>
</processor-chain>
</scatter-gather>
<set-payload value="{
"table_1": #[payload[0]],
"table_2": #[payload[1]]}"
doc:name="Set Payload" mimeType="application/json"/>

Related

Anypoint mule: CopyOnArraylist in output while writing to file with scatter-gather element

The problem statement is to do multiple things parallely and aggregate the response and store it in a file.
link to the mule flow image as in studio:
image
In this flow, what I was trying to do was to set two constant strings in two branches of scatter and gather and aggregate and store in file. I tried overwriting the payload with a "set payload" with "my response". I am expecting "my response" as the content of the file. But instead the file content is:
¨Ìsr)java.util.concurrent.CopyOnWriteArrayListx]ü’F´ê√xpwtmsg 1tmsg 2x
I did debug and the payload at File endpoint was "my response". How and why is the collection getting written into file.
Can anyone help me to get it working.
Following is the xml:
<flow name="mule-assignFlow21123">
<quartz:inbound-endpoint jobName="dummyflow" repeatInterval="10000" responseTimeout="10000" doc:name="Quartz">
<quartz:event-generator-job/>
</quartz:inbound-endpoint>
<scatter-gather doc:name="Scatter-Gather1" >
<threading-profile maxThreadsActive="1" poolExhaustedAction="RUN"/>
<processor-chain>
<set-payload value="msg 1" doc:name="Set Payload"/>
<logger level="INFO" doc:name="Logger"/>
</processor-chain>
<processor-chain>
<set-payload value="msg 2" doc:name="Set Payload"/>
<logger level="INFO" doc:name="Logger"/>
</processor-chain>
</scatter-gather>
<set-payload value="my response" doc:name="Set Payload"/>
<file:outbound-endpoint path="/Users/premkumar/Desktop" outputPattern="Results.txt" responseTimeout="10000" mimeType="text/plain" doc:name="Save 2 File"/>
</flow>
The flow will automatically determine the processingStrategy from the in-flight event which will be async because of the quartz endpoint, so the file endpoint will fire async also.
Instead explicitly set the flow's processingStrategy to synchronous:
<flow name="mule-assignFlow21123" processingStrategy="synchronous">
<quartz:inbound-endpoint jobName="dummyflow" repeatInterval="10000" responseTimeout="10000" doc:name="Quartz">
<quartz:event-generator-job/>
</quartz:inbound-endpoint>
<scatter-gather doc:name="Scatter-Gather1" >
<threading-profile maxThreadsActive="1" poolExhaustedAction="RUN"/>
<processor-chain>
<set-payload value="msg 1" doc:name="Set Payload"/>
<logger level="INFO" doc:name="Logger"/>
</processor-chain>
<processor-chain>
<set-payload value="msg 2" doc:name="Set Payload"/>
<logger level="INFO" doc:name="Logger"/>
</processor-chain>
</scatter-gather>
<set-payload value="my response" doc:name="Set Payload"/>
<file:outbound-endpoint path="/Users/premkumar/Desktop" outputPattern="Results.txt" responseTimeout="10000" mimeType="text/plain" doc:name="Save 2 File"/>

Mule Flow not able to invoke exception strategy

I have following Code for my Mule FLOW and seems like something is wrong with my code which is not allowing to invoke Catch Exception Strategy, What are the corrections required here ?
<flow name="Mule_Common_FTP_EDI_FilesProcessing_Inbound" doc:name="Mule_Common_FTP_EDI_FilesProcessing_Inbound"
processingStrategy="synchronous">
<quartz:inbound-endpoint cronExpression="${ftp.edi.inbound.cronExpression}" responseTimeout="${ftp.edi.inbound.root.responseTimeout}" jobName="pollingFTPFiles" doc:name="Quartz" connector-ref="pollingFTPFilesConnector">
<quartz:event-generator-job/>
</quartz:inbound-endpoint>
<set-payload value="#[[${ftp.edi.inbound.carriers.path}]]" doc:name="Set Payload"/>
<foreach doc:name="For Each">
<flow-ref name="Initialise-logging-variables" doc:name="Flow Reference"/>
<set-variable value="#[payload]" variableName="processing_folder_path" doc:name="Variable processing_folder_path" />
<mulerequester:request config-ref="FTP_Requester" resource="ftp://${ftp.edi.inbound.root.user}:${ftp.edi.inbound.root.password}#${ftp.edi.inbound.root.host}:${ftp.edi.inbound.root.port}${ftp.edi.inbound.root.path}#[payload]" timeout="${ftp.edi.inbound.root.responseTimeout}" doc:name="Mule Requester"/>
<choice doc:name="Choice">
<when expression="#[message.payload != null]">
<set-variable variableName="careerName" value="#[groovy:String s = flowVars.processing_folder_path; s.substring(0,s.indexOf('-'))]" doc:name="Variable careerName"/>
<set-variable variableName="filename" value="#[variable:careerName]-_#[message.outboundProperties['originalFilename']]" doc:name="Variable filename"/>
<file:file-to-string-transformer doc:name="File to String"/>
<set-variable variableName="InputPayload" value="#[payload:]" doc:name="Variable InputPayload"/>
<choice doc:name="Choice">
<when expression="#[regex(app.registry.configProperties[careerName])]">
<scripting:component doc:name="Script">
<scripting:script engine="Groovy" file="generateWSRequest.groovy"/>
</scripting:component>
<message-properties-transformer doc:name="Message Properties">
<add-message-property key="SOAPAction" value="/ProcessService/PDoc"/>
</message-properties-transformer>
<http:outbound-endpoint exchange-pattern="request-response" host="${http.host}" port="${http.port}" path="yourservices/service.svc" responseTimeout="${http.timeout.sync}" contentType="text/xml" connector-ref="httpConnector" doc:name="HTTP"/>
<byte-array-to-string-transformer doc:name="Byte Array to String"/>
<set-variable value="#[message.payload.toString()]" variableName="OutputPayload" doc:name="Variable"/>
<choice doc:name="Choice">
<when expression="message.getInboundProperty('http.status') == '200'" evaluator="groovy">
<set-payload value="#[variable:InputPayload]" doc:name="Set Payload"/>
<file:outbound-endpoint path="${ftp.edi.outbound.root.success.path}" outputPattern="#[variable:filename]" responseTimeout="10000" doc:name="File"/>
<set-payload value="#[variable:InputPayload]" doc:name="Set Payload"/>
<ftp:outbound-endpoint host="${ftp.edi.inbound.root.host}" port="${ftp.edi.inbound.root.port}" path="${ftp.edi.elementum.copy.path}" user="${ftp.edi.inbound.root.user}" password="${ftp.edi.inbound.root.password}" outputPattern="#[message.outboundProperties['originalFilename']]" mimeType="text/plain" doc:name="FTP"/>
</when>
<otherwise>
<set-payload value="#[variable:InputPayload]" doc:name="Set Payload"/>
<set-variable variableName="original_message" value="#[groovy:message.toString()]" doc:name="original_message" />
<set-attachment attachmentName="#[message.outboundProperties['originalFilename']]" value="#[payload]" contentType="text/xml" doc:name="originalFileAttachment"/>
<set-variable variableName="content" value="${smtp.service.failure.body}, Splunk txId=#[sessionVars.logTxid] #[System.getProperty('line.separator')] #[System.getProperty('line.separator')] #[variable:logErrmsg] #[System.getProperty('line.separator')] #[System.getProperty('line.separator')] #[variable:original_message] #[System.getProperty('line.separator')]" doc:name="EmailBodyContent" />
<set-payload value="#[variable:content]" doc:name="Set Payload"/>
<set-variable variableName="subjectEmail" value="${smtp.service.failure.subject}filename=#[message.outboundProperties['originalFilename']]" doc:name="Variable"/>
<smtp:outbound-endpoint host="${smtp.host}" port="${smtp.port}" to="${smtp.to}" from="${smtp.from}" subject="#[variable:subjectEmail]" responseTimeout="10000" doc:name="HTTP_EMAIL_ALERT" mimeType="text/plain"/>
<set-payload value="#[variable:InputPayload]" doc:name="Set Payload"/>
<file:outbound-endpoint path="${ftp.edi.outbound.root.reprocesspath}" outputPattern="#[variable:filename]" responseTimeout="10000" doc:name="File"/>
<set-payload value="#[variable:InputPayload]" doc:name="Set Payload"/>
<ftp:outbound-endpoint host="${ftp.edi.inbound.root.host}" port="${ftp.edi.inbound.root.port}" path="${ftp.edi.elementum.copy.path}" user="${ftp.edi.inbound.root.user}" password="${ftp.edi.inbound.root.password}" outputPattern="#[message.outboundProperties['originalFilename']]" mimeType="text/plain" doc:name="FTP"/>
</otherwise>
</choice>
</when>
<otherwise>
<set-variable variableName="original_message" value="#[groovy:message.toString()]" doc:name="original_message" />
<set-attachment attachmentName="#[message.outboundProperties['originalFilename']]" value="#[payload]" contentType="text/xml" doc:name="OriginalFileAttachment"/>
<set-variable variableName="content" value="${smtp.file.not.supported.body}, Splunk txId=#[sessionVars.logTxid] #[System.getProperty('line.separator')] #[System.getProperty('line.separator')] #[variable:logErrmsg] #[System.getProperty('line.separator')] #[System.getProperty('line.separator')] #[variable:original_message] #[System.getProperty('line.separator')]" doc:name="EmailBodyContent" />
<set-payload value="#[variable:content]" doc:name="Set Payload"/>
<set-variable variableName="subjectEmail" value="${smtp.file.not.supported.subject}filename=#[message.outboundProperties['originalFilename']]" doc:name="Variable"/>
<smtp:outbound-endpoint host="${smtp.host}" port="${smtp.port}" to="${smtp.to}" from="${smtp.from}" subject="#[variable:subjectEmail]" responseTimeout="10000" doc:name="REGEX_NOT_FOUND_EMAIL_ALERT" mimeType="text/plain"/>
<logger message="{message = Email Sent to ${smtp.to}}" level="ERROR" doc:name="Logger"/>
<file:outbound-endpoint path="${ftp.edi.outbound.root.failedpath}" outputPattern="#[variable:filename]" responseTimeout="10000" doc:name="File"/>
</otherwise>
</choice>
</when>
<otherwise>
<logger message="{InboundPath=#[processing_folder_path], Message= REQUEST NOT PROCESSED NO MATCHING TEXT FOUND}" level="DEBUG" doc:name="Logger"/>
</otherwise>
</choice>
</foreach>
<choice-exception-strategy doc:name="Choice Exception Strategy">
<rollback-exception-strategy when="exception.causedBy(java.net.ConnectException) || exception.causedBy(java.net.SocketTimeoutException) || exception.causedBy(java.net.SocketException) || exception.causedBy(java.lang.Throwable)" maxRedeliveryAttempts="4" doc:name="Rollback Exception Strategy">
<on-redelivery-attempts-exceeded doc:name="Redelivery exhausted">
<set-variable variableName="errorMessage" value="{InboundPath=#[processing_folder_path], Ref= #[message.outboundProperties['originalFilename']], Content= #[groovy:payload.toString().replaceAll('\n', '')], Flow= #[variable:careerName], Message= MAXIMUM RETRIES REACHED}" doc:name="Variable"/>
<logger message="#[variable:errorMessage]" level="ERROR" doc:name="Logger"/>
<set-variable variableName="original_message" value="#[groovy:message.toString()]" doc:name="original_message" />
<set-attachment attachmentName="#[message.outboundProperties['originalFilename']]" value="#[payload]" contentType="text/xml" doc:name="OriginalFileAttachment"/>
<set-variable variableName="content" value="${smtp.rollback.body}, Splunk txId=#[sessionVars.logTxid] #[System.getProperty('line.separator')] #[System.getProperty('line.separator')] #[variable:logErrmsg] #[System.getProperty('line.separator')] #[System.getProperty('line.separator')] #[variable:original_message] #[System.getProperty('line.separator')]" doc:name="EmailBodyContent" />
<set-payload value="#[variable:content]" doc:name="Set Payload"/>
<set-variable variableName="subjectEmail" value="${smtp.rollback.subject}filename=#[message.outboundProperties['originalFilename']]" doc:name="Variable"/>
<smtp:outbound-endpoint host="${smtp.host}" port="${smtp.port}" to="${smtp.to}" from="${smtp.from}" subject="#[variable:subjectEmail]" responseTimeout="10000" doc:name="RETRY_FAILURE_EMAIL_ALERT" mimeType="text/plain"/>
<logger message="{message = Email Sent to ${smtp.to}}" level="ERROR" doc:name="Logger"/>
<file:file-to-string-transformer doc:name="File to String"/>
<file:outbound-endpoint path="${ftp.edi.outbound.root.failedpath}" responseTimeout="10000" outputPattern="#[message.outboundProperties['originalFilename']]" doc:name="File"/>
</on-redelivery-attempts-exceeded>
</rollback-exception-strategy>
<catch-exception-strategy when="exception.causeMatches(java.*)" doc:name="Data Errors Catch Exception Strategy">
<set-variable variableName="errorMessage" value="{InboundPath=#[processing_folder_path], Ref= #[message.outboundProperties['originalFilename']], Content= #[groovy:payload.toString().replaceAll('\n', '')], Flow= #[variable:careerName], Message=Common Exception Occured look at exception stackTrace for more details}" doc:name="Variable"/>
<logger message="#[variable:errorMessage]" level="ERROR" doc:name="Logger"/>
<set-variable variableName="original_message" value="#[groovy:message.toString()]" doc:name="original_message" />
<set-attachment attachmentName="#[message.outboundProperties['originalFilename']]" value="#[payload]" contentType="text/xml" doc:name="OriginalFileAttachment"/>
<set-variable variableName="content" value="${smtp.catch.exception.body}, Splunk txId=#[sessionVars.logTxid] #[System.getProperty('line.separator')] #[System.getProperty('line.separator')] #[variable:logErrmsg] #[System.getProperty('line.separator')] #[System.getProperty('line.separator')] #[variable:original_message] #[System.getProperty('line.separator')]" doc:name="EmailBodyContent" />
<set-payload value="#[variable:content]" doc:name="Set Payload"/>
<set-variable variableName="subjectEmail" value="${smtp.catch.exception.subject}filename=#[message.outboundProperties['originalFilename']]" doc:name="Variable"/>
<smtp:outbound-endpoint host="${smtp.host}" port="${smtp.port}" to="${smtp.to}" from="${smtp.from}" subject="#[variable:subjectEmail]" responseTimeout="10000" doc:name="CATCH_EXCEPTION_EMAIL_ALERT" mimeType="text/plain"/>
<logger message="{message = Email Sent to ${smtp.to}}" level="ERROR" doc:name="Logger"/>
<file:file-to-string-transformer doc:name="File to String"/>
<file:outbound-endpoint path="${ftp.edi.outbound.root.failedpath}" responseTimeout="10000" outputPattern="#[message.outboundProperties['originalFilename']]" doc:name="File"/>
</catch-exception-strategy>
</choice-exception-strategy>
</flow>
I am trying to put invalid WEBSERVICE URL here which is causing this error and i want to catch those kind of errors and send out emails if service is down or configuration is wrong and not able to process the files :
What i need to change here in my code which will allow me to receive email notification about this error ? Can you please advice me what's wrong with Exception Strategy here?
INFO 2015-04-29 11:31:40,105 [scheduler-edi-ftp-file-processing-1.0.0-SNAPSHOT-v20150427-19+32_Worker-1] org.mule.transport.http.transformers.ObjectToHttpClientMethodRequest: Content-Type not set on outgoing request, defaulting to: text/plain
ERROR 2015-04-29 11:31:40,123 [scheduler-edi-ftp-file-processing-1.0.0-SNAPSHOT-v20150427-19+32_Worker-1] org.mule.exception.DefaultSystemExceptionStrategy:
********************************************************************************
Message : Failed to route event via endpoint: DefaultOutboundEndpoint{endpointUri=http://myservice.company.com:80/services/service.svc, connector=HttpConnector
{
name=httpConnector
lifecycle=start
this=4d81bcef
numberOfConcurrentTransactedReceivers=4
createMultipleTransactedReceivers=true
connected=true
supportedProtocols=[http]
serviceOverrides=
session.handler=org.mule.session.NullSessionHandler
}
, name='endpoint.http.myservice.company.com.80.services.service.svc', mep=REQUEST_RESPONSE, properties={Content-Type=text/xml}, transactionConfig=Transaction{factory=null, action=INDIFFERENT, timeout=0}, deleteUnacceptedMessages=false, initialState=started, responseTimeout=120000, endpointEncoding=UTF-8, disableTransportTransformer=false}. Message payload is of type: PostMethod
Code : MULE_ERROR-42999
--------------------------------------------------------------------------------
Exception stack is:
1. myservice.company.com (java.net.UnknownHostException)
java.net.AbstractPlainSocketImpl:175 (null)
2. Failed to route event via endpoint: DefaultOutboundEndpoint{endpointUri=http://myservice.mycompany.com:80/services/service.svc, connector=HttpConnector
{
name=httpConnector
lifecycle=start
this=4d81bcef
numberOfConcurrentTransactedReceivers=4
createMultipleTransactedReceivers=true
connected=true
supportedProtocols=[http]
serviceOverrides=
session.handler=org.mule.session.NullSessionHandler
}
, name='endpoint.http.myservice.company.com.80.services.service.svc', mep=REQUEST_RESPONSE, properties={Content-Type=text/xml}, transactionConfig=Transaction{factory=null, action=INDIFFERENT, timeout=0}, deleteUnacceptedMessages=false, initialState=started, responseTimeout=120000, endpointEncoding=UTF-8, disableTransportTransformer=false}. Message payload is of type: PostMethod (org.mule.api.transport.DispatchException)
org.mule.transport.http.HttpClientMessageDispatcher:155 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/api/transport/DispatchException.html)
--------------------------------------------------------------------------------
Root Exception stack trace:
java.net.UnknownHostException: myservice.company.com
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:175)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
at java.net.Socket.connect(Socket.java:546)
+ 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
Either one of the following:
Add "|| exception.causedBy(java.net.UnknownHostException)" to your first exception strategy to catch only unknown hosts exception.
Change your second catch exception strategy into exception.causeMatches('java.net.*') to catch all network exceptions (notice the quotes).
Change your second catch exception strategy into exception.causeMatches('java.*') to catch all java exceptions (notice the quotes).

mule move file to another location dynamically from the db query

I want to move files to location D:\target dynamically from the query select filepath from emp where status='y':
This is my table:
emp_Name File Path File Name Status
ABC D:\emp abc.txt y
xyz D:\emp xyz.txt y
bcs D:\emp bcs.txt n
Following is my source code :
<flow name="testdbFlow1">
<http:listener config-ref="HTTP_Listener_Configuration"
path="/" doc:name="HTTP" />
<jdbc-ee:outbound-endpoint queryKey="allEmps"
queryTimeout="-1" connector-ref="JDBCConnector" exchange-pattern="request-response"
doc:name="Database" />
<foreach doc:name="Foreach">
<choice doc:name="Choice">
<when expression="#[payload.status == 'Y']">
<processor-chain doc:name="Processor Chain">
<set-variable variableName="filePath" value="#[payload.filepath]"
doc:name="Variable" />
<set-variable variableName="filename" value="#[payload.filename]"
doc:name="Variable" />
<logger message="#[filePath]" level="INFO"
doc:name="Logger" />
<logger message="#[filename]"
level="INFO" doc:name="Logger" />
<file:inbound-endpoint path="#[filePath]" name="input" doc:name="File"
pollingFrequency="12000" responseTimeout="10000"> <file:filename-wildcard-filter
pattern="#[filename]" /> </file:inbound-endpoint>
<file:outbound-endpoint name="output" path="D:\target" doc:name="File"/>
</processor-chain>
</when>
<otherwise>
<processor-chain doc:name="Processor Chain">
<set-variable variableName="filePath" value="#[payload.filepath]"
doc:name="Variable" />
<set-variable variableName="filename" value="#[payload.filename]"
doc:name="Variable" />
<logger message="#[filePath]" level="INFO"
doc:name="Logger" />
<logger message="#[filename]"
level="INFO" doc:name="Logger" />
</processor-chain>
</otherwise>
</choice>
</foreach>
</flow>
But it's not working.
Inbound doesn't work out of the box in the middle of the flow. I would suggest a simpler solution using Groovy because you know exactly the file you need to consume.
Instead of using file:inbound create a input stream manually:
<set-payload value="#[groovy: new java.io.FileInputStream(new java.io.File(filename))]"/>
You will have to delete the file with another groovy script after consumed.

Mule scatter-gather router throwing exception

I was exploring Mule scatter-gather for parallel processing of flows in Mule fork and join pattern. My Mule config is as follows:
<flow name="fork" doc:name="fork">
<http:inbound-endpoint host="localhost" port="8090" path="mainPath" exchange-pattern="request-response" doc:name="HTTP"/>
<set-property propertyName="MULE_CORRELATION_GROUP_SIZE" value="2" doc:name="Property"/>
<!-- <all enableCorrelation="IF_NOT_SET" doc:name="All"> -->
<scatter-gather timeout="6000">
<processor-chain>
<async doc:name="Async">
<set-property propertyName="MULE_CORRELATION_SEQUENCE" value="1" doc:name="Property"/>
<flow-ref name="Flow1" doc:name="Flow Reference"/>
</async>
</processor-chain>
<async doc:name="Async">
<set-property propertyName="MULE_CORRELATION_SEQUENCE" value="2" doc:name="Property"/>
<flow-ref name="Flow2" doc:name="Flow Reference"/>
</async>
<!-- </all> -->
</scatter-gather>
<logger message="Main Flow" level="INFO" doc:name="Logger"/>
</flow>
<sub-flow name="Flow1" doc:name="Flow1">
<logger level="INFO" message="Flow1: processing started" doc:name="Logger"/>
<set-payload value="Flow1 Payload" doc:name="Set Payload"/>
<!-- Transformation payload -->
<logger level="INFO" message="Flow1: processing finished" doc:name="Logger"/>
<flow-ref name="Join-Flow" doc:name="Flow Reference"/>
</sub-flow>
<sub-flow name="Flow2" doc:name="Flow2">
<logger level="INFO" message="Flow2: processing started" doc:name="Logger"/>
<set-payload value="Flow2 Payload" doc:name="Set Payload"/>
<scripting:component doc:name="Groovy">
<scripting:script engine="Groovy"><![CDATA[sleep(2000); return message.payload;]]></scripting:script>
</scripting:component>
<!-- Transformation payload -->
<logger level="INFO" message="Flow2: processing finished" doc:name="Logger"/>
<flow-ref name="Join-Flow" doc:name="Flow Reference"/>
</sub-flow>
<sub-flow name="Join-Flow" doc:name="Join-Flow">
<collection-aggregator timeout="6000" failOnTimeout="true" doc:name="Collection Aggregator"/>
<combine-collections-transformer doc:name="Combine Collections"/>
<logger level="INFO" message="Combined Payload: #[message.payload]" doc:name="Logger"/>
<set-payload value="Soap XML Response" doc:name="Set Payload"/>
</sub-flow>
The scatter and gather is throwing following exception though I am getting the aggregated Payload in Logger:
Exception stack is:
1. null (java.lang.UnsupportedOperationException)
org.mule.VoidMuleEvent:50 (null)
2. null (java.lang.UnsupportedOperationException). Message payload is of type: String (org.mule.api.MessagingException)
org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor:32 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/api/MessagingException.html)
--------------------------------------------------------------------------------
Root Exception stack trace:
java.lang.UnsupportedOperationException
at org.mule.VoidMuleEvent.getMessage(VoidMuleEvent.java:50)
at org.mule.api.routing.AggregationContext$1.evaluate(AggregationContext.java:41)
at org.apache.commons.collections.CollectionUtils.select(CollectionUtils.java:517)
+ 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************
INFO 2014-08-17 18:04:52,008 [[try1].fork.2.02] org.mule.api.processor.LoggerMessageProcessor: Flow2: processing started
INFO 2014-08-17 18:04:52,016 [[try1].fork.1.02] org.mule.api.processor.LoggerMessageProcessor: Flow1: processing started
INFO 2014-08-17 18:04:52,017 [[try1].fork.1.02] org.mule.api.processor.LoggerMessageProcessor: Flow1: processing finished
INFO 2014-08-17 18:04:54,309 [[try1].fork.2.02] org.mule.api.processor.LoggerMessageProcessor: Flow2: processing finished
INFO 2014-08-17 18:04:54,347 [[try1].fork.2.02] org.mule.api.processor.LoggerMessageProcessor: Combined Payload: [Flow1 Payload, Flow2 Payload]
One interesting fact is that if I use <all> router instead of scatter-gather, I don't get this exception. but I guess <all> router process the flows sequentially and not in parallel as per Mule documentation. So for that reason I tried to choose scatter-gather over all router. But I am not sure how to handle the above exception thrown by the scatter-gather. Is there any way out?
UPDATED FLOW :-
<flow name="fork" doc:name="fork">
<http:inbound-endpoint host="localhost" port="8090" path="mainPath" exchange-pattern="request-response" doc:name="HTTP"/>
<scatter-gather timeout="6000">
<processor-chain>
<flow-ref name="Flow1" doc:name="Flow Reference"/>
</processor-chain>
<flow-ref name="Flow2" doc:name="Flow Reference"/>
</scatter-gather>
<logger message="Main Flow" level="INFO" doc:name="Logger"/>
</flow>
<sub-flow name="Flow1" doc:name="Flow1">
<logger level="INFO" message="Flow1: processing started" doc:name="Logger"/>
<set-payload value="Flow1 Payload" doc:name="Set Payload"/>
<!-- Transformation payload -->
<logger level="INFO" message="Flow1: processing finished" doc:name="Logger"/>
<flow-ref name="Join-Flow" doc:name="Flow Reference"/>
</sub-flow>
<sub-flow name="Flow2" doc:name="Flow2">
<logger level="INFO" message="Flow2: processing started" doc:name="Logger"/>
<set-payload value="Flow2 Payload" doc:name="Set Payload"/>
<scripting:component doc:name="Groovy">
<scripting:script engine="Groovy"><![CDATA[sleep(2000); return message.payload;]]>
</scripting:script>
</scripting:component>
<!-- Transformation payload -->
<logger level="INFO" message="Flow2: processing finished" doc:name="Logger"/>
<flow-ref name="Join-Flow" doc:name="Flow Reference"/>
</sub-flow>
<sub-flow name="Join-Flow" doc:name="Join-Flow">
<collection-aggregator timeout="6000" failOnTimeout="true" doc:name="Collection Aggregator"/>
<combine-collections-transformer doc:name="Combine Collections"/>
<logger level="INFO" message="Combined Payload: #[message.payload]" doc:name="Logger"/>
<set-payload value="Soap XML Response" doc:name="Set Payload"/>
</sub-flow>
EXCEPTION :-
[try1].ScatterGatherWorkManager.02] org.mule.api.processor.LoggerMessageProcessor: Flow2: processing started
INFO 2014-08-18 13:34:24,361 [[try1].ScatterGatherWorkManager.01] org.mule.api.processor.LoggerMessageProcessor: Flow1: processing started
INFO 2014-08-18 13:34:24,364 [[try1].ScatterGatherWorkManager.01] org.mule.api.processor.LoggerMessageProcessor: Flow1: processing finished
WARN 2014-08-18 13:34:24,366 [[try1].ScatterGatherWorkManager.01] org.mule.routing.correlation.CollectionCorrelatorCallback: Correlation Group Size not set, but correlation aggregator is being used. Message is being forwarded as is
INFO 2014-08-18 13:34:24,401 [[try1].ScatterGatherWorkManager.01] org.mule.api.processor.LoggerMessageProcessor: Combined Payload: [Flow1 Payload]
INFO 2014-08-18 13:34:26,615 [[try1].ScatterGatherWorkManager.02] org.mule.api.processor.LoggerMessageProcessor: Flow2: processing finished
ERROR 2014-08-18 13:34:26,625 [[try1].connector.http.mule.default.receiver.03] org.mule.exception.DefaultMessagingExceptionStrategy:
********************************************************************************
Message : null (java.lang.NullPointerException). Message payload is of type: String
Code : MULE_ERROR--2
--------------------------------------------------------------------------------
Exception stack is:
1. null (java.lang.NullPointerException)
org.mule.api.routing.AggregationContext$1:41 (null)
2. null (java.lang.NullPointerException). Message payload is of type: String (org.mule.api.MessagingException)
org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor:32 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/api/MessagingException.html)
--------------------------------------------------------------------------------
Root Exception stack trace:
java.lang.NullPointerException
at org.mule.api.routing.AggregationContext$1.evaluate(AggregationContext.java:41)
at org.apache.commons.collections.CollectionUtils.select(CollectionUtils.java:517)
at org.apache.commons.collections.CollectionUtils.select(CollectionUtils.java:498)
+ 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************
Now as you can see it's not combining the both payload of both flow after removing the set-property ..... It is getting only payload of first flow :- Combined Payload: [Flow1 Payload] ..
Remove the async scope around the message processors in the scatter-gather, which indeed does the parallelization for you so you don't have to.
EDIT: Also remove the set-property message processors that deal with the MULE_CORRELATION properties. The scatter-gather is supposed to do that for you.
EDIT2: You can remove the processor-chain around the single flow-ref: it's useless.
Also, there seem to be a deep misunderstanding about what scatter-gather does: you forcefully make its sub-flow responses converge to a single Join-Flow where you aggregate stuff, effectively by-passing and re-implementing all what the scatter-gather proposes to do for you.
Remove the flow-refs towards Join-Flow, remove Join-Flow and just put its processing logic (not the aggregator) after the scatter-gather.
So as per David's suggestion the final working solution is :-
<flow name="fork" doc:name="fork">
<http:inbound-endpoint host="localhost" port="8090" path="scattergather" exchange-pattern="request-response" doc:name="HTTP"/>
<scatter-gather timeout="6000">
<!-- Calling Flow1 -->
<flow-ref name="Flow1" doc:name="Flow Reference"/>
<!-- Calling Flow2 -->
<flow-ref name="Flow2" doc:name="Flow Reference"/>
</scatter-gather>
<!-- <collection-aggregator timeout="6000" failOnTimeout="true" doc:name="Collection Aggregator"/>
<combine-collections-transformer doc:name="Combine Collections"/> -->
<logger level="INFO" message="Combined Payload: #[message.payload]" doc:name="Logger"/>
<logger level="INFO" message="Payload1: #[message.payload[0]] and Payload2: #[message.payload[1]] " doc:name="Logger"/>
<set-payload value="Done Merging ...!!!" doc:name="Set Payload"/>
<logger message="Back to Main Flow" level="INFO" doc:name="Logger"/>
</flow>
<sub-flow name="Flow1" doc:name="Flow1">
<logger level="INFO" message="Flow1: processing started" doc:name="Logger"/>
<set-payload value="Flow1 Payload" doc:name="Set Payload"/>
<logger level="INFO" message="Flow1: processing finished" doc:name="Logger"/>
</sub-flow>
<sub-flow name="Flow2" doc:name="Flow2">
<logger level="INFO" message="Flow2: processing started" doc:name="Logger"/>
<!-- Sleep function to delay the flow2 payload -->
<set-payload value="Flow2 Payload" doc:name="Set Payload"/>
<scripting:component doc:name="Groovy">
<scripting:script engine="Groovy"><![CDATA[sleep(3000); return message.payload;]]></scripting:script>
</scripting:component>
<logger level="INFO" message="Flow2: processing finished" doc:name="Logger"/>
</sub-flow>
</mule>

Mule downloads e-mail body as well as attachment

How can I get Mule to download just attachments using pop3? I tried following the example at http://www.mulesoft.org/documentation/display/current/POP3+Transport+Reference as closely as possible, but I keep getting two files: one with the e-mail body and one with the actual attachment. Here's the flow I'm using:
<pop3:connector name="pop3Connector" checkFrequency="5000" doc:name="POP3"/>
<expression-transformer name="returnAttachments" doc:name="Expression">
<return-argument evaluator="attachments-list" expression="*" />
</expression-transformer>
<file:connector name="fileName" doc:name="File">
<file:expression-filename-parser/>
</file:connector>
<flow name="incoming-orders" doc:name="incoming-orders">
<pop3s:inbound-endpoint host="pop.gmail.com" port="995" user="myuser%40mydomain" password="mypassword" responseTimeout="10000" doc:name="POP3" transformer-refs="returnAttachments" />
<collection-splitter doc:name="Collection Splitter"/>
<file:outbound-endpoint path="C:/popthreetest" outputPattern="#[function:datestamp].dat" doc:name="File">
<expression-transformer>
<return-argument expression="payload.inputStream" evaluator="groovy" />
</expression-transformer>
</file:outbound-endpoint>
</flow>
Thanks!
edit:
Here's the final flow based on #David Dossot's answer. I have an added complexity in that I'm reading in a JSON file that specifies attachment names and an arbitrary destinations for the attachment. I included the replaceAll because I was getting an error about an invalid character in the path file:///C:\.
<pop3:connector name="pop3Connector" checkFrequency="5000" doc:name="POP3"/>
<expression-transformer name="returnAttachments" doc:name="Expression">
<return-argument evaluator="attachments-list" expression="*" />
</expression-transformer>
<file:connector name="DestinationsFileConnector" doc:name="File" autoDelete="false" streaming="true" validateConnections="true">
<file:expression-filename-parser/>
</file:connector>
<file:endpoint path="C:/popthreetest/" name="DestinationsFileEndpoint" responseTimeout="10000" doc:name="File" connector-ref="DestinationsFileConnector">
<file:filename-regex-filter pattern="destinations\.json" caseSensitive="true"/>
</file:endpoint>
<mulerequester:config name="DestinationsMuleRequestorConnector" doc:name="Mule Requester"/>
<flow name="incoming-orders" doc:name="incoming-orders">
<pop3s:inbound-endpoint host="pop.gmail.com" port="995" user="myusername%40mydomain" password="mypassword" responseTimeout="10000" doc:name="POP3" transformer-refs="returnAttachments" />
<collection-splitter doc:name="Collection Splitter"/>
<set-variable variableName="MessagePart" value="#[message.payload]" doc:name="MessagePart"/>
<logger message="Got #[message.payload.dataSource.name]." level="INFO" doc:name="Logger"/>
<mulerequester:request config-ref="DestinationsMuleRequestorConnector" resource="DestinationsFileEndpoint" doc:name="GetDestinations"/>
<json:json-to-object-transformer returnClass="java.util.HashMap" doc:name="JSON to Object"/>
<choice doc:name="Choice">
<when expression="#[message.payload.get(MessagePart.dataSource.name) != null]">
<set-payload value="#[message.payload.get(MessagePart.dataSource.name)]" doc:name="Destination List"/>
<foreach doc:name="For Each">
<logger message="Saving to #[message.payload]." level="INFO" doc:name="Logger"/>
<set-variable variableName="DestinationPath" value="#[java.nio.file.Paths.get(message.payload).getParent().toString().replaceAll('\\\\', '/')]" doc:name="DestinationPath"/>
<set-variable variableName="DestinationPattern" value="#[java.nio.file.Paths.get(message.payload).getFileName()]" doc:name="DestinationPattern"/>
<logger message="Saving to #[DestinationPattern] in #[DestinationPath]." level="INFO" doc:name="Logger"/>
<set-payload value="#[MessagePart]" doc:name="MessagePart"/>
<file:outbound-endpoint path="#[DestinationPath]" outputPattern="#[DestinationPattern]" doc:name="File">
<expression-transformer>
<return-argument expression="payload.inputStream" evaluator="groovy"/>
</expression-transformer>
</file:outbound-endpoint>
</foreach>
</when>
<otherwise>
<logger message="Did not find destination(s) for #[MessagePart.dataSource.name]." level="INFO" doc:name="Logger"/>
</otherwise>
</choice>
</flow>
For completeness, here's the JSON file:
{
"attachment-name.txt": [
"C:/popthreetest/firstDestination.txt"
, "C:/path/to/secondDestination.txt"
, "C:\\popthreetest\\destination\\using-backslashes.txt"
]
}
An email with attachment is a multi-part email into which attachments are parts but also the body. Hence Mule can only download "the whole package" and give the different parts to you.
You should be able to filter the body part after the collection-splitter based on the name of the part.
Alternatively, you could use a MEL expression to drop the first element of the collection, which is typically the body (Mule uses this technique internally to set the message payload: https://github.com/mulesoft/mule/blob/mule-3.x/transports/email/src/main/java/org/mule/transport/email/transformers/EmailMessageToString.java#L50 )