I read data from salesforce and split it using collection splitter. I want to aggregate all data into single list. First I count all records and I set that counter with MULE_CORRELATION_GROUP_SIZE. But my counter gives me wrong count..And my data didn't aggregate..
How can I solve this problem
Is my following flow right?
Following is my code
<flow name="davesalesforceFlow1Tmp" doc:name="davesalesforceFlow1Tmp">
<http:inbound-endpoint exchange-pattern="request-response" host="localhost" port="8080" doc:name="HTTP" path="test" contentType="text/plain" />
<sfdc:query config-ref="salesforce" doc:name="Salesforce" query="My query"/>
<collection-splitter doc:name="Collection Splitter"/>
<set-session-variable variableName="cnt"
value="#[org.mule.util.StringUtils.countMatches(message.payload, '\n') + org.mule.util.StringUtils.countMatches(message.payload, ',') - 1]" doc:name="Session Variable"/>
<set-property propertyName="MULE_CORRELATION_GROUP_SIZE"
value="#[sessionVars.cnt]" doc:name="Property"/>
<collection-aggregator failOnTimeout="true" doc:name="Collection Aggregator"/>
I also tried following code But in output got only single record
<flow name="davesalesforceFlow1Tmp" doc:name="davesalesforceFlow1Tmp">
<http:inbound-endpoint exchange-pattern="request-response" host="localhost" port="8080" doc:name="HTTP" path="test" contentType="text/plain" />
<sfdc:query config-ref="salesforce" doc:name="Salesforce" query="My query"/>
<collection-splitter doc:name="Collection Splitter"/>
<set-property propertyName="MULE_CORRELATION_GROUP_SIZE"
value="#[message.outboundProperties['MULE_CORRELATION_SEQUENCE']]" doc:name="Property"/>
<collection-aggregator failOnTimeout="true" doc:name="Collection Aggregator"/>
<logger message="MY PAYLOAD IS #[payload]" level="INFO" doc:name="Logger"/>
</flow>
</flow>
If you use the collection splitter and aggregator together theres no need to keep track of the count or anything.
Mule will take care of this via the MULE_CORRELATION_* header behind the scenes.
Only if you are dong for complex splits and aggregates may you need to modify these headers yourself.
<logger level="INFO" message="size before: #[payload.size()]" />
<collection-splitter />
<logger level="INFO" message="Individual item: #[payload]" />
<collection-aggregator />
<logger level="INFO" message="size after: #[payload.size()]" />
Related
I want to move files to location D:\target dynamically from the query select filepath from emp where status='y':
This is my table:
emp_Name File Path File Name Status
ABC D:\emp abc.txt y
xyz D:\emp xyz.txt y
bcs D:\emp bcs.txt n
Following is my source code :
<flow name="testdbFlow1">
<http:listener config-ref="HTTP_Listener_Configuration"
path="/" doc:name="HTTP" />
<jdbc-ee:outbound-endpoint queryKey="allEmps"
queryTimeout="-1" connector-ref="JDBCConnector" exchange-pattern="request-response"
doc:name="Database" />
<foreach doc:name="Foreach">
<choice doc:name="Choice">
<when expression="#[payload.status == 'Y']">
<processor-chain doc:name="Processor Chain">
<set-variable variableName="filePath" value="#[payload.filepath]"
doc:name="Variable" />
<set-variable variableName="filename" value="#[payload.filename]"
doc:name="Variable" />
<logger message="#[filePath]" level="INFO"
doc:name="Logger" />
<logger message="#[filename]"
level="INFO" doc:name="Logger" />
<file:inbound-endpoint path="#[filePath]" name="input" doc:name="File"
pollingFrequency="12000" responseTimeout="10000"> <file:filename-wildcard-filter
pattern="#[filename]" /> </file:inbound-endpoint>
<file:outbound-endpoint name="output" path="D:\target" doc:name="File"/>
</processor-chain>
</when>
<otherwise>
<processor-chain doc:name="Processor Chain">
<set-variable variableName="filePath" value="#[payload.filepath]"
doc:name="Variable" />
<set-variable variableName="filename" value="#[payload.filename]"
doc:name="Variable" />
<logger message="#[filePath]" level="INFO"
doc:name="Logger" />
<logger message="#[filename]"
level="INFO" doc:name="Logger" />
</processor-chain>
</otherwise>
</choice>
</foreach>
</flow>
But it's not working.
Inbound doesn't work out of the box in the middle of the flow. I would suggest a simpler solution using Groovy because you know exactly the file you need to consume.
Instead of using file:inbound create a input stream manually:
<set-payload value="#[groovy: new java.io.FileInputStream(new java.io.File(filename))]"/>
You will have to delete the file with another groovy script after consumed.
<flow name="listobjects">
<http:inbound-endpoint exchange-pattern="request-response" host="localhost" port="8081" path="listobjects" contentType="text/plain" doc:name="HTTP"/>
<s3:list-objects config-ref="Amazon_S3" bucketName="demo" doc:name="Amazon S3" maxKeys="5" />
<!-- <payload-type-filter expectedType="java.util.List" doc:name="Payload"/> -->
<foreach collection="#[payload]" doc:name="For Each">
<!-- <foreach doc:name="For Each file"> -->
<logger message=" inside foreach...... #[payload.getKey()] ...." level="INFO" doc:name="Logger" />
<s3:get-object-content config-ref="Amazon_S3" bucketName="demo" key="#[payload.getKey()]" doc:name="Amazon S3"/>
<object-to-byte-array-transformer/>
<file:outbound-endpoint path="C:\output" responseTimeout="10000" doc:name="File" outputPattern="#[payload.getKey()] "></file:outbound-endpoint>
</foreach>
</flow>
I have bucket name called demo.
In that bucket I have 3 pdf files. I want to download all files and put it in c:\output folder.
I hit my url like http://localhost:8081/listobjects.
But I got the error:
Could not find a transformer to transform "CollectionDataType{type=org.mule.module.s3.simpleapi.SimpleAmazonS3AmazonDevKitImpl$S3ObjectSummaryIterable, itemType=com.amazonaws.services.s3.model.S3ObjectSummary, mimeType='/'}" to "SimpleDataType{type=org.mule.api.transport.OutputHandler, mimeType='/'}". (org.mule.api.transformer.TransformerException) (org.mule.api.transformer.TransformerException). Message payload is of type: SimpleAmazonS3AmazonDevKitImpl$S3ObjectSummaryIterable
The error occurs because after the foreach processor the payload is an instance of an S3 class, and you haven't specified any Content-Type to return. So Mule tries to transform the S3 instance to the default SimpleDataType and fails.
One way to solve it is simply to add something like
<set-property propertyName="Content-Type" value="application/json" doc:name="Content-Type" />
<set-payload value="{'result': 'ok'}"/>
at the end to make it explicit.
Also note that in your flow after running:
<object-to-byte-array-transformer/>
the S3 payload is gone, so #[payload.getKey()] will fail in the next processor:
<file:outbound-endpoint path="C:\output" responseTimeout="10000" doc:name="File" outputPattern="#[payload.getKey()] "></file:outbound-endpoint>
I've run this without problems:
<flow name="listobjects">
<http:inbound-endpoint exchange-pattern="request-response" host="localhost" port="8083" path="listobjects" contentType="text/plain" doc:name="HTTP"/>
<s3:list-objects config-ref="Amazon_S3" bucketName="mule_test" doc:name="Amazon S3" maxKeys="5" />
<foreach collection="#[payload]" doc:name="For Each">
<logger message=" inside foreach...... #[payload.getKey()] ...." level="INFO" doc:name="Logger" />
<set-variable variableName="fileKey" value="#[payload.getKey()]" doc:name="Variable" />
<s3:get-object-content config-ref="Amazon_S3" bucketName="#[payload.getBucketName()]" key="#[payload.getKey()]" doc:name="Amazon S3"/>
<object-to-byte-array-transformer/>
<file:outbound-endpoint path="/tmp" responseTimeout="10000" doc:name="File" outputPattern="#[flowVars.fileKey] "></file:outbound-endpoint>
</foreach>
<set-property propertyName="Content-Type" value="application/json" doc:name="Content-Type" />
<set-payload value="{'result': 'ok'}"/>
</flow>
Suppose I have a main flow and there are several sub-flows. I need to call a particular sub-flow if I have the right parameter without using choice component.
Basically, I want to ask if sub-flows can be parametrized.
Yes you can. Here's an example:
<flow name="testFlow1" doc:name="testFlow1">
<poll doc:name="Poll">
<logger level="INFO" doc:name="Logger" />
</poll>
<set-variable variableName="flowName" value="testFlow2"
doc:name="Variable" />
<flow-ref name="#[flowVars.flowName]" />
</flow>
<sub-flow name="testFlow2" doc:name="testFlow2">
<logger level="INFO" doc:name="Logger" message="sub" />
</sub-flow>
How can I get Mule to download just attachments using pop3? I tried following the example at http://www.mulesoft.org/documentation/display/current/POP3+Transport+Reference as closely as possible, but I keep getting two files: one with the e-mail body and one with the actual attachment. Here's the flow I'm using:
<pop3:connector name="pop3Connector" checkFrequency="5000" doc:name="POP3"/>
<expression-transformer name="returnAttachments" doc:name="Expression">
<return-argument evaluator="attachments-list" expression="*" />
</expression-transformer>
<file:connector name="fileName" doc:name="File">
<file:expression-filename-parser/>
</file:connector>
<flow name="incoming-orders" doc:name="incoming-orders">
<pop3s:inbound-endpoint host="pop.gmail.com" port="995" user="myuser%40mydomain" password="mypassword" responseTimeout="10000" doc:name="POP3" transformer-refs="returnAttachments" />
<collection-splitter doc:name="Collection Splitter"/>
<file:outbound-endpoint path="C:/popthreetest" outputPattern="#[function:datestamp].dat" doc:name="File">
<expression-transformer>
<return-argument expression="payload.inputStream" evaluator="groovy" />
</expression-transformer>
</file:outbound-endpoint>
</flow>
Thanks!
edit:
Here's the final flow based on #David Dossot's answer. I have an added complexity in that I'm reading in a JSON file that specifies attachment names and an arbitrary destinations for the attachment. I included the replaceAll because I was getting an error about an invalid character in the path file:///C:\.
<pop3:connector name="pop3Connector" checkFrequency="5000" doc:name="POP3"/>
<expression-transformer name="returnAttachments" doc:name="Expression">
<return-argument evaluator="attachments-list" expression="*" />
</expression-transformer>
<file:connector name="DestinationsFileConnector" doc:name="File" autoDelete="false" streaming="true" validateConnections="true">
<file:expression-filename-parser/>
</file:connector>
<file:endpoint path="C:/popthreetest/" name="DestinationsFileEndpoint" responseTimeout="10000" doc:name="File" connector-ref="DestinationsFileConnector">
<file:filename-regex-filter pattern="destinations\.json" caseSensitive="true"/>
</file:endpoint>
<mulerequester:config name="DestinationsMuleRequestorConnector" doc:name="Mule Requester"/>
<flow name="incoming-orders" doc:name="incoming-orders">
<pop3s:inbound-endpoint host="pop.gmail.com" port="995" user="myusername%40mydomain" password="mypassword" responseTimeout="10000" doc:name="POP3" transformer-refs="returnAttachments" />
<collection-splitter doc:name="Collection Splitter"/>
<set-variable variableName="MessagePart" value="#[message.payload]" doc:name="MessagePart"/>
<logger message="Got #[message.payload.dataSource.name]." level="INFO" doc:name="Logger"/>
<mulerequester:request config-ref="DestinationsMuleRequestorConnector" resource="DestinationsFileEndpoint" doc:name="GetDestinations"/>
<json:json-to-object-transformer returnClass="java.util.HashMap" doc:name="JSON to Object"/>
<choice doc:name="Choice">
<when expression="#[message.payload.get(MessagePart.dataSource.name) != null]">
<set-payload value="#[message.payload.get(MessagePart.dataSource.name)]" doc:name="Destination List"/>
<foreach doc:name="For Each">
<logger message="Saving to #[message.payload]." level="INFO" doc:name="Logger"/>
<set-variable variableName="DestinationPath" value="#[java.nio.file.Paths.get(message.payload).getParent().toString().replaceAll('\\\\', '/')]" doc:name="DestinationPath"/>
<set-variable variableName="DestinationPattern" value="#[java.nio.file.Paths.get(message.payload).getFileName()]" doc:name="DestinationPattern"/>
<logger message="Saving to #[DestinationPattern] in #[DestinationPath]." level="INFO" doc:name="Logger"/>
<set-payload value="#[MessagePart]" doc:name="MessagePart"/>
<file:outbound-endpoint path="#[DestinationPath]" outputPattern="#[DestinationPattern]" doc:name="File">
<expression-transformer>
<return-argument expression="payload.inputStream" evaluator="groovy"/>
</expression-transformer>
</file:outbound-endpoint>
</foreach>
</when>
<otherwise>
<logger message="Did not find destination(s) for #[MessagePart.dataSource.name]." level="INFO" doc:name="Logger"/>
</otherwise>
</choice>
</flow>
For completeness, here's the JSON file:
{
"attachment-name.txt": [
"C:/popthreetest/firstDestination.txt"
, "C:/path/to/secondDestination.txt"
, "C:\\popthreetest\\destination\\using-backslashes.txt"
]
}
An email with attachment is a multi-part email into which attachments are parts but also the body. Hence Mule can only download "the whole package" and give the different parts to you.
You should be able to filter the body part after the collection-splitter based on the name of the part.
Alternatively, you could use a MEL expression to drop the first element of the collection, which is typically the body (Mule uses this technique internally to set the message payload: https://github.com/mulesoft/mule/blob/mule-3.x/transports/email/src/main/java/org/mule/transport/email/transformers/EmailMessageToString.java#L50 )
Requirement is to develop mule flow which calls 3 different sync services in parallel and then aggregate the response of each of these and send it back to caller.
I have followed fork join approach as mentioned in docs and How to make parallel outbound calls.
My config file looks like below :
<flow name="fork">
<http:inbound-endpoint host="localhost" port="8090" path="mainPath" exchange-pattern="request-response">
<set-property propertyName="MULE_CORRELATION_GROUP_SIZE"
value="2" />
<all enableCorrelation="IF_NOT_SET">
<async>
<set-property propertyName="MULE_CORRELATION_SEQUENCE"
value="1" />
<flow-ref name="parallel1" />
</async>
<async>
<set-property propertyName="MULE_CORRELATION_SEQUENCE"
value="2" />
<flow-ref name="parallel2" />
</async>
</all>
</flow>
<sub-flow name="parallel1">
<logger level="INFO" message="parallel1: processing started" />
<!- Transformation payload -->
<http:outbound-endpoint address="..."
exchange-pattern="request-response" />
<logger level="INFO" message="parallel1: processing finished" />
<flow-ref name="join" />
</sub-flow>
<sub-flow name="parallel2">
<logger level="INFO" message="parallel2: processing started" />
<!- Transformation payload -->
<http:outbound-endpoint address="..."
exchange-pattern="request-response" />
<logger level="INFO" message="parallel2: processing finished" />
<flow-ref name="join" />
</sub-flow>
<sub-flow name="join">
<collection-aggregator timeout="6000"
failOnTimeout="true" />
<combine-collections-transformer />
<logger level="INFO" message="Continuing processing of: #[message.payloadAs(java.lang.String)]" />
<set-payload value="Soap XML Response"/>
</sub-flow>
I am able to verify that till "join" subflow is working fine but the response is not coming back as "Soap XML Response".
The response is the same initial SOAP Request.
How can I make this thread wait till sub-flow processing is complete and it sends back response whatever "join" sub-flow returns ??
The fork join in the above post looks good. The issue here is there is no way to capture the payload after the join and bring it back to the main flow.
As the calls to parallel made async the main flow continues without waiting for the join output.
I have modified the flow to address this issue. Now the flow will have a processor to wait for the reply and read the joined output to be written onto the http transformer.
<flow name="fork">
<http:inbound-endpoint host="localhost" port="8090" path="mainPath" exchange-pattern="request-response">
<!-- To get back the response after the fork-join -->
<request-reply timeout="60000">
<jms:outbound-endpoint queue="parallel.processor.queue">
<message-properties-transformer scope="outbound">
<delete-message-property key="MULE_REPLYTO" />
</message-properties-transformer>
</jms:outbound-endpoint>
<jms:inbound-endpoint queue="join.queue" >
</jms:inbound-endpoint>
</request-reply>
</flow>
<flow name="fork_join_flow" >
<jms:inbound-endpoint queue="parallel.processor.queue" exchange-pattern="one-way" />
<set-property propertyName="MULE_CORRELATION_GROUP_SIZE"
value="2" />
<all enableCorrelation="IF_NOT_SET">
<async>
<set-property propertyName="MULE_CORRELATION_SEQUENCE"
value="1" />
<flow-ref name="parallel1" />
</async>
<async>
<set-property propertyName="MULE_CORRELATION_SEQUENCE"
value="2" />
<flow-ref name="parallel2" />
</async>
</all>
</flow>
<sub-flow name="parallel1">
<logger level="INFO" message="parallel1: processing started" />
<!- Transformation payload -->
<http:outbound-endpoint address="..."
exchange-pattern="request-response" />
<logger level="INFO" message="parallel1: processing finished" />
<flow-ref name="join" />
</sub-flow>
<sub-flow name="parallel2">
<logger level="INFO" message="parallel2: processing started" />
<!- Transformation payload -->
<http:outbound-endpoint address="..."
exchange-pattern="request-response" />
<logger level="INFO" message="parallel2: processing finished" />
<flow-ref name="join" />
</sub-flow>
<sub-flow name="join">
<collection-aggregator timeout="6000"
failOnTimeout="true" />
<combine-collections-transformer />
<logger level="INFO" message="Continuing processing of: #[message.payloadAs(java.lang.String)]" />
<set-payload value="Soap XML Response"/>
<jms:outbound-endpoint queue="join.queue">
</jms:outbound-endpoint>
</sub-flow>
Hope this helps.