I guess that the problem may be the 'MULE_CORRELATION_ID'. I use the tag to deal a batch of datas. the configuration as follows:
<jdbc-ee:connector name="jdbcConnector" dataSource-ref="dataSource" pollingFrequency="2000" queryTimeout="-1" transactionPerMessage="false" resultSetHandler-ref="resutSetHandler" doc:name="Database">
<jdbc-ee:query key="read" value="SELECT 1 as CID, A.* from ewell_login_user A"></jdbc-ee:query>
</jdbc-ee:connector>
<file:connector name="output" outputAppend="true" outputPattern="#[function:datestamp].txt" doc:name="File"></file:connector>
<flow name="zhicall-zhongshan" doc:name="zhicall-zhongshan">
<jdbc-ee:inbound-endpoint queryKey="read" connector-ref="jdbcConnector" doc:name="Database">
</jdbc-ee:inbound-endpoint>
<message-properties-transformer doc:name="Message Properties" >
<add-message-property key="MULE_CORRELATION_GROUP_SIZE" value="10"/>
<add-message-property key="MULE_CORRELATION_ID" value="10000"/>
</message-properties-transformer>
<collection-aggregator timeout="5000" failOnTimeout="false" doc:name="Collection Aggregator"/>
<custom-transformer encoding="UTF-8" class="com.zhicall.esb.transformer.TestTransFormer" doc:name="Java"/>
<stdio:outbound-endpoint doc:name="STDIO" responseTimeout="10000" system="OUT" encoding="UTF-8"/>
</flow>
MULE_CORRELATION_ID must be unique for each group of messages.
You shouldn't be setting it and if you do, you need to make sure that you use a different id for each group.
Related
I read data from salesforce and split it using collection splitter. I want to aggregate all data into single list. First I count all records and I set that counter with MULE_CORRELATION_GROUP_SIZE. But my counter gives me wrong count..And my data didn't aggregate..
How can I solve this problem
Is my following flow right?
Following is my code
<flow name="davesalesforceFlow1Tmp" doc:name="davesalesforceFlow1Tmp">
<http:inbound-endpoint exchange-pattern="request-response" host="localhost" port="8080" doc:name="HTTP" path="test" contentType="text/plain" />
<sfdc:query config-ref="salesforce" doc:name="Salesforce" query="My query"/>
<collection-splitter doc:name="Collection Splitter"/>
<set-session-variable variableName="cnt"
value="#[org.mule.util.StringUtils.countMatches(message.payload, '\n') + org.mule.util.StringUtils.countMatches(message.payload, ',') - 1]" doc:name="Session Variable"/>
<set-property propertyName="MULE_CORRELATION_GROUP_SIZE"
value="#[sessionVars.cnt]" doc:name="Property"/>
<collection-aggregator failOnTimeout="true" doc:name="Collection Aggregator"/>
I also tried following code But in output got only single record
<flow name="davesalesforceFlow1Tmp" doc:name="davesalesforceFlow1Tmp">
<http:inbound-endpoint exchange-pattern="request-response" host="localhost" port="8080" doc:name="HTTP" path="test" contentType="text/plain" />
<sfdc:query config-ref="salesforce" doc:name="Salesforce" query="My query"/>
<collection-splitter doc:name="Collection Splitter"/>
<set-property propertyName="MULE_CORRELATION_GROUP_SIZE"
value="#[message.outboundProperties['MULE_CORRELATION_SEQUENCE']]" doc:name="Property"/>
<collection-aggregator failOnTimeout="true" doc:name="Collection Aggregator"/>
<logger message="MY PAYLOAD IS #[payload]" level="INFO" doc:name="Logger"/>
</flow>
</flow>
If you use the collection splitter and aggregator together theres no need to keep track of the count or anything.
Mule will take care of this via the MULE_CORRELATION_* header behind the scenes.
Only if you are dong for complex splits and aggregates may you need to modify these headers yourself.
<logger level="INFO" message="size before: #[payload.size()]" />
<collection-splitter />
<logger level="INFO" message="Individual item: #[payload]" />
<collection-aggregator />
<logger level="INFO" message="size after: #[payload.size()]" />
How can I get Mule to download just attachments using pop3? I tried following the example at http://www.mulesoft.org/documentation/display/current/POP3+Transport+Reference as closely as possible, but I keep getting two files: one with the e-mail body and one with the actual attachment. Here's the flow I'm using:
<pop3:connector name="pop3Connector" checkFrequency="5000" doc:name="POP3"/>
<expression-transformer name="returnAttachments" doc:name="Expression">
<return-argument evaluator="attachments-list" expression="*" />
</expression-transformer>
<file:connector name="fileName" doc:name="File">
<file:expression-filename-parser/>
</file:connector>
<flow name="incoming-orders" doc:name="incoming-orders">
<pop3s:inbound-endpoint host="pop.gmail.com" port="995" user="myuser%40mydomain" password="mypassword" responseTimeout="10000" doc:name="POP3" transformer-refs="returnAttachments" />
<collection-splitter doc:name="Collection Splitter"/>
<file:outbound-endpoint path="C:/popthreetest" outputPattern="#[function:datestamp].dat" doc:name="File">
<expression-transformer>
<return-argument expression="payload.inputStream" evaluator="groovy" />
</expression-transformer>
</file:outbound-endpoint>
</flow>
Thanks!
edit:
Here's the final flow based on #David Dossot's answer. I have an added complexity in that I'm reading in a JSON file that specifies attachment names and an arbitrary destinations for the attachment. I included the replaceAll because I was getting an error about an invalid character in the path file:///C:\.
<pop3:connector name="pop3Connector" checkFrequency="5000" doc:name="POP3"/>
<expression-transformer name="returnAttachments" doc:name="Expression">
<return-argument evaluator="attachments-list" expression="*" />
</expression-transformer>
<file:connector name="DestinationsFileConnector" doc:name="File" autoDelete="false" streaming="true" validateConnections="true">
<file:expression-filename-parser/>
</file:connector>
<file:endpoint path="C:/popthreetest/" name="DestinationsFileEndpoint" responseTimeout="10000" doc:name="File" connector-ref="DestinationsFileConnector">
<file:filename-regex-filter pattern="destinations\.json" caseSensitive="true"/>
</file:endpoint>
<mulerequester:config name="DestinationsMuleRequestorConnector" doc:name="Mule Requester"/>
<flow name="incoming-orders" doc:name="incoming-orders">
<pop3s:inbound-endpoint host="pop.gmail.com" port="995" user="myusername%40mydomain" password="mypassword" responseTimeout="10000" doc:name="POP3" transformer-refs="returnAttachments" />
<collection-splitter doc:name="Collection Splitter"/>
<set-variable variableName="MessagePart" value="#[message.payload]" doc:name="MessagePart"/>
<logger message="Got #[message.payload.dataSource.name]." level="INFO" doc:name="Logger"/>
<mulerequester:request config-ref="DestinationsMuleRequestorConnector" resource="DestinationsFileEndpoint" doc:name="GetDestinations"/>
<json:json-to-object-transformer returnClass="java.util.HashMap" doc:name="JSON to Object"/>
<choice doc:name="Choice">
<when expression="#[message.payload.get(MessagePart.dataSource.name) != null]">
<set-payload value="#[message.payload.get(MessagePart.dataSource.name)]" doc:name="Destination List"/>
<foreach doc:name="For Each">
<logger message="Saving to #[message.payload]." level="INFO" doc:name="Logger"/>
<set-variable variableName="DestinationPath" value="#[java.nio.file.Paths.get(message.payload).getParent().toString().replaceAll('\\\\', '/')]" doc:name="DestinationPath"/>
<set-variable variableName="DestinationPattern" value="#[java.nio.file.Paths.get(message.payload).getFileName()]" doc:name="DestinationPattern"/>
<logger message="Saving to #[DestinationPattern] in #[DestinationPath]." level="INFO" doc:name="Logger"/>
<set-payload value="#[MessagePart]" doc:name="MessagePart"/>
<file:outbound-endpoint path="#[DestinationPath]" outputPattern="#[DestinationPattern]" doc:name="File">
<expression-transformer>
<return-argument expression="payload.inputStream" evaluator="groovy"/>
</expression-transformer>
</file:outbound-endpoint>
</foreach>
</when>
<otherwise>
<logger message="Did not find destination(s) for #[MessagePart.dataSource.name]." level="INFO" doc:name="Logger"/>
</otherwise>
</choice>
</flow>
For completeness, here's the JSON file:
{
"attachment-name.txt": [
"C:/popthreetest/firstDestination.txt"
, "C:/path/to/secondDestination.txt"
, "C:\\popthreetest\\destination\\using-backslashes.txt"
]
}
An email with attachment is a multi-part email into which attachments are parts but also the body. Hence Mule can only download "the whole package" and give the different parts to you.
You should be able to filter the body part after the collection-splitter based on the name of the part.
Alternatively, you could use a MEL expression to drop the first element of the collection, which is typically the body (Mule uses this technique internally to set the message payload: https://github.com/mulesoft/mule/blob/mule-3.x/transports/email/src/main/java/org/mule/transport/email/transformers/EmailMessageToString.java#L50 )
I am stuck up with a problem with the request-reply router in mule. I am using the mule release 3.4.0. The request-reply is used in a fork-join pattern , i followed this article on the blog site http://blogs.mulesoft.org/aggregation-with-mule-fork-and-join-pattern/.
<request-reply>
<all enableCorrelation="ALWAYS">
<vm:outbound-endpoint path="PathA"/>
<vm:outbound-endpoint path="PathB"/>
</all>
<vm:inbound-endpoint path="FinalResponse">
<message-properties-transformer>
<add-message-property key="MULE_CORRELATION_GROUP_SIZE" value="2" />
</message-properties-transformer>
<collection-aggregator />
</vm:inbound-endpoint>
</request-reply>
The issue that i face is the processor placed after the request-reply is not reached.In mule i see 2 responses been sent to the FinalResponse vm endpoint. I tried placing my own custom aggregator in place of the collection-aggregator. But this is not reached. I am not sure how i can debug this issue. Please note that i have 2 such request-reply routers in two flows. Can someone help on this.
Adding the mule configuration. For abbrevity attaching only the parent flow and one of the child flow
ParentFlow
<flow name="ParentFlow" doc:name="ParentFlow" tracking:enable-default-events="true">
<file:inbound-endpoint path="${SourceFilePath}"
connector-ref="ParentFlowInboundConnector"
responseTimeout="10000" tracking:enable-default-events="true">
<file:filename-regex-filter pattern="${REGEX}" caseSensitive="false"/>
</file:inbound-endpoint>
<object-to-string-transformer doc:name="Object to String"/>
<message-properties-transformer
doc:name="Message Properties">
<add-message-property key="Source_System" value="XXX" />
<add-message-property key="publisherName" value="YYY" />
<add-message-property key="FlowName" value="ZZZ"/>
</message-properties-transformer>
<!-- This would store the message in the db -->
<wire-tap>
<vm:outbound-endpoint path="PERSITENCE.OUT"/>
</wire-tap>
<request-reply>
<all doc:name="Move the message to all endpoints" enableCorrelation="ALWAYS">
<vm:outbound-endpoint path="PathA"/>
<vm:outbound-endpoint path="PathB"/>
<vm:outbound-endpoint path="PathC"/>
<vm:outbound-endpoint path="PathD"/>
</all>
<vm:inbound-endpoint path="Response">
<message-properties-transformer>
<add-message-property key="MULE_CORRELATION_GROUP_SIZE" value="4" />
</message-properties-transformer>
<collection-aggregator failOnTimeout="false" doc:name="Collection Aggregator" />
</vm:inbound-endpoint>
</request-reply>
<component class="ProcessorA" />
<component class="ProcessorB" />
<exception-strategy ref="Exception Strategy" doc:name="Reference Exception Strategy"/>
</flow>
Child Flow
<flow name="PathA" doc:name="PathA">
<vm:inbound-endpoint path="PathA" >
<vm:transaction action="ALWAYS_BEGIN"/>
</vm:inbound-endpoint>
<message-properties-transformer doc:name="Message Properties">
<add-message-property key="originalFilename_processing" value="#[message.inboundProperties.originalFilename]" />
</message-properties-transformer>
<message-properties-transformer doc:name="Message Properties">
<add-message-property key="directory" value="#[message.inboundProperties.directory]" />
</message-properties-transformer>
<message-properties-transformer doc:name="Message Properties">
<add-message-property key="Destination_System" value="XXX" />
</message-properties-transformer>
<file:outbound-endpoint path="${OutPathA}" connector-ref="FileConnector_1"
tracking:enable-default-events="true"/>
<!--Send email alerts -->
<all doc:name="All" >
<processor-chain>
<message-property-filter pattern="turnOffEmail=false" caseSensitive="true" doc:name="Message Property"/>
<custom-transformer class="com.XXX.YYY.ZZZ.VelocityTemplateResolver" doc:name="Java">
<spring:property name="flowName" value="PPPP"/>
</custom-transformer>
<smtp:outbound-endpoint host="${SMTP_HOSTNAME}" port="${SMTP_PORT}" user="${SMTP_USERNAME}" password="${SMTP_PASSWORD}" to="${TO_ADDRESS}" from="${FROM_ADDRESS}" subject="${MAIL_SUBJECT}" cc="${MAIL_CC_ADDRESS}" bcc="${MAIL_BCC_ADDRESS}" responseTimeout="10000" transformer-refs="emailVelocityMessageTransformer" tracking:enable-default-events="true"/>
</processor-chain>
<processor-chain>
<message-property-filter pattern="turnOffEmail=false" caseSensitive="true" doc:name="Message Property"/>
<custom-transformer class="com.XXX.YYY.ZZZ.VelocityTemplateResolver" doc:name="Java">
<spring:property name="flowName" value="FlowName"/>
</custom-transformer>
<smtp:outbound-endpoint host="${SMTP_HOSTNAME}" port="${SMTP_PORT}" user="${SMTP_USERNAME}" password="${SMTP_PASSWORD}" to="${MAIL_TO_ADDRESS}" from="${MAIL_FROM_ADDRESS}" subject="${MAIL_SUBJECT}" cc="${MAIL_CC_ADDRESS}" bcc="${BCC_ADDRESS}" responseTimeout="10000" transformer-refs="emailVelocityMessageTransformer" tracking:enable-default-events="true"/>
</processor-chain>
<processor-chain>
<message-property-filter pattern="turnOffEmail=false" caseSensitive="true" doc:name="Message Property"/>
<custom-transformer class="com.XXX.YYY.ZZZ.VelocityTemplateResolver" doc:name="Java">
<spring:property name="flowName" value="AAAA"/>
</custom-transformer>
<smtp:outbound-endpoint host="${SMTP_HOSTNAME}" port="${SMTP_PORT}" user="${SMTP_USERNAME}" password="${SMTP_PASSWORD}" to="${MAIL_TO_ADDRESS}" from="${MAIL_FROM_ADDRESS}" subject="${MAIL_SUBJECT}" cc="${MAIL_CC_ADDRESS}" bcc="${MAIL_BCC_ADDRESS}" responseTimeout="10000" transformer-refs="emailVelocityMessageTransformer" tracking:enable-default-events="true"/>
</processor-chain>
</all>
<rollback-exception-strategy doc:name="Rollback Exception Strategy" maxRedeliveryAttempts="${MAX_REDELIVERY_ATTEMPTS}" enableNotifications="false">
<logger message="==Rollback log ===" level="INFO" doc:name="Logger"/>
</rollback-exception-strategy>
</flow>
The most likely cause of the the problem is correlation ID.
You might want to inspect the correlation ID of the messages sent out to Path A and Path B. The message received on FinalResponse VM inbound endpoint needs to have the same correlation ID, otherwise it won't be accepted as a response to the messages previously sent. In other words the request-reply will NOT accept just anything in the inbound-endpoint but just the message matching the correlation ID of the posted on outbound-endpoint with the request-reply router.
You can set the custom correlation ID using this --
<set-property propertyName="MULE_CORRELATION_ID" value="some-correlation-id" />
I'm trying to get my choice branch in Mule to branch on number of parameters in http.query.params is that possible?
How do I return the value of the MULE_JDBC_UDATE_COUNT variable to the client?
My config.xml
<spring:beans>
<spring:bean id="InformixDatasource" name="InformixDatasource" class="org.enhydra.jdbc.standard.StandardDataSource">
<spring:property name="password" value="xxxxx"></spring:property>
<spring:property name="driverName" value="com.informix.jdbc.IfxDriver"></spring:property>
<spring:property name="user" value="informix"></spring:property>
<spring:property name="url" value="jdbc:informix-sqli://browning.frett.ehf:1527/dev:informixserver=ol_brown"></spring:property>
</spring:bean>
</spring:beans>
<jdbc-ee:connector name="InformixConnector" dataSource-ref="InformixDatasource" validateConnections="true" queryTimeout="-1" pollingFrequency="0" doc:name="Database"></jdbc-ee:connector>
<flow name="postdrefing_fetch_dataFlow1" doc:name="postdrefing_fetch_dataFlow1">
<http:inbound-endpoint exchange-pattern="request-response" host="0.0.0.0" port="8084" doc:name="HTTP" path="postdreifing"></http:inbound-endpoint>
<logger message="before db Choice #[message.payload]" level="INFO" doc:name="Logger"></logger>
<choice doc:name="Choice">
<when expression="message.inboundProperties['dreifing'] != null and message.inboundProperties['dags'] != null or message.inboundProperties['dags'] != null">
<logger message="delete #[message.inboundProperties['http.query.params']]" level="INFO" doc:name="Logger"></logger>
<jdbc-ee:outbound-endpoint exchange-pattern="request-response" queryKey="deleteById" queryTimeout="-1" connector-ref="InformixConnector" doc:name="Delete from nafnadreifing by id">
<jdbc-ee:query key="deleteById" value="DELETE FROM fdr_nafnadreifing WHERE dreifing = #[message.inboundProperties['dreifing']] and dags = TO_DATE(#[message.inboundProperties['dags']],'%d%m%Y')"></jdbc-ee:query>
</jdbc-ee:outbound-endpoint>
<logger message="Deleted #[flowVars['MULE_JDBC_UDATE_COUNT']] rows" level="INFO" doc:name="Logger"></logger>
<set-payload value="Deleted #[flowVars['MULE_JDBC_UDATE_COUNT']] rows" doc:name="Set Payload"/>
</when>
<otherwise>
<logger message="default" level="INFO" doc:name="Logger"></logger>
<jdbc-ee:outbound-endpoint exchange-pattern="request-response" queryKey="selectStatement" queryTimeout="-1" connector-ref="InformixConnector" doc:name="Select all from ibudaskra">
<jdbc-ee:query key="selectStatement" value="SELECT skyring,gata,gata_thgf,husnumer,auka,pnr,hverfi,ibudafjoldi,fjolpostur,athugasemdir,breytt,bannmerki,uppfaert,sortnr,pakki,active,tegund,ibud FROM fdr_ibudaskra"></jdbc-ee:query>
</jdbc-ee:outbound-endpoint>
<json:object-to-json-transformer doc:name="Object to JSON"></json:object-to-json-transformer>
</otherwise>
</choice>
</flow>
Yes, this is possible, with something like #[message.inboundProperties['http.query.params']['myQueryParamName']]. Be sure to use #[ ] around your expression and also use '&&' and '||' for boolean operations.
<when expression="#[message.inboundProperties['dreifing'] != null && message.inboundProperties['dags'] != null || message.inboundProperties['dags'] != null]">
It also seems you're missing parenthesis around the part of this expression.
Use set-payload:
<set-payload value="#[flowVars['MULE_JDBC_UDPATE_COUNT']]" />
I have a back end service that I need to throttle access to. I'm trying to use the approach described here: http://blogs.mulesoft.org/synchronous-and-asynchronous-throttling-2/
I started with a simple pass through flow that receives a SOAP request and forwards it. When I hit this using the SOAPUI utility, I get the expected response in a second or two.
<http:connector name="httpConnector" doc:name="HTTP\HTTPS">
<receiver-threading-profile maxThreadsActive="1" maxBufferSize="100" />
</http:connector>
<jms:activemq-connector name="amqConnector" brokerURL="tcp://localhost:61616" specification="1.1" doc:name="AMQ" />
<flow name="Flow1" processingStrategy="synchronous" doc:name="Flow1">
<http:inbound-endpoint exchange-pattern="request-response"
host="localhost" port="8088" path="test" doc:name="HTTP"
mimeType="text/xml" encoding="UTF-8" connector-ref="httpConnector"/>
<http:outbound-endpoint
address="http://dnbdirect-api.dnb.com/DnBAPI-11"
exchange-pattern="request-response" doc:name="HTTP" mimeType="text/xml"/>
</flow>
If I then move the outbound call to a separate flow and add in the request-reply block, the behavior changes. I get no response back (nor do I get the "After queue" message from the logger) and SOAPUI eventually times out.
<http:connector name="httpConnector" doc:name="HTTP\HTTPS">
<receiver-threading-profile maxThreadsActive="1" maxBufferSize="100" />
</http:connector>
<jms:activemq-connector name="amqConnector" brokerURL="tcp://localhost:61616" specification="1.1" doc:name="AMQ" />
<flow name="Flow1" processingStrategy="synchronous" doc:name="Flow1">
<http:inbound-endpoint exchange-pattern="request-response"
host="localhost" port="8088" path="test" doc:name="HTTP"
mimeType="text/xml" encoding="UTF-8" connector-ref="httpConnector"/>
<message-properties-transformer doc:name="Message Properties">
<add-message-property key="AMQ_SCHEDULED_DELAY" value="5000"/>
</message-properties-transformer>
<logger message="Before queue" level="INFO"/>
<request-reply>
<jms:outbound-endpoint queue="request" connector-ref="amqConnector"></jms:outbound-endpoint>
<jms:inbound-endpoint queue="response" connector-ref="amqConnector"></jms:inbound-endpoint>
</request-reply>
<logger message="After queue" level="INFO"/>
</flow>
<flow name="flow2" doc:name="Flow2">
<jms:inbound-endpoint queue="request" connector-ref="amqConnector" doc:name="JMS"/>
<http:outbound-endpoint
address="http://dnbdirect-api.dnb.com/DnBAPI-11"
exchange-pattern="request-response" doc:name="HTTP" mimeType="text/xml" />
</flow>
The throttling behavior works as I see the delays if I pull out the call to the back end service. But I can't get it to work with the service call there.
What am I missing?
I found that the message's payload will be "ArrayList" after "request-reply".
so i add a java component to split it, then the result will be corrected.
#Override
public Object onCall(MuleEventContext eventContext) throws Exception {
MuleMessage message = eventContext.getMessage();
//int groupSize = message.getCorrelationGroupSize();
//System.out.println("############# correlationGroupSize: " + groupSize);
Object payload = message.getPayload();
if (payload != null && payload instanceof ArrayList) {
//message.setPayload(((ArrayList)payload).get(0));
return ((ArrayList)payload).get(0);
}
return message.getPayload();
}
the completed flow is:
<message-properties-transformer doc:name="Message Properties">
<add-message-property key="AMQ_SCHEDULED_DELAY" value="10000"/>
</message-properties-transformer>
<request-reply storePrefix="mainFlow">
<jms:outbound-endpoint queue="request" connector-ref="amqConnector" doc:name="JMS"></jms:outbound-endpoint>
<jms:inbound-endpoint queue="response" connector-ref="amqConnector" doc:name="JMS"></jms:inbound-endpoint>
</request-reply>
<component class="com.neusoft.fx.JmsMessageTransformer" doc:name="Java"/>
<message-properties-transformer doc:name="Set Content Type">
<delete-message-property key="Content-type" />
<add-message-property key="Content-Type" value="text/xml"/>
</message-properties-transformer>
<logger message="----- LOGGER ----- after #[groovy:message.toString()]" level="INFO" doc:name="Logger" />
</flow>
Try adding the following before the HTTP outbound endpoint in flow2:
<copy-properties propertyName="MULE_*"/>