I needed to split a message into 3 different payloads and transform and send to 3 routers. So the payload initially will have a header a body or detail and a footer. These 3 different payloads need to be extracted and send to 3 different routers. What would be the most efficient way to do it.
It depends on your body/payload type. If your payload is XML, you can easily split it using xpath and route it using content based routing similar to:
<splitter expression="#[xpath('//nodes/node)']" />
<choice>
<when expression="#[xpath('//node/id').text ='myid']">
<!-- Route somewhere -->
</when>
<otherwise>
<!-- Route somewhere else -->
</otherwise>
</choice>
The expression splitter above can take any MEL expression to split up your payload. There are many other splitters, for example if your payload is already a java Collection, you can simply use the collection-splitter.
Other splitter info can be found here: http://www.mulesoft.org/documentation-3.2/display/32X/Message+Splitting+and+Aggregation
Also there are other routers that can help you with fork and join patterns if you need to process messages asynchronously as well. Here's a good post on that: http://java.dzone.com/articles/aggregation-mule-%E2%80%93-%E2%80%9Cfork-and
Related
I am using the Mule 4.4 community edition on premise.
Thanks to help, I have been able to read a large file without consuming memory and processing it, which is all good (here).
Now building on this further - my use case is to read all .csv files from within a directory.
And then process them one by one:
\opt\out\
students.csv
teachers.csv
collesges.csv
....
So my plan was to list the files in the directory:
<sftp:list doc:name="List" config-ref="SFTP_Config" directoryPath="/opt/out">
<non-repeatable-iterable />
<sftp:matcher filenamePattern="#['*.csv' ]"
directories="EXCLUDE" symLinks="EXCLUDE" />
</sftp:list>
And then I wanted to only read file names from directory and not read payload.
As per this early access article we are advised to use <non-repeatable-iterable />. However, after the list file operation as per article when I try to extract attributes:
<set-payload doc:name="Set Payload" value="#[output application/json --- payload map $.attributes]"/>
No attributes are available... (my plan is to extract the file names and then run a for loop for each file name and then a choice condition to determine if file name has student, use student transformer, if teacher use teacher transformer, etc.)
However, as attributes are not available, I am not able to pass file names to the for loop (yet to be written).
So I changed from <non-repeatable-iterable /> to <repeatable-in-memory-iterable />
Code below:
<sftp:list doc:name="List" config-ref="SFTP_Config" directoryPath="/opt/out">
<repeatable-in-memory-iterable />
<sftp:matcher filenamePattern="#['*.csv' ]"
directories="EXCLUDE" symLinks="EXCLUDE" />
</sftp:list>
Using the above, I can extract the attributes of file names.
I am confused about the following:
The files to be processed in the above directory will be large (each file 700 MB), so while iterating the directory by using repeatable-in-memory-iterable, will it cause any memory issues? (I do not want to read file content, simply get file names at this stage)
Here is the complete payload till now (note - it does not contain any for loop to iterate over files, which I will plug in...)
<flow name="employee-process-flow">
<http:listener doc:name="Listener" config-ref="HTTP_Listener_config" path="/processFiles"/>
<set-variable value='#[now() as String { format: "ddMMuu" }]' doc:name="Set todays date as ddmmyy" doc:id="c6a91a41-65b1-46df-a720-9c13fe360b6b" variableName="today"/>
<sftp:list doc:name="List" config-ref="SFTP_Config" directoryPath="/opt/out">
<repeatable-in-memory-iterable />
<sftp:matcher filenamePattern="#['*.csv' ]"
directories="EXCLUDE" symLinks="EXCLUDE" />
</sftp:list>
<set-payload doc:name="Set Payload" value="#[output application/json --- payload map $.attributes]"/>
<foreach doc:name="For Each" >
<logger level="INFO" doc:name="Logger" message="we are here"/>
</foreach>
</flow>
The List operation returns a list of messages, and each has a payload and attributes. The content of the files is returned as the payload, in a lazy way, meaning that the file's content is read only if you try to access that element's payload.
It makes sense that if you a non-repeatable-iterator and don't access the payload of each item in the <foreach> then you should not have any memory issues, because the contents are not read.
By using in memory repeatable streaming it is possible that the entire payload is being read into memory. Try reading a file a few gigabytes in size and see what happens there.
I'm not sure what the problem is with the attributes. It should work the same in any streaming mode.
Note that if you plan on doing something with the attributes—other than printing them—then you should output to application/java instead of JSON, to avoid unneeded conversions to and from JSON. For example, in your flow the output is used as input for the <foreach>, so it would be better for it to be Java.
Example:
output application/java --- payload map $.attributes
I'm trying to connect to SQL Database with some configuration. But based on the input from the API we are supposed to hit different DB.
As of now, we have the code as
<choice doc:name="Check myFlag">
<when expression="#[flowVars.myFlag == 'true']">
<db:stored-procedure config-ref="Database_Configuration_1" doc:name="DB_config_1">
<db:dynamic-query><![CDATA[#[flowVars.callSPName]]]></db:dynamic-query>
</db:stored-procedure>
</when>
<otherwise>
<db:stored-procedure config-ref="Database_Configuration_2" doc:name="DB_config_2">
<db:dynamic-query><![CDATA[#[flowVars.callSPName]]]></db:dynamic-query>
</db:stored-procedure>
</otherwise>
</choice>
Instead of repeating <db:stored-procedure../> twice, is there a way where I can set a flow var with the DB config reference and use it?
Something like,
<db:stored-procedure config-ref="#[flowvars.db_config]" doc:name="DB_config_2">
<db:dynamic-query><![CDATA[#[flowVars.callSPName]]]></db:dynamic-query>
</db:stored-procedure>
In Mule 3, no. config-ref's are evaluated at application startup, not runtime.
In Mule 4 this is possible using Dynamic Configurations: https://docs.mulesoft.com/mule-sdk/1.1/static-dynamic-configs
Potential Mule 3 solutions documented here: https://help.mulesoft.com/s/article/How-to-configure-connector-with-dynamic-parameters
Put the DB call inside a sub-flow and call it from the rest of the flows with a <flow-ref>.
i want payload value like below in <http:outbound-endpoint>:
<set-variable variableName="url" value="#[json:url]" doc:name="Variable"></set-variable>
<set-variable variableName="payload" value="#[json:param]" doc:name="Variable"></set-variable>
<http:outbound-endpoint exchange-pattern="request-response" address="http://admin:admin##[url.substring(7)]" method="POST" doc:name="HTTP" password="admin" user="admin">
<set-payload value="action=start¶ms={input:#[payload]}&createTask=false&parts=all"/>
</http:outbound-endpoint>
but it is giving error.
thanks.
You can get rid of the error by replacing & with &
However, that is not your only problem.
payload is a reserved variable to hold the current payload. You can not use set-variable with variableName="payload", your setting will just be ignored when you call #[payload] later.
Your POST data would be something like action=start¶ms={input:yourdata}&createTask=false&parts=all. This is some kind of hybrid of JSON and HTTP GET syntax, and I doubt that this is what you are trying to achieve. If you want to send a POST request in key=val format, set a payload of type Map. If you want to send the request as JSON, set a payload with a JSON String, you have the object-to-json-transformer to help in Mule. If you want to have parameters in the URL, put them in the URL. But you can not mix these different syntaxes.
I have a requirement where I run xPath evaluator against xml payload in mule flow. This xPath evaluator can return single or multiple values. I need to store these values in flow variable and use somewhere later in the flow. Can someone help me implementing this changes?
Appreciate your help on this.
Thanks
For extracting values from an XML document, use the XPath extractor.
<mulexml:xpath-extractor-transformer expression="/a:my/b:xpath/text()"/>
You can also use Mule Expression Language to create dynamic XPath expressions:
<expression-transformer mimeType="text/xml" evaluator="xpath" expression="//school/day[#date= #[function:datestamp:yyyy-MM-dd] ]/name
"/>
However this can get somewhat messy for complicated expressions, so I have created my own dynamic XPath transformer:
<dx:dxpath expression="/b:team[name = $teamName]/b:player[b:name = $playerName]/b:goals/text()">
<dx:variable key="playerName" value="#[header:invocation:playerName]"/>
<dx:variable key="teamName" value="#[header:invocation:teamName]"/>
<!-- unlimited number of variables -->
</dx:dxpath>
which is somewhat more easy on the eyes.
Then Wrap your flow with an ericher:
<enricher target="#[variable:myData]">
<processor-chain>
<!-- your flow here -->
</processor-chain>
</enricher>
In mule flow I need to compare the last two params from an input string message
2012-04-30,2012-05-30,1,5
And if param1(1) <= param2(5), I need perform jms:outbound-endpoint.
Is there a standard choice element to do it? Could you please provide example.
While using mule 3.2.x your best option is to use the groovy evaluator and use a choice router that look like the following
<when expression="#[groovy:payload.split(',')[2] <= payload.split(',')[3] ]">
<processor-chain>
<jms:outbound-endpoint queue="out" doc:name="JMS"/>
</processor-chain>
</when>
Otherwise if you are willing to upgrade to Mule 3.3.0 you can take advantage of the new Mule Expression Language that allows to do the same without the need of the groovy operator
you can also use splitter and split on the basis of ',' and then have the compare logic for the params
You need not use Groovy for this. You can directly apply the split() function on payload.
<when expression="#[payload.split(',')[2]<=payload.split(',')[3]]">
<processor-chain doc:name="Processor Chain">
<jms:outbound-endpoint queue="data" connector-ref="Active_MQ"doc:name="JMS"/>
</processor-chain>
</when>
Apart from expression filter you can also use choice flow control. Capture the 2 values to be compared in a flowvars and use below expression in when condition of choice flow control.
[flowVars.param1 <= flowVars.param1]
Giving below the link for more information.
https://docs.mulesoft.com/mule-user-guide/v/3.7/choice-flow-control-reference
you can take advantage of the new Mule Expression Language that allows to do the same without the need of the groovy operator