How can I do the equivalent of a SQL join in Dataweave? I'm basically implementing a lookup. I have CSV data and JSON data in flow variables. I'm able to use both of them, but I can't figure out how connect them. Let's say the CSV has two columns:
Name,ExternalId
Foo,1
Bar,2
Baz,2
The JSON data is:
{
ExternalEntities: [
{ "Id": 1, "Name": "One", "Description": "Thing One" }
, { "Id": 2, "Name": "Two", "Description": "Thing Two" }
]
}
In the end, I'd like a List<Hashmap> with the following (expressed here in JSON format for convenience.)
[
{ "Name": "Foo", "ExternalName": "One", "ExternalDescription": "Thing One" }
, { "Name": "Bar", "ExternalName": "Two", "ExternalDescription": "Thing Two" }
, { "Name": "Baz", "ExternalName": "Two", "ExternalDescription": "Thing Two" }
]
So here is a sample using the json as the payload and the csv as a lookup table. The first flows parses the json and does a flow lookup passing the 'Id' as the payload. The second flow loads the csv as an array and does a search by the Id passed in. This returns a map of the CSV record found where you can then extract the 'Name' field.
It is possible to do this in one transformer, but for demo sake this was easier. You can also reverse the lookup with some tinkering if you want the csv as your payload and the json as your lookup.
<flow name="lookuptest" processingStrategy="synchronous">
<poll doc:name="Poll">
<logger level="INFO" doc:name="Logger" />
</poll>
<set-payload
value="{ "ExternalEntities": [ { "Id": 1, "Name": "One", "Description": "Thing One" } , { "Id": 2, "Name": "Two", "Description": "Thing Two" } ] }"
doc:name="Set Payload" mimeType="application/json"></set-payload>
<dw:transform-message doc:name="Transform Message">
<dw:input-payload doc:sample="string_2.dwl" />
<dw:set-payload><![CDATA[%dw 1.0
%input payload application/json
%output application/json
---
payload.ExternalEntities map ((value , index) -> {
Name: lookup("NameLookup", [value.Id as :string]).Name
})]]></dw:set-payload>
</dw:transform-message>
<object-to-string-transformer doc:name="Object to String"/>
<logger level="ERROR" message="#[payload]" doc:name="Logger"/>
</flow>
<flow name="NameLookup" processingStrategy="synchronous">
<set-variable value="#[payload[0]]" variableName="ExternalId"
doc:name="Variable" />
<set-payload
value="#[Thread.currentThread().getContextClassLoader().getResourceAsStream('lookuptables/namelookup.csv')]"
mimeType="application/csv" doc:name="Variable" />
<dw:transform-message doc:name="Transform Message">
<dw:input-payload doc:sample="string_2.dwl" />
<dw:set-payload><![CDATA[%dw 1.0
%input payload application/csv
%output application/java
---
(payload[?($.ExternalId == flowVars['ExternalId'])])[0]
]]></dw:set-payload>
</dw:transform-message>
</flow>
Here's a second solution that works, though I accepted #Ryan Carter's answer.
%dw 1.0
%output application/json
---
flowVars.myCsv map (
(row0, i) -> using (lookupElement = (flowVars.jsonLookup.ExternalEntities filter ((obj1) -> row0.ExternalId ~= obj1.Id))[0]) {
Name: row0.Name
, ExternalName: lookupElement.Name
, ExternalDescription: lookupElement.Description
} mapObject ({"$$": $, (StartsWithB:$[0] == "B") when $$ ~= "Name"})
)
Related
I used to convert payload to string whenever i need to log the payload on console using the syntax in mule 3 [message.payloadAs(java.lang.String)] and some times used to use object to string converter or object to array converter.
But in mule 4 i don't see any converter as in mule 3 and [message.payloadAs(java.lang.String)] is alos not working.
Please assist me any new syntax of component or consept has come in mule 4 ?
Input
[
{
"firstName": "Ram",
"lastName": "Ram1",
"address": [
{
"street": "a",
"city": "b",
"state": "c"
},
{
"street": "a1",
"city": "b1",
"state": "c1"
}
]
}
Batch Image
Code
<batch:job jobName="Batch_ProcessingBatch_Job" doc:id="2582eccb-720f-499f-99fe-424cab3c9a33" >
<batch:process-records >
<batch:step name="Batch_Step1" doc:id="cb045f6b-edd3-426f-8f28-0d20b301fb6d" >
<logger level="INFO" doc:name="Logger" doc:id="2491184f-e8d7-47b7-bef0-29bafd43247f" message="In Batch_step1........... #[payload]"/>
<batch:aggregator doc:name="Batch Aggregator" doc:id="aa9c1b60-395f-4376-804d-bced200dfb43" size="2">
<set-payload value='#[write(payload, "application/json")]' doc:name="Set Payload" doc:id="9d4b27ee-5692-4a99-a93e-bd7c694f398d" />
<ee:transform doc:name="Transform Message" doc:id="06064422-1352-4ff9-8817-3e87eb1f4264" >
<ee:message >
<ee:set-payload ><![CDATA[%dw 2.0
output application/json
---
payload map(item,index) -> {
name: item.firstName,
surname: item.lastName
}]]></ee:set-payload>
</ee:message>
</ee:transform>
<logger level="INFO" doc:name="Logger" doc:id="c955490e-fab1-436e-8265-f706a577c9d2" message="In Batch_step1 Aggrigator........... #[payload]" />
</batch:aggregator>
</batch:step>
</batch:process-records>
<batch:on-complete >
<logger level="INFO" doc:name="Logger" doc:id="a3923c90-557b-4955-9c40-afd748719abd" message="#['#####It is oncomplete result####': write(payload, "application/json")]"/>
</batch:on-complete>
</batch:job>
Getting Error
Message : "You called the function 'map' with these arguments:
1: String ("[\n \"{\\n \\\"firstName\\\": \\\"Ram\\\",\\n \\\"lastName\\\": \\\"Ram1\...)
2: Function ((item:Any, index:Any) -> ???)
But it expects arguments of these types:
1: Array
2: Function
4| payload map(item,index) -> {
| ...
7| }
Trace:
at map (line: 4, column: 1)
at main (line: 4, column: 9)" evaluating expression: "%dw 2.0
output application/json
---
payload map(item,index) -> {
name: item.firstName,
surname: item.lastName
}".
Error type : MULE:EXPRESSION
Element : Batch_ProcessingFlow/processors/0/route/0/route/0/aggregator/processors/1 # training-1:Batch Processing.xml:27 (Transform Message)
Element XML : <ee:transform doc:name="Transform Message" doc:id="06064422-1352-4ff9-8817-3e87eb1f4264">
<ee:message>
<ee:set-payload>%dw 2.0
output application/json
---
payload map(item,index) -> {
name: item.firstName,
surname: item.lastName
}</ee:set-payload>
</ee:message>
</ee:transform>
(set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************
After quick googling the new way is to use #[message.payload as String] as shown in link.
You can use the payload.^raw syntax for the loggers. It was introduced in mule 4 and can handles cases where string conversion is needed. Typically pairs well with output text/plain format in a logger.
Ref: https://docs.mulesoft.com/mule-runtime/4.3/dataweave-cookbook-extract-data#metadata_selector
NOTE its never a good idea to log entire payloads (that could contain PII etc..) with maybe an exception where you are building a report that you need to log at the end of your batch process which possibly could be the case here...
Want to extract each object from my json payload which looks like this: Need to pass each object from this map into splitter or collection splitter. I cannot use foreach scope here.
[
{
"Name": null,
"Key": "4",
"Header": {
"Id": "14"
}
},
{
"Name": null,
"Key": "5",
"Header": {
"Id": "15"
}
}
]
To get the entire object from the array we can use simple #[payload] and this will take care just like for each scope.
If you want to parse and access the JSON elements then
**<json:json-to-object-transformer/>** is the way to do the trick.
java.util.HashMap or java.util.List or java.util.Map
depends on the type of your JSON data.
You can also use java.lang.Object
There are multiple ways to do this :
1) As mentioned by Vijay, use json-to-object transformer to map input json to any Collection type e.g.
<json:json-to-object-transformer returnClass="java.util.List" doc:name="JSON to Object"/>
<collection-splitter doc:name="Collection Splitter"/>
2) Use dataweave to get a Collection from input json data :
<dw:transform-message doc:name="Transform Message">
<dw:set-payload><![CDATA[%dw 1.0
%output application/java
---
payload
]]></dw:set-payload>
</dw:transform-message>
<collection-splitter doc:name="Collection Splitter"/>
I am integrating data from Salesforce to another API. The source has 1 object with related child object. However target API needs to take form like below. Each related Salesforce record in child object needs node within subforms. The POST example below is the ultimate desired form for the payload.
I suppose I need to recurse using DataWeave utilizing a For Each with each child record from Salesforce. But I'm not quite sure where to start with that.
Here is the incoming JSON from Salesforce post topic:
{
"LastModifiedDate": "2016-09-22T14:19:08.000+0000",
"Engagement__c": "1234",
"Product_CoC__c": null,
"Name": "Prod-22880",
"Year_1_Approved_Production_Hectares__c": 100.0,
"Year_1_Approved_Production_Volume__c": 1000.0,
"Product_Variety__c": null,
"CreatedById": "1234",
"Year_2_Estimated_Production_Volume__c": 1001.0,
"Product__c": "1234",
"Engagement_Decision_Date__c": null,
"Year_1_Harvest_Period__c": "2016",
"CPH_Number__c": null,
"Product_Type__c": null,
"Certificate_Product__c": "1234",
"IsDeleted": false,
"Crop_Yield__c": 10.0,
"Year_2_Estimated_Production_Hectares__c": 100.0,
"Valid_Period_End_Date__c": null,
"Intercrop__c": false,
"SystemModstamp": "2016-09-22T14:19:08.000+0000",
"Year_2_Harvest_Period__c": null,
"Crop_Production_Unit__c": "kg",
"Certificate__c": "1234",
"Year__c": 1.0,
"Harvest_Frequency__c": "Single",
"CreatedDate": "2016-07-28T15:44:18.000+0000",
"Valid_Period_Start_Date__c": null,
"Id": "1234",
"LastModifiedById": "1234",
"RA_Id__c": null
}
Here is the Transform Message I have for the child object:
%dw 1.0
%output application/java
---
{
acv_crop3:flowVars.ChainpointCropID default "",
acv_cph3:payload.Name default "",
acv_harvest_period2:payload.Year_1_Harvest_Period__c default "",
acv_frequency2:flowVars.frequency default "",
acv_aq2:payload.Year_1_Approved_Production_Volume__c default "",
acv_cph4:payload.Year_1_Approved_Production_Hectares__c default "",
acv_yield2:payload.Crop_Yield__c default "",
acv_start_date2:payload.Valid_Period_Start_Date__c default "",
acv_end_date2:payload.Valid_Period_End_Date__c default "",
acv_uom2:flowVars.uom default ""
}
Here is an example of a desired POST payload:
{
"ownerId": 24695,
"Q_CERT_SFGUID": "a00G000000WR3xxxxx",
"Q_CERT": "467",
"Q_CERTCROP2": [
"1199"
],
"textId": "auto",
"resultDate": null,
"authorId": null,
"Q_CERTNO": "RA-G-",
"Q_CERTHL": 24695,
"Q_CERTCB": 43986,
"Q_CERTST": "NO_STATUS",
"Q_CERTSUB": "",
"Q_CERTSTDATE": "2016-06-30",
"Q_CERT_expiration_date": "2019-06-29",
"Q_CERTENDATE": "",
"Q_CERT_redate": "2016-06-30",
"Q_CERT_CHA": "8.0",
"Q_CERT_TPH": "8.0",
"Q_CERT_CONTACT_EMAIL": "m#gmail.com",
"Q_CERT_CONTACT_NAME": 1300,
"subforms": [
{
"textId": "acv2",
"rows": [
{
"acv_crop3": "1199",
"acv_cph3": "Prod-22880",
"acv_harvest_period2": "2016",
"acv_frequency2": 668,
"acv_aq2": 1000,
"acv_cph4": 100,
"acv_yield2": 10,
"acv_start_date2": "2099-12-31",
"acv_end_date2": "2099-12-31",
"acv_uom2": 557
},
{
"acv_crop3": "1100",
"acv_cph3": "Prod-22881",
"acv_harvest_period2": "2017",
"acv_frequency2": 668,
"acv_aq2": 1000,
"acv_cph4": 100,
"acv_yield2": 1000,
"acv_start_date2": "2017-12-31",
"acv_end_date2": "2018-12-31",
"acv_uom2": 557
}
]
}
]
}
Here is the flow I have so far with the Transform message...
<flow name="prepareEP">
<set-variable variableName="API_Path" value="v0/forms/FR_CERT" doc:name="engagementProduct API_URL"/>
<set-variable variableName="entityType" value="engagementProduct" doc:name="set Entity"/>
<component class="com.ww.mule.reporting.et.ObjectStoreController" doc:name="load countryDiccionary"/>
<set-variable variableName="originalPayloadEP" value="#[payload]" doc:name="original payload"/>
<json:json-to-object-transformer returnClass="java.util.Map" doc:name="JSON to Object"/>
<set-variable variableName="originalPayloadEPMap" value="#[payload]" doc:name="original payload map"/>
<logger message="#['About to query certificate ' + flowVars.originalPayloadEPMap.Certificate__c + ' with engagement product Id ' + flowVars.originalPayloadEPMap.Id]" level="INFO" doc:name="Logger"/>
<sfdc:query config-ref="Salesforce__Basic_Authentication" doc:name="Get Engagement Products By Cert" query="dsql:SELECT Id,Name,Year_1_Harvest_Period__c,Harvest_Frequency__c,Year_1_Approved_Production_Volume__c,Year_1_Approved_Production_Hectares__c,Crop_Yield__c,Valid_Period_Start_Date__c,Valid_Period_End_Date__c FROM Engagement_Product__c WHERE Certificate__c = '#[flowVars.originalPayloadEPMap.Certificate__c]'"/>
<foreach doc:name="For Each">
<json:object-to-json-transformer doc:name="Object to JSON"/>
<objectstore:retrieve config-ref="CountryStoreObjectStore" key="#[payload.Cert_Country__c == empty?'UNKNOWN_COUNTRY':payload.Cert_Country__c]" targetProperty="countryISO" doc:name="ObjectStore" defaultValue-ref="#['XX']"/>
<set-payload value="#[flowVars.originalPayloadEP]" mimeType="application/json" doc:name="Set Payload"/>
<set-variable variableName="frequency" value="#[originalPayloadEPMap.Harvest_Frequency__c == 'Single' ? 668 : originalPayloadEPMap.Harvest_Frequency__c == 'Multiple' ? 669 : originalPayloadEPMap.Harvest_Frequency__c == 'Permanent/Continuous' ? 667 : originalPayloadEPMap.Harvest_Frequency__c == 'Occasional' ? 1910 : null]" doc:name="Set Harvest Frequency Variable"/>
<set-variable variableName="uom" value="#[originalPayloadEPMap.Crop_Production_Unit__c == 'kg' ? 557 : originalPayloadEPMap.Crop_Production_Unit__c == 'Liters' ? 596 : originalPayloadEPMap.Crop_Production_Unit__c == 'Number of Stems' ? 1802 : originalPayloadEPMap.Crop_Production_Unit__c == 'mts2' ? 556 : null]" doc:name="Set UOM Variable"/>
<!--
<set-payload doc:name="Set Payload"/>
-->
<dw:transform-message doc:name="Transform Message">
<dw:set-payload><![CDATA[%dw 1.0
%output application/java
---
{
acv_crop3:flowVars.ChainpointCropID default "",
acv_cph3:payload.Name default "",
acv_harvest_period2:payload.Year_1_Harvest_Period__c default "",
acv_frequency2:flowVars.frequency default "",
acv_aq2:payload.Year_1_Approved_Production_Volume__c default "",
acv_cph4:payload.Year_1_Approved_Production_Hectares__c default "",
acv_yield2:payload.Crop_Yield__c default "",
acv_start_date2:payload.Valid_Period_Start_Date__c default "",
acv_end_date2:payload.Valid_Period_End_Date__c default "",
acv_uom2:flowVars.uom default ""
}]]>
</dw:set-payload>
</dw:transform-message>
<object-to-string-transformer mimeType="application/json" doc:name="Object to String"/>
<set-variable variableName="acv2_info" value="#[flowVars.acv2_info == null ? payload : flowVars.acv2_info + ',' + payload]" doc:name="acv2_info"/>
<logger message="#['acv2_info = ' + flowVars.acv2_info]" level="INFO" doc:name="Logger"/>
<set-variable variableName="transformedPayload" value="#[payload]" doc:name="save payload"/>
<logger message="#['transformedPayload = ' + flowVars.transformedPayload]" level="INFO" doc:name="Print transformedPayload"/>
</foreach>
<flow-ref name="sendAuthenticatedChainPointRequest" doc:name="sendAuthenticatedChainPointRequest"/>
</flow>
I want to convert DB records into XML. Select query from DB records is giving LinkedHashMap as payload. Now I want to use dataweave to transform this into XML. I am pretty new to mule so a bit lost. What all steps do I need to follow to do this?
Thank you.
As informed by Victor please go through Dataweave Documentation which is the Data transformation component in Mule.
For you question the below methodology will help you to transform. The below script is an example that you can refer to be added to the data weave component in the mule code.
%dw 1.0
%output application/xml
---
payload map {
element: $.columnName_Or_aliasName
}
Map function will iterate over an array or complex xml or array in JSON.
Here for your DB response you can consider as every row in your DB resulset will be iterated
Transform Java Map to XML using DataWeave :
You can try below POC :
<sub-flow name="testdataweaveSub_Flow16_MapToXml" doc:description="http://blogs.mulesoft.com/dev/getting-started-with-dataweave-part-1/ Literal Expressions Variable Reference Expressions">
<set-payload value="{ "item_id": "B0002345W45", "item_type": "Item Type 1", "item_type_name": "Item Type 1 Name", "item_name": "item 1 name", "item_summary": "item 1 summary", "item_brand": "Brand 1", "image_type_name": "SmallImage", "url": "http://a/b/c" }" mimeType="application/json" doc:name="Set JSON Payload"/>
<json:json-to-object-transformer returnClass="java.util.Map" doc:name="JSON to Object"/>
<dw:transform-message metadata:id="3dad12b9-422d-4c59-9ec6-dd220ebca9ef" doc:name="Transform Message">
<dw:input-payload doc:sample="sample_data\flow16_HashMap_1.dwl" mimeType="application/java"/>
<dw:set-payload><![CDATA[%dw 1.0
%namespace mes http://www.namespace1.com/test/message/1.0
%namespace mod htto://www.namespace2.com/test/model/1.0
%output application/xml
---
mes#getItemResponse: {
mod#item : {
(payload)
}
}]]></dw:set-payload>
</dw:transform-message>
<byte-array-to-string-transformer doc:name="Byte Array to String"/>
<logger message="Test1 Transformer output : #[payload]" level="INFO" doc:name="Logger"/>
</sub-flow>
POC contain DataWeave script to transform Java Map to SOAP XML response
%dw 1.0
%namespace mes http://www.namespace1.com/test/message/1.0
%namespace mod htto://www.namespace2.com/test/model/1.0
%output application/xml
---
mes#getItemResponse: {
mod#item : {
(payload)
}
}
Hardcoded Input Java Map :
{
"item_id": "B0002345W45",
"item_type": "Item Type 1",
"item_type_name": "Item Type 1 Name",
"item_name": "item 1 name",
"item_summary": "item 1 summary",
"item_brand": "Brand 1",
"image_type_name": "SmallImage",
"url": "http://a/b/c"
}
Output :
<mes:getItemResponse xmlns:mes="http://www.namespace1.com/test/message/1.0">
<mod:item xmlns:mod="htto://www.namespace2.com/test/model/1.0">
<item_id>B0002345W45</item_id>
<item_type>Item Type 1</item_type>
<item_type_name>Item Type 1 Name</item_type_name>
<item_name>item 1 name</item_name>
<item_summary>item 1 summary</item_summary>
<item_brand>Brand 1</item_brand>
<image_type_name>SmallImage</image_type_name>
<url>http://a/b/c</url>
</mod:item>
</mes:getItemResponse>
You can try using
{`%dw 1.0
%output application/xml
{
xmlRootElement :
tagName1 : payload.DBEntryName1,
tagName2 : payload.DBEntryName2
}
} `
Friends,
I have two inputs, A and B, both JSON.
I need to generate with DataMapper, a single output (custom) also in JSON format.
Example:
My Input A:
{
"name": "John Doe",
"age": "40"
}
My Input B:
{
"country": "Brazil"
"city": "Rio de Janeiro"
}
My Custom Output:
{
"customerName": "John Doe",
"customerAge": "40",
"customerCountry": "Brazil",
"customerCity": "Rio de Janeiro"
}
It can generate this output using the Mule DataMapper? How to?
At the time, I know use only one Input with DataMapper.
You need to take one of the input as the payload and the other as the variable.
My Payload was
{
"name": "John Doe",
"age": "40"
}
My Variable was
{
"country": "Brazil"
"city": "Rio de Janeiro"
}
Please find the below sample that I tested and works.
<flow name="combineFlow">
<http:listener config-ref="HTTP_Listener_Configuration"
path="/testings" doc:name="HTTP" />
<logger message="Coming here " level="INFO" doc:name="Logger" />
<set-variable variableName="mypayload"
value="{"country": "Brazil","city": "Rio de Janeiro"}"
doc:name="Variable" mimeType="application/json" />
<logger level="INFO" doc:name="Logger" />
<dw:transform-message doc:name="Transform Message">
<dw:set-payload><![CDATA[%dw 1.0
%output application/json
---
{
"customerName": payload.name,
"customerAge": payload.age,
"customerCountry": flowVars.mypayload.country,
"customerCity": flowVars.mypayload.city
}]]></dw:set-payload>
</dw:transform-message>
</flow>
Hope this helps
Slightly similar with previous answer: you need to take one input as payload and another one as variable.
Following is the implementation using DataMapper (not DataWeave/Transform Message)
XML:
<flow name="genericFlow">
<http:listener config-ref="HTTP_Listener_Configuration" path="/dm" doc:name="HTTP" allowedMethods="POST"/>
<set-variable variableName="inputB"
value="#['{"country": "Brazil", "city": "Rio de Janeiro"}']" mimeType="application/json"
doc:name="Input B" />
<data-mapper:transform config-ref="JSON_To_JSON" doc:name="JSON To JSON">
<data-mapper:input-arguments>
<data-mapper:input-argument key="inputBjson">#[new org.json.JSONObject(flowVars.inputB)]</data-mapper:input-argument>
</data-mapper:input-arguments>
</data-mapper:transform>
<echo-component doc:name="Echo" />
</flow>
GRF:
...
<Dictionary>
<Entry id="DictionaryEntry0" input="true" name="inputPayload" output="false" type="object"/>
<Entry id="DictionaryEntry1" input="false" name="outputPayload" output="true" type="object"/>
<Entry dictval.__javaType="org.json.JSONObject" id="DictionaryEntry2" input="true" name="inputBjson" output="false" type="object"/>
</Dictionary>
...
<attr name="melScript"><![CDATA[//MEL
//START -> DO NOT REMOVE
output.__id = input.__id;
//END -> DO NOT REMOVE
output.customerName = input.name;
output.customerAge = input.age;
output.customerCountry = inputArguments.inputBjson.getString("country");
output.customerCity = inputArguments.inputBjson.getString("city");]]></attr>
Notes: input B is registered to DataMapper as Input Argument with Type = Object (MEL only), and its class is: org.json.JSONObject
One input give it as payload and other one use message properties and store the second input values as variable.Now inside data mapper use the variable in input argument field and map it.Simple!! hope it helps.