I am using dataweave for transforming XML to CSV. I want to know how to implement nested for loop in dataweave.
Below is the input xml:
<employee>
<id>1236</id>
<emplinfo>
<emplid>1961</emplid>
<jobinfo>
<status>T</status>
<title>Manager</title>
<start_date>2016-09-01</start_date>
</jobinfo>
<jobinfo>
<status>P</status>
<end_date>2016-08-31</end_date>
<title>Integration Manager</title>
<start_date>2016-08-01</start_date>
</jobinfo>
<jobinfo>
<status>A</status>
<end_date>2016-07-31</end_date>
<title>Communications Manager</title>
<start_date>2016-07-17</start_date>
</jobinfo>
</emplinfo>
<emplinfo>
<emplid>1801</emplid>
<jobinfo>
<status>T</status>
<title>AM</title>
<start_date>2016-09-01</start_date>
</jobinfo>
</emplinfo>
</employee>
Excepted output:
id empl_id status end_date title start_date
1236 1961 T Manager 2016-09-01
1236 1961 P 2016-08-31 Integration Manager 2016-08-01
1236 1961 A 2016-07-31 Communications Manager 2016-07-17
1236 1801 T AM 2016-09-01
Any help is greatly appreciated.
This works for me.
%dw 1.0
%output application/csv
---
flatten (payload map ((parent, parentindex) -> {
emplinfo:(parent.*emplinfo map ((emplinfo,empindex) -> {
jobinfo:(emplinfo.*jobinfo map ((jobinfo,jobindex) -> {
id: parent.id,
emplid : emplinfo.emplid,
status: jobinfo.status,
end_date:jobinfo.end_date,
title:jobinfo.title,
start_date:jobinfo.start_date
}))
}))
}))..jobinfo
I have used normal csv. you can choose any kind of format. Output is
id,emplid,status,end_date,title,start_date
1236,1961,T,,Manager,2016-09-01
1236,1961,P,2016-08-31,Integration Manager,2016-08-01
1236,1961,A,2016-07-31,Communications Manager,2016-07-17
1236,1801,T,,AM,2016-09-01
Hope this helps.
Do it something like this
Segment:(payload.segments map ((segment, segmentIndex)->
{
Leg:(segment.legs map ((leg, legIndex)->
{
Origin:leg.departureField.airportCodeField,
Destination:leg.arrivalField.airportCodeField
}))
}))
You have to use map operator for looping through the members of an array. The left hand side of map is the array which is processed and right hand side will have code for how members in it should be mapped. We can use lambda functions as well.
You can name the current member and refer it as well.
(payload.ARRAY map ((value,index) ->
{
id: parent.id,
emplid : value.emplid,
status: value.status,
}
)
Similiar depending on how you want to process the input array we can have nested map operators.
(payload.ARRAY map ((value,index) ->
{
(payload.ARRAY2 map ((value,index) ->
{
id: parent.id,
emplid : value.emplid,
status: value.status,
}
)
Related
I have one dataframe and one dataset :
Dataframe 1 :
+------------------------------+-----------+
|City_Name |Level |
+------------------------------+------------
|{City -> Paris} |86 |
+------------------------------+-----------+
Dataset 2 :
+-----------------------------------+-----------+
|Country_Details |Temperature|
+-----------------------------------+------------
|{City -> Paris, Country -> France} |31 |
+-----------------------------------+-----------+
I am trying to make a join of them by checking if the map in the column "City_Name" is included in the map of the Column "Country_Details".
I am using the following UDF to check the condition :
val mapEqual = udf((col1: Map[String, String], col2: Map[String, String]) => {
if (col2.nonEmpty){
col2.toSet subsetOf col1.toSet
} else {
true
}
})
And I am making the join this way :
dataset2.join(dataframe1 , mapEqual(dataset2("Country_Details"), dataframe1("City_Name"), "leftanti")
However, I get such error :
terminated with error scala.MatchError: UDF(Country_Details#528) AS City_Name#552 (of class org.apache.spark.sql.catalyst.expressions.Alias)
Has anyone previously got the same error ?
I am using Spark version 3.0.2 and SQLContext, with scala language.
There are 2 issues here, the first one is that when you're calling your function, you're passing one extra parameter leftanti (you meant to pass it to join function, but you passed it to the udf instead).
The second one is that the udf logic won't work as expected, I suggest you use this:
val mapContains = udf { (col1: Map[String, String], col2: Map[String, String]) =>
col2.keys.forall { key =>
col1.get(key).exists(_ eq col2(key))
}
}
Result:
scala> ds.join(df1 , mapContains(ds("Country_Details"), df1("City_Name")), "leftanti").show(false)
+----------------------------------+-----------+
|Country_Details |Temperature|
+----------------------------------+-----------+
|{City -> Paris, Country -> France}|31 |
+----------------------------------+-----------+
How to transform json data as shown in input into XML as shown in output using MuleSoft's DataWeave language.
Input
{
"sponsors":[{
"SP1": "John Smith",
"NINO": "SC105012B",
"INCOME1": 10000,
"INCOME2":12000,
"DEPENDENTDETAILS": "No, FirstName, LastName, DateOfBirth, EduStatus, Income; 1, John, Smith, 12/12/84, studying at college or university, yes; 2, williams, Smith, 20/06/84, studying at college or university, No;"
}]
}
Expected Output
<sponsors>
<sp1>john smith</sp1>
<nino>sc15012b</nino>
<income1>10000</income1>
<income2>12000</income2>
<dependentdetails>
<no>1</no>
<firstname>john</firstname>
<lastname>smith</lastname>
<dateOfBirth>12/12/84</dateOfBirth>
<EduStatus>studying in college or university<EduStatus/>
<Income>yes</Income>
<no>2</no>
<firstname>williams</firstname>
<lastname>smith</lastname>
<dateOfBirth>20/06/84</dateOfBirth>
<EduStatus>studying in college or university<EduStatus/>
<Income>No</Income>
</dependentdetails>
</sponsors>
DEPENDENTDETAILS seems to be a string of headers and records separated by ; and each field separated by ,. Once we split by each of those characters we can get a list of field names and a list of values for each records that we can manipulate easily.
%dw 2.0
output application/xml
fun recordsToObjects(a)= do {
var headers=a[0]
---
a[1 to -1] map ($ map ((headers[$$]): $) reduce ($$++$)) reduce ($$++$)
}
---
sponsors: {
sp1: payload.sponsors[0].SP1,
nino: payload.sponsors[0].NINO,
income1: payload.sponsors[0].INCOME1,
income2: payload.sponsors[0].INCOME2,
dependentdetails:recordsToObjects( (payload.sponsors[0].DEPENDENTDETAILS splitBy ";") map ($ splitBy "," map trim($)) )
}
There is one more option that you can try
%dw 2.0
output application/xml
var dependentdetails = payload.sponsors.DEPENDENTDETAILS[0] splitBy ";" map( $ splitBy ", ")
var keys=dependentdetails[0]
var dependentXml={(flatten(dependentdetails filter $$>0 map (
$ map ((item, index) ->
(keys[index]) : trim(item)
)
)))}
---
sponsors : payload.sponsors map ($ mapObject ((value, key, index) ->
if(key as String == "DEPENDENTDETAILS")(key): dependentXml
else (key):value
))
I need to extract the email from an intricate 'dict' (I am new to sql)
I have seen several previous posts on the same topic (e.g. this one) however, none seem to work on my data
select au.details
from table_au au
result:
{
"id":3526,
"contacts":[
{
"contactType":"EMAIL",
"value":"name#email.be",
"private":false
},
{
"contactType":"PHONE",
"phoneType":"PHONE",
"value":"025/6251111",
"private":false
}
]
}
I need:
name#email.be
select d.value -> 0 -> 'value' as Email
from json_each('{"id":3526,"contacts":[{"contactType":"EMAIL","value":"name#email.be","private":false},{"contactType":"PHONE","phoneType":"PHONE","value":"025/6251111","private":false}]}') d
where d.key::text = 'contacts'
Output:
| | email |
-------------------
|1 |"name#email.be"|
You can run it here: https://rextester.com/VHWRQ89385
I have response coming like this -
<rsp stat="ok" version="1.0">
<result>
<tagObject>
<id>xx</id>
<tag_id>001</tag_id>
<type>Test</type>
<object_id>101</object_id>
<created_at>2013-10-09 11:41:45</created_at>
</tagObject>
<tagObject>
<id>xy</id>
<tag_id>002</tag_id>
<type>Test</type>
<object_id>102</object_id>
<created_at>2013-10-09 11:41:45</created_at>
</tagObject>
<tagObject>
<id>xz</id>
<tag_id>003</tag_id>
<type>Test</type>
<object_id>103</object_id>
<created_at>2013-10-09 11:43:44</created_at>
</tagObject>
</result>
</rsp>
Now I have to create a payload out of it, so that I can insert only <tag_id> and <object_id> into a database table.
Any suggestion on this? I was trying to work with Dataweave with Xpath3 but it did not work for me.
enter image description hereYou can convert Xml to Java map using dataweave
%dw 1.0
%output application/java
---
payload.rsp.result.*tagObject map ((tagObject , indexOfTagObject) -> {
tag_id: tagObject.tag_id as :string,
object_id: tagObject.object_id as :string
})
Out put: [{tag_id=001, object_id=101}, {tag_id=002, object_id=102}, {tag_id=003, object_id=103}].
I am new to mule esb. I am trying to fetch values from mySQL and transforming to XML. Then hitting service and then result will be stored back in DB table. I have two questions
1) I can see two different way of mapping in ESB mule.
a) Using "Data Mapper"
b) Using "Transform Message" - This will use data Weaver.
Which one of the transformation method will suit for my requirement.
2) I tried using "Transform Message" and Data Weaver for transformation.
My input from DB will fetch three rows as like below.
college = abc college = abc college = abc
dept = IT dept = CSE dept = MECH
No of students = 5 No of students = 4 No of students = 7
Expected output is
<University>
<college>abc</college> <!-- this is simple tag/ It will not repeat -->
<dept> <!-- This is complex tag -->
<dept Name>IT</dept Name>
<No of students>5</No of students>
</dept>
<dept>
<dept Name>IT</dept Name>
<No of students>5</No of students>
</dept>
<dept>
<dept Name>IT</dept Name>
<No of students>5</No of students>
</dept>
I tried data weaver something like below.
%dw 1.0
%output application/xml
"University":{
"college": payload.college,
dept: {
dept_Name: payload."dept_Name",
No of students: payload."No of students"
}
}
But the above dw is not giving expected result. Could you please help me on this ?
You can try this DataWeave, and modify as you need. Hope that helps.
%dw 1.0
%output application/xml
---
{
University: {
college: payload.college[0],
(payload map ((payload01 , indexOfPayload01) -> {
dept: {
deptName: payload01.dept,
Noofstudents: payload01."No of students" as :number
}
}))
}
}