I am trying to migrate data from postgre to Hubspot. I am doing it with integration APIs provided by the Hubspot CRM. The number of API calls can reach up to 1000 at a time. I want to maintain logs showing which records were migrated successfully and which weren't. For this, I need to know which HTTP request returned the result. How can I implement this?
<flow name="migration-to-hubspootFlow" doc:id="0b2433b4-0de3-4f10-92e9-fc1014015043" >
<scheduler doc:name="Scheduler" doc:id="63c6576a-1cca-4a17-b7d7-1f033ba88453" >
<scheduling-strategy >
<fixed-frequency frequency="60" timeUnit="MINUTES"/>
</scheduling-strategy>
</scheduler>
<db:select doc:name="Select" doc:id="b258bb4c-6e34-4b66-a59f-8ae634975373" config-ref="Database_Config">
<db:sql ><![CDATA[select distinct c.email, c.lastmodifieddate, c.id, c.sfid, c.firstname, c.lastname, c.title, c.phone,c.mailingstate, c.mailingstreet, c.mailingcity, c.mailingpostalcode, c.mailingcountry
from salesforceuat.contact c
where c.email is not null
limit 500]]></db:sql>
</db:select>
<batch:job jobName="migration-to-hubspootBatch_Job" doc:id="ba2f58cd-ddd2-4e28-911f-107d6e7333db" >
<batch:process-records >
<batch:step name="Batch_Step" doc:id="f4fa295c-bc1f-4c22-8fb3-6193f415cafb" >
<http:request method="POST" doc:name="Request" doc:id="c7eb69e0-0a43-4526-91e2-022469c203bc" config-ref="HTTP_Request_configuration" path="/contacts/v1/contact/createOrUpdate/email/{userEmail}/" sendBodyMode="ALWAYS" requestStreamingMode="AUTO">
<http:body><![CDATA[#[%dw 2.0
output application/json
---
{
"properties": [
{
"property": "JobTitle",
"value": payload.title
},
{
"property": "firstname",
"value": payload.firstname
},
{
"property": "lastname",
"value": payload.lastname
},
{
"property": "website",
"value": ""
},
{
"property": "company",
"value": "HubSpot"
},
{
"property": "phone",
"value": payload.phone
},
{
"property": "address",
"value": payload.mailingstreet
},
{
"property": "city",
"value": payload.mailingcity
},
{
"property": "state",
"value": payload.mailingstate
},
{
"property": "zip",
"value": payload.mailingpostalcode
}
]
}]]]></http:body>
<http:headers><![CDATA[#[output application/java
---
{
"Host" : "api.hubapi.com",
"Content-Type" : "application/json"
}]]]></http:headers>
<http:uri-params><![CDATA[#[output application/java
---
{
"userEmail" : payload.email
}]]]></http:uri-params>
<http:query-params><![CDATA[#[output application/java
---
{
"hapikey" : "************"
}]]]></http:query-params>
</http:request>
<logger level="INFO" doc:name="Logger" doc:id="531d9bdf-3eca-4159-88b4-6ab86016927b" message="#[message.payload] #[message.attributes]" />
</batch:step>
</batch:process-records>
</batch:job>
</flow>
If you set maxFailedRecords to -1 (docs), then records that result in a mule error (like HTTP:BAD_REQUEST) can be handled in a subsequent batch step that uses acceptPolicy=ONLY_FAILURES (docs). There you can handle them one at a time or all together using a Batch Aggregator.
Before an http call store the unique id in a variable and make a call to target and get the status and frame a message along with the status and unique Id.
store/process to MQ or in VM QM of success and failed records and process them back to DB to update the status.
Related
I having input as Array (json) which needs to groupBy and orderBy with clientId so that its internal Lines are grouped and ordered in to one root in xml (repetitive lines of clientId). I'm stuck how to use dataweave for the XMl response. Please find the request and expected response.
Request:
[
{
"ClientId": 2,
"Code": "string",
"ReceivedDate": "2018-10-23",
"ReceivedTime": "2217",
"Warehouse": "30",
"Quantity": "20"
},
{
"ClientId": 1,
"Code": "string",
"ReceivedDate": "2018-10-23",
"ReceivedTime": "2217",
"Warehouse": "56",
"Quantity": "20"
},
{
"ClientId": 1,
"Code": "string",
"ReceivedDate": "2018-10-23",
"ReceivedTime": "2217",
"Warehouse": "70",
"Quantity": "20"
}
]
Response
<?xml version='1.0' encoding='UTF-8'?>
<Receipt>
<client clientId="1">
<code>string</code>
<Lines>
<Warehouse>56</Warehouse>
<Quantity>20</Quantity>
</Lines>
<Lines>
<Warehouse>70</Warehouse>
<Quantity>20</Quantity>
</Lines>
</client>
<client clientId="2">
<code>string</code>
<Lines>
<Warehouse>30</Warehouse>
<Quantity>20</Quantity>
</Lines>
</client>
</Receipt>
Let me if question is unclear. Thanks in advance.
Try this, comments are embedded in the code:
%dw 2.0
output application/xml
var data = [
{
"ClientId": 2,
"Code": "string",
"ReceivedDate": "2018-10-23",
"ReceivedTime": "2217",
"Warehouse": "30",
"Quantity": "20"
},
{
"ClientId": 1,
"Code": "string",
"ReceivedDate": "2018-10-23",
"ReceivedTime": "2217",
"Warehouse": "56",
"Quantity": "20"
},
{
"ClientId": 1,
"Code": "string",
"ReceivedDate": "2018-10-23",
"ReceivedTime": "2217",
"Warehouse": "70",
"Quantity": "20"
}
]
---
Receipt: do {
// Group by the data
var groupedData = data groupBy $.ClientId
// Order the client Ids
var orderedClientIds = groupedData pluck $$ orderBy $ as Number
---
// Iterate over the ordered clientIds and create an object, hence the use of reduce
orderedClientIds reduce (cId, cIds={}) -> cIds ++ {
// Add the clientId attribute to the client tag
client #(clientId: cId ): {
// Get the code from the first element in the array
code: groupedData[cId][0].Code,
// Create the Lines, should you avoid repeating tags at the
// same level with other tags? i.e. code and Lines
// IMHO a best practice XML should have code, a single Lines
// and nested in Lines, one or more Line tags organizing the
// data
Lines: groupedData[cId] reduce (
(l,ls={}) -> ls ++ Lines :{
Warehouse: l.Warehouse,
Quantity: l.Quantity
}
)
}
}
}
What is still unclear to me is whether you want to order the Warehouse and/or Quantity values. I did not in the above code but you can easily do it, just let me know and I 'll amend.
EDIT: If you copy and paste the code in a Transform Message processor, Studio will show an error--this error is a false positive, turn on the preview or even better stick an HTTP Listener to the source and give it a go :)
I hate xml. But here's my go at it. Hope it helps.
%dw 2.0
output application/json
var data = [
{
"ClientId": 2,
"Code": "string",
"ReceivedDate": "2018-10-23",
"ReceivedTime": "2217",
"Warehouse": "30",
"Quantity": "20"
},
{
"ClientId": 1,
"Code": "string",
"ReceivedDate": "2018-10-23",
"ReceivedTime": "2217",
"Warehouse": "56",
"Quantity": "20"
},
{
"ClientId": 1,
"Code": "string",
"ReceivedDate": "2018-10-23",
"ReceivedTime": "2217",
"Warehouse": "70",
"Quantity": "20"
}
]
var clients = (data orderBy $.ClientId groupBy $.ClientId)
---
{
Receipt: clients mapObject (v,k,i) -> {client #(clientId: k): {
code: v.Code[0]
} ++ ((v map (item,ind) -> {
Lines: {
Warehouse: item.Warehouse,
Quantity: item.Quantity
}
}) reduce (value, acc={}) -> acc ++ value)
}
}
Gives output:
<?xml version='1.0' encoding='UTF-8'?>
<Receipt>
<client clientId="1">
<code>string</code>
<Lines>
<Warehouse>56</Warehouse>
<Quantity>20</Quantity>
</Lines>
<Lines>
<Warehouse>70</Warehouse>
<Quantity>20</Quantity>
</Lines>
</client>
<client clientId="2">
<code>string</code>
<Lines>
<Warehouse>30</Warehouse>
<Quantity>20</Quantity>
</Lines>
</client>
</Receipt>
I have json response that I need to break into multiple parts before sending it to target system.
I was trying with for-each but couldn't reach my desired output.
input json look like this
{
parent :
{
child:
[{
a:[{},{},{},{}]
}]
}
}
and out should look like this
part1
{
parent :
{
child:
[{
a:[{},{}]
}]
}
}
part2
{
parent :
{
child:
[{
a:[{},{}]
}]
}
}
Can someone please help me out here?
It is hard to tell based on your sample json as its not formatted json.
NOTE The expected input/output changed from my original answer so this is an update
However, this works and can tweaked based on your actual input.
First, use foreach to iterate the collection part of your payload(based on your example it is #[flatten(payload.parent.child.a)] (but this might change as its not clear if the 'child' array would also more than one element.)
You also don't mention how you want to split it. Based on your example, it is for every 2 records. So use the batchSize attribute and set to 2(change this to your actual requirement).
Then you need to add the the wrapping json elements back, as you will lose that in the foreach:
<foreach doc:name="For Each" collection="#[flatten(payload.parent.child.a)]" batchSize="2">
<ee:transform doc:name="Transform Message" doc:id="7b1cccb7-fcbb-41b3-a08f-bac5600df2f2" >
<ee:message >
<ee:set-payload ><![CDATA[%dw 2.0
output application/json
---
{
parent :
{
child:[
a: payload
]
}
}]]></ee:set-payload>
</ee:message>
</ee:transform>
<logger level="ERROR" message="Split items here: #[payload]" />
</foreach>
The output matches your new expected output:
Split items here: {
"parent": {
"child": [
{
"a": [
{
},
{
}
]
}
]
}
}
Split items here: {
"parent": {
"child": [
{
"a": [
{
},
{
}
]
}
]
}
}
There are two http calls(calling rest api) which gives two different responses.
first one is giving json response for Shops like below
{
"shops": [
{
"shop": {
"code": "a",
"streetName": "a",
"city": "a",
"district": "a",
"state": "a",
"postalCode": "a",
"country": "a"
}
},
{
"shop": {
"code": "b",
"streetName": "b",
"city": "b",
"district": "b",
"state": "b",
"postalCode": "b",
"country": "b"
}
}
]
}
other call is giving sales office for given code(code is same as for shop).it returns a single sales office for given code
{
"salesOffice": {
"shop": {
"code": "a"
},
"office": "a",
"branch": "a",
"district": "a",
"subRegion": "a",
"region": "a"
}
}
my requirement is to get shops which is an json array and add sales office for each shop and return a single json response like below
{
"shops": [
{
"shop": {
"code": "a",
"streetName": "a",
"city": "a",
"district": "a",
"state": "a",
"postalCode": "a",
"country": "a",
"salesOffice": {
"office": "a",
"branch": "a",
"district": "a",
"subRegion": "a",
"region": "a"
}
}
},
{
"shop": {
"code": "b",
"streetName": "b",
"city": "b",
"district": "b",
"state": "b",
"postalCode": "b",
"country": "b",
"salesOffice": {
"office": "b",
"branch": "b",
"district": "b",
"subRegion": "",
"region": "b"
}
}
}
]
}
is there any way to achieve the result?
Edit
I need to get sales office for both shops( there will be a http request for each shop code) and merge it into the response. so if I get two shops then I need to send two http request to get salesoffice for those code and then merge it with the response.
so the first call will give me shops (lets say two shops). then I need to make two http calls by getting "code" present in each shop to get sales office , and then I need to merge both shops with respective sales office.
You can do something like this.
XML file:
<flow name="merge-jsonFlow1">
<http:listener config-ref="HTTP_Listener_Configuration" path="/json-merge" doc:name="HTTP"/>
<expression-component doc:name="Mock HTTP Response"><![CDATA[payload = '{"shops":[{"shop":{"code":"a","streetName":"a","city":"a","district":"a","state":"a","postalCode":"a","country":"a"}},{"shop":{"code":"b","streetName":"b","city":"b","district":"b","state":"b","postalCode":"b","country":"b"}}]}']]></expression-component>
<set-variable variableName="shops" value="#[payload]" doc:name="Store Orig Payload to Variable"/>
<processor-chain doc:name="Processor Chain">
<splitter expression="#[json:/shops]" doc:name="Splitter"/>
<set-variable variableName="storeCode" value="#[json:/shop/code]" doc:name="Set Store Code"/>
<expression-component doc:name="Mock HTTP Call to Get SalesOffice"><![CDATA[if (flowVars['storeCode'] == 'a') {
payload = '{"salesOffice":{"shop":{"code":"a"},"office":"a","branch":"a","district":"a","subRegion":"a","region":"a"}}';
} else if (flowVars['storeCode'] == 'b') {
payload = '{"salesOffice":{"shop":{"code":"b"},"office":"b","branch":"b","district":"b","subRegion":"b","region":"b"}}';
}]]></expression-component>
<collection-aggregator failOnTimeout="true" doc:name="Collection Aggregator"/>
</processor-chain>
<set-payload value="#[[flowVars['shops'], payload]]" doc:name="Merge Two JSON"/>
<json:json-to-object-transformer doc:name="JSON to Object"/>
<dw:transform-message metadata:id="c72e3e02-8350-42ec-a3cb-ca61c7b722b4" doc:name="Transform Message">
<dw:input-payload doc:sample="json-merge.json"/>
<dw:set-payload><![CDATA[%dw 1.0
%output application/json
---
using (
shops = payload[0].shops,
so = payload[1].*salesOffice
)
{
shops : shops.shop map {
shop: using (mycode = $.code) {
code: $.code,
streetName: $.streetName,
city: $.city,
district: $.district,
state: $.state,
postalCode: $.postalCode,
country: $.country,
salesOffice: ((so map {
shopCode: $.shop.code,
office: $.office,
branch: $.branch,
district: $.district,
subRegion: $.subRegion,
region: $.region
}) filter $.shopCode == mycode)[0]
}
}
}]]></dw:set-payload>
</dw:transform-message>
</flow>
You can do something like this. I used dataweave to transform json.
The Mock API Response is http call to your api.
XML file:
<flow name="merge-jsonFlow">
<http:listener config-ref="HTTP_Listener_Configuration" path="/merge" doc:name="HTTP"/>
<scatter-gather doc:name="Scatter-Gather">
<expression-component doc:name="Mock API1 Response"><![CDATA[payload = '{"shops":[{"shop":{"code":"a","streetName":"a","city":"a","district":"a","state":"a","postalCode":"a","country":"a"}},{"shop":{"code":"b","streetName":"b","city":"b","district":"b","state":"b","postalCode":"b","country":"b"}}]}']]></expression-component>
<expression-component doc:name="Mock API2 Response"><![CDATA[payload = '{"salesOffice":{"shop":{"code":"a"},"office":"a","branch":"a","district":"a","subRegion":"a","region":"a"}}']]></expression-component>
</scatter-gather>
<json:json-to-object-transformer doc:name="JSON to Object"/>
<dw:transform-message metadata:id="0066a92a-85d8-4514-8459-b70252cb2f7b" doc:name="Transform Message">
<dw:set-payload><![CDATA[%dw 1.0
%output application/json
---
using (
shops = payload[0].shops,
salesOffice = payload[1].salesOffice
)
{
shops : shops.shop map {
shop: {
code: $.code,
streetName: $.streetName,
city: $.city,
district: $.district,
state: $.state,
postalCode: $.postalCode,
country: $.country,
(salesOffice: {
office: salesOffice.office,
branch: salesOffice.branch,
district: salesOffice.district,
subRegion: salesOffice.subRegion,
region: salesOffice.region
}) when $.code == salesOffice.shop.code
}
}
}]]></dw:set-payload>
</dw:transform-message>
<logger message="#[payload]" level="INFO" doc:name="Logger"/>
</flow>
Sample Response:
{
"shops": [
{
"shop": {
"code": "a",
"streetName": "a",
"city": "a",
"district": "a",
"state": "a",
"postalCode": "a",
"country": "a",
"salesOffice": {
"office": "a",
"branch": "a",
"district": "a",
"subRegion": "a",
"region": "a"
}
}
},
{
"shop": {
"code": "b",
"streetName": "b",
"city": "b",
"district": "b",
"state": "b",
"postalCode": "b",
"country": "b"
}
}
]
}
Hope this helps
I want to set to http.status code in Dataweave transformation. For this I am setting http.status on Outbound Property tab using below code:
Outbound Property ===> http.status
%dw 1.0
%output application/java
---
"404" when payload[0] == null
otherwise "200"
But it is not reflected even when payload is null. Can any suggest?
EDIT:
For reference here is the full Dataweave code
<dw:transform-message doc:name="buildResponse">
<dw:set-payload>
<![CDATA[%dw 1.0
%input payload application/java
%output application/json
---
{
"customer": {
"id": payload[0].ID,
"name": payload[0].NAME,
"age": payload[0].AGE,
"address": {
"line1": payload[0].LINE1,
"line2": payload[0].LINE2,
"city": payload[0].CITY,
"state": payload[0].STATE,
"pincode": payload[0].PINCODE
}
}
} when (sizeOf payload) > 0
otherwise
{
"customer" : "not found"
}]]>
</dw:set-payload>
<dw:set-property propertyName="http.status">
<![CDATA[%dw 1.0
%output application/java
---
"404" when (sizeOf payload) == 0
otherwise "200"]]>
</dw:set-property>
</dw:transform-message>
EDIT.
This is because the first part of the dataweave script is overwriting the payload to the JSON. So when you use sizeOf in the set-property dw script it's actually doing a sizeOf the JSON string which is not 0.
To make this work, I would probably store the original payload in a flowVar and use that in the dw script like so:
<http:listener-config name="HTTP_Listener_Configuration"
host="0.0.0.0" port="8085" doc:name="HTTP Listener Configuration" />
<flow name="dataweave-testFlow">
<http:listener config-ref="HTTP_Listener_Configuration"
path="/test" doc:name="HTTP" />
<set-payload value="#[[]]" doc:name="Set Payload" />
<set-variable variableName="originalPayload" value="#[payload]"
doc:name="Variable" />
<dw:transform-message doc:name="buildResponse">
<dw:set-payload>
<![CDATA[%dw 1.0
%input payload application/java
%output application/json
---
{
"customer": {
"id": payload[0].ID,
"name": payload[0].NAME,
"age": payload[0].AGE,
"address": {
"line1": payload[0].LINE1,
"line2": payload[0].LINE2,
"city": payload[0].CITY,
"state": payload[0].STATE,
"pincode": payload[0].PINCODE
}
}
} when (sizeOf payload) > 0
otherwise
{
"customer" : "not found"
}]]>
</dw:set-payload>
<dw:set-property propertyName="http.status">
<![CDATA[%dw 1.0
%output application/java
---
"404" when (sizeOf flowVars.originalPayload) == 0
otherwise "200"]]>
</dw:set-property>
</dw:transform-message>
</flow>
This is a bug with Dataweave in my opinion and created it here: https://www.mulesoft.org/jira/browse/MULE-9021
Ryan Carter's answer is correct. However, there is a specific answer without storing the original payload into a flow variable.
<dw:transform-message doc:name="buildResponse">
<dw:set-payload>
<![CDATA[%dw 1.0
%input payload application/java
%output application/json
---
{
"customer": {
"id": payload[0].ID,
"name": payload[0].NAME,
"age": payload[0].AGE,
"address": {
"line1": payload[0].LINE1,
"line2": payload[0].LINE2,
"city": payload[0].CITY,
"state": payload[0].STATE,
"pincode": payload[0].PINCODE
}
}
} when (sizeOf payload) > 0
otherwise
{
"customer" : "not found"
}]]>
</dw:set-payload>
<dw:set-property propertyName="http.status">
<![CDATA[%dw 1.0
%output application/java
---
"404" when payload.customer == "not found"
otherwise "200"]]>
</dw:set-property>
</dw:transform-message>
I did like this: "404" when payload.customer == "not found"
{
"product1": {
"addedBy": "1014",
"addedDate": "1429681809598",
"campaignName": promotion,
"status": "Available"
},
"product2": {
"addedBy": "1015",
"addedDate": "1429681809598",
"campaignName": bulkoffer,
"status": "Available"
},
"product3": {
"addedBy": "1015",
"addedDate": "1429681809598",
"campaignName": premiumoffer,
"status": "Available"
}
}
in mule ,the above json is input data for me. I want to change the status value in runtime in mule. can anyone tell the way for it
Convert it to a map first:
<json:object-to-json-transformer returnClass="java.util.HashMap" />
Then use MEL to update the map
<expression-component>payload.product1.status = 'Unavailable'</expression-component>
Then convert it back to json when you are ready:
<json:object-to-json-transformer />
Alternative way is to use Dataweave which can fill your requirement easily and can change the value of "status" dynamically at runtime whichever "status"element you want to change
<dw:transform-message doc:name="Transform Message">
<dw:set-payload><![CDATA[%dw 1.0
%output application/json
---
{
product1: {
"addedBy": payload.product1.addedBy,
"addedDate": payload.product1.addedDate,
"campaignName": payload.product1.campaignName,
"status": "Unavailable"
},
product2:{
"addedBy": payload.product2.addedBy,
"addedDate": payload.product2.addedDate,
"campaignName": payload.product2.campaignName,
"status": "Unavailable"
},
product3:{
"addedBy": payload.product3.addedBy,
"addedDate": payload.product3.addedDate,
"campaignName": payload.product3.campaignName,
"status": "Unavailable"
}
} ]]></dw:set-payload>
</dw:transform-message>