Related
In Azure Data Factory, I need to tap into a HTTP requests via URL using the HTTP connector. I was able to do this as well as setup the dataset. Where I'm having issues is on the pipeline. Here's what I need to do. What is the best way to accomplish this?
Call out to the service base URL and retrieve the header returned of TotalPages.
Using the value for TotalPages, make subsequent requests to the URL with the parameter page (e.g., page=1, page=2, etc.) using the value from TotalPages to form those requests.
Thanks.
Ok. So the issue here is that you cannot nest control structures in Data Factory more than 1 time. The solution is to create two or more pipelines (aka Master and Child).
From the Master pipeline retrieve the number of tasks you will need to execute, and pass them to a for loop. Within the for loop launch for each activity pair a new Child pipeline which will then execute the second activity.
If the Activity is simple enough you can skip the Child Pipeline altogether and do it directly inside the first for loop.
As a Json representation of pipelines in question it should look along these lines:
{
"name": "generic_master",
"properties": {
"activities": [
{
"name": "Web1",
"type": "WebActivity",
"dependsOn": [],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"url": "https://jsonplaceholder.typicode.com/posts/1",
"method": "GET"
}
},
{
"name": "ForEach1",
"type": "ForEach",
"dependsOn": [
{
"activity": "Web1",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"items": {
"value": "#activity('Web1').output",
"type": "Expression"
},
"activities": [
{
"name": "Execute Pipeline1",
"type": "ExecutePipeline",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"pipeline": {
"referenceName": "generic_child",
"type": "PipelineReference"
},
"waitOnCompletion": true
}
}
]
}
}
],
"annotations": []
}
}
{
"name": "generic_child",
"properties": {
"activities": [
{
"name": "Web1",
"type": "WebActivity",
"dependsOn": [],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"url": "https://jsonplaceholder.typicode.com/posts/1",
"method": "POST"
}
}
],
"annotations": []
}
}
In order to read the TotalPages values from the HTTP Request's response, you can use a "Lookup" activity to submit the HTTP request and store the TotalPages value in a variable with the "Set variable" activity.
Actions:
Pipeline level:
create a variable called TotalPages
Lookup activity:
tick the first row only box on the Settings tab
As a source dataset, use the data set defined for your HTTP request
Select the GET method.
Set variable activity:
Select the TotalPages variable on the Variables tab
In the value box, click on "Add dynamic content" and enter something like this: #{activity('GetTotalPages').output.firstRow.RegisterSearch['#TotalPages']}
In my case, the lookup activity is called GetTotalPages, and my HTTP request returns the total number of pages in a RegisterSearch array, under a column name #TotalPages
I'm currently making a call to an SQLite local database in my react native
Expo app like so:
db.transaction(tx => {
tx.executeSql('select * from dr_report_templates', [], (_, { rows }) => {
const templateData = JSON.stringify(rows);
this.setState({ options: templateData, isLoading: false }, () => console.log(this.state))
});
},
error => {
alert(error);
},
() => console.log('Loaded template settings')
);
I'm returning the data and making it a JSON string with: JSON.stringify
Data appears like so:
Object {
"isLoading": false,
"options": "{\"_array\":[{\"id\":30,\"name\":\"SFR General\",\"description\":\"SFR1\"},{\"id\":31,\"name\":\"SFR Extended\",\"description\":\"SFR2\"},{\"id\":7790,\"name\":\"test_new_template\",\"description\":\"test_new_template\"},{\"id\":7792,\"name\":\"apart_1\",\"description\":\"apart_1\"},{\"id\":7793,\"name\":\"SFR\",\"description\":\"Single Family\"},{\"id\":7798,\"name\":\"Condo\",\"description\":\"Condo \"},{\"id\":7799,\"name\":\"Duplex\",\"description\":\"Duplex\"},{\"id\":7800,\"name\":\"Triplex \",\"description\":\"3\"},{\"id\":7801,\"name\":\"Apartments\",\"description\":\"Apartment complex\"},{\"id\":7802,\"name\":\"Commercial retail store \",\"description\":\"Storefront \"},{\"id\":7803,\"name\":\"5-10 unit\",\"description\":\"5\"},{\"id\":7804,\"name\":\"Commercial Industrial \",\"description\":\"Industrial \"},{\"id\":7805,\"name\":\"Industrial Property\",\"description\":\"RE\"}],\"length\":13}",
"selected": "",
}
Attempting to get values for just the first array element like so:
this.state.options[0]
does not work. I'm obviously doing something wrong in the way that I'm doing this but can't figure out what. Any ideas?
EDIT: I had also ran the query with out JSON.Stringify. The data returned like so with this "t" in front of it. I've never hard this before and I couldn't loop through it so that's why I did a JSON.stringify.
t {
"_array": Array [
Object {
"description": "SFR1",
"id": 30,
"name": "SFR General",
},
Object {
"description": "SFR2",
"id": 31,
"name": "SFR Extended",
},
Object {
"description": "test_new_template",
"id": 7790,
"name": "test_new_template",
},
Object {
"description": "apart_1",
"id": 7792,
"name": "apart_1",
},
Object {
"description": "Single Family",
"id": 7793,
"name": "SFR",
},
Object {
"description": "Condo ",
"id": 7798,
"name": "Condo",
},
Object {
"description": "Duplex",
"id": 7799,
"name": "Duplex",
},
Object {
"description": "3",
"id": 7800,
"name": "Triplex ",
},
Object {
"description": "Apartment complex",
"id": 7801,
"name": "Apartments",
},
Object {
"description": "Storefront ",
"id": 7802,
"name": "Commercial retail store ",
},
Object {
"description": "5",
"id": 7803,
"name": "5-10 unit",
},
Object {
"description": "Industrial ",
"id": 7804,
"name": "Commercial Industrial ",
},
Object {
"description": "RE",
"id": 7805,
"name": "Industrial Property",
},
],
"length": 13,
}
this.setState({ options: templateData._array, isLoading: false });
or change how you destructure in 3rd parameter of executeSql to:
(_, { rows: { _array } }) => {
const templateData = JSON.stringify(_array);
}
Why you're conveting it with JSON.stringify()? You can iterate over array or access it with array's key name.
NOTE: JSON.stringify() does not convert it to JSON. It converts to JSON string
The JSON.stringify() method converts a JavaScript object or value to a
JSON string, optionally replacing values if a replacer function is
specified or optionally including only the specified properties if a
replacer array is specified.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/stringify
You're actually converting your database response to string.
So Change
const templateData = JSON.stringify(rows);
to
const templateData = rows;
and use this array where you want.
In a Zapier Zap I am using an an API GET call to Tsheets to grab a list of Timesheets. I would like to split out each time sheet into line items like line items in a Xero invoice because I would like to save item data from each timesheet to its own row in a Google sheet. (Ideally I would like to save the line data directly to a MySQL database but I see that Zapier currently only support Google sheets saving multiple lines at a time.) However I am having no joy.
I suspect one of two issues:
Zapier expects the word lineitems in the response or
The format of the response is not correct - I seem to have two "results" categories
In my step to Set up Google Sheets Spreadsheet Row I don't get a selection of comma separated items as shown in the example shown on the picture here:
Add an action app that supports line items, and each item will be saved individually
The image is from this page: https://zapier.com/blog/formatter-line-item-automation/ with the caption "Add an action app that supports line items, and each item will be saved individually" For what I get see photo https://cdn.zapier.com/storage/photos/f055dcf11a4b11b86f912f9032780429.png
In the step that returns the data from the API the text response is shown in https://cdn.zapier.com/storage/photos/33129fb7425cfae44be4a81533d6e892.png
and if I return json data it is like this: https://cdn.zapier.com/storage/photos/34da1b98f8941324c35befef8efe350d.png
Can anyone confirm that my suspicions are correct and whether 1 or 2 is the likely culprit.
Is it possible this link Zapier - Catch Hook - JSON Array - Loop over each item in array will lead me to the solution? It looks like it may but I don't see exactly how the writer incorporated it in to his Zap.
Edit: My data returned from the API looks like this:
{
"results": {
"timesheets": {
"11515534": {
"id": 11515534,
"user_id": 1260679,
"jobcode_id": 11974818,
"start": "2018-07-13T14:58:00+10:00",
"end": "2018-07-13T14:58:00+10:00",
"duration": 0,
"date": "2018-07-13",
"tz": 10,
"tz_str": "Australia\/Brisbane",
"type": "regular",
"location": "(Brisbane, Queensland, AU?)",
"on_the_clock": false,
"locked": 0,
"notes": "",
"customfields": {
"118516": "",
"121680": "",
"118530": "",
"118518": "Field supplies, materials"
},
"last_modified": "2018-07-13T04:59:27+00:00",
"attached_files": [
]
},
"11515652": {
"id": 11515652,
"user_id": 1260679,
"jobcode_id": 11974830,
"start": "2018-07-13T14:59:00+10:00",
"end": "2018-07-13T14:59:00+10:00",
"duration": 0,
"date": "2018-07-13",
"tz": 10,
"tz_str": "Australia\/Brisbane",
"type": "regular",
"location": "(Brisbane, Queensland, AU?)",
"on_the_clock": false,
"locked": 0,
"notes": "",
"customfields": {
"118516": "",
"121680": "",
"118530": ""
},
"last_modified": "2018-07-13T05:00:30+00:00",
"attached_files": [
]
},
"39799840": {
"id": 39799840,
"user_id": 1260679,
"jobcode_id": 19280104,
"start": "2018-10-24T11:45:00+11:00",
"end": "2018-10-24T12:00:00+11:00",
"duration": 900,
"date": "2018-10-24",
"tz": 11,
"tz_str": "Australia\/Brisbane",
"type": "regular",
"location": "(Sydney, New South Wales, AU?)",
"on_the_clock": false,
"locked": 0,
"notes": "",
"customfields": {
"118516": "",
"121680": "FP - Field plant Installation",
"118530": "Site cleanup"
},
"last_modified": "2018-10-24T05:56:27+00:00",
"attached_files": [
]
},
"39801850": {
"id": 39801850,
"user_id": 1260679,
"jobcode_id": 19280204,
"start": "2018-10-24T12:00:00+11:00",
"end": "2018-10-24T13:45:00+11:00",
"duration": 6300,
"date": "2018-10-24",
"tz": 11,
"tz_str": "Australia\/Brisbane",
"type": "regular",
"location": "(Sydney, New South Wales, AU?)",
"on_the_clock": false,
"locked": 0,
"notes": "",
"customfields": {
"118516": "",
"121680": "OP - Plant, Vehicles",
"118530": "Load\/Unload"
},
"last_modified": "2018-10-24T05:57:04+00:00",
"attached_files": [
]
},
"40192757": {
"id": 40192757,
"user_id": 1260679,
"jobcode_id": 19280110,
"start": "2018-10-25T08:00:00+11:00",
"end": "2018-10-25T10:00:00+11:00",
"duration": 7200,
"date": "2018-10-25",
"tz": 11,
"tz_str": "Australia\/Brisbane",
"type": "regular",
"location": "TSheets Android App",
"on_the_clock": false,
"locked": 0,
"notes": "From my mobile",
"customfields": {
"118516": "",
"121680": "FW - Plant Assembly",
"118530": "Panels"
},
"last_modified": "2018-10-24T23:02:56+00:00",
"attached_files": [
]
},
"40193033": {
"id": 40193033,
"user_id": 1260679,
"jobcode_id": 19280108,
"start": "2018-10-25T10:00:00+11:00",
"end": "2018-10-25T10:00:00+11:00",
"duration": 0,
"date": "2018-10-25",
"tz": 11,
"tz_str": "Australia\/Brisbane",
"type": "regular",
"location": "TSheets Android App",
"on_the_clock": false,
"locked": 0,
"notes": "",
"customfields": {
"118516": "",
"121680": "FW - Plant Assembly",
"118530": "Panels"
},
"last_modified": "2018-10-24T23:06:05+00:00",
"attached_files": [
]
}
}
},
"more": false
}
And this is my Python code: https://imgur.com/a/8W1X1em
Alright so I think I've worked something out for you. The example you provided Zapier - Catch Hook - JSON Array - Loop over each item in array is definitely on the right track, but, because it relies on webhooks, it probably won't work for you unless you can POST the data from your invoicing application.
Note: I code in Python so my examples will be in Python, that said these examples are pretty much code agnostic and can be replicated in Javascript as well.
I setup a dummy Zap to replicate what is happening with your zap currently
# results = requests.get(url, headers=header)
# results = results.json()
# Dummy result data converted to JSON object after API GET request:
results = {
"results" : {
"timesheets" : {
"timesheet_id_1" : {
"data_1" : "data",
"data_2" : "data",
"data_3" : "data"
},
"timesheet_id_2" : {
"data_1" : "data",
"data_2" : "data",
"data_3" : "data"
},
"timesheet_id_3" : {
"data_1" : "data",
"data_2" : "data",
"data_3" : "data"
}
}
}
}
return results
Reading a bit further here in order for Zapier to map line items it needs to receive the data in an array. The above output is a dictionary object, Zapier does map the values in this dictionary to data that can be accessed later, however it maps the entire dictionary which is why you are seeing the output as multiple fields and as is replicated in my output. What you are looking to do is map a subset of the dictionary AND provide each subset as separate outputs.
What you will want to do is loop through the inner fields of the results dictionary object and execute zaps on the nested "timesheet_id_n". To do so we will have to return a list of line items, as stated above line items must be placed into an array. And so my code to achieve this looks like:
# results = requests.get(url, headers=header)
# results = results.json()
# Dummy result data converted to JSON object after API GET request:
results = {
"results" : {
"timesheets" : {
"timesheet_id_1" : {
"data_1" : "data",
"data_2" : "data",
"data_3" : "data"
},
"timesheet_id_2" : {
"data_1" : "data",
"data_2" : "data",
"data_3" : "data"
},
"timesheet_id_3" : {
"data_1" : "data",
"data_2" : "data",
"data_3" : "data"
}
}
}
}
# Container for my line items. Each element in this list will be executed on separately
return_results = []
results = results.get("results")
results = results.get("timesheets")
for item in results:
return_results.append({"sheet_id" : item, "sheet_data" : results.get(item)})
return return_results
The output of return_results will be an array of dictionary objects. As these dictionary objects are in array Zapier will treat them as line items, additionally because each line item is a dictionary object Zapier will automatically map each value so that they can be independently be used in later action steps. You can see this demonstrated in the output of my trigger zap in the following screenshots:
output 1
output 2
output 3
Hope this helped!
I fell kind of bad after checking all other SO question on this matter and not being able to solve it, but here we go.
When I populate my datatable with a simple $.ajax call it's fine, now I want to do it using sAjaxSource in the oTable definition and I'm facing a problem.
This is the JSON coming from a MySQL call.
[
[
{
"field1": "Relation A",
"field2": 6378,
"field3": 3.97,
"field4": 16.76
},
{
"field1": "Relation B",
"field2": 131626,
"field3": 6.49,
"field4": 11.1
}
],
{
"fieldCount": 0,
"affectedRows": 0,
"insertId": 0,
"serverStatus": 34,
"warningCount": 0,
"message": "",
"protocol41": true,
"changedRows": 0
}
]
I set sAjaxDataProp to '' so it won't look for aaData but still no luck.
So, is there any property I can play with in the oTable definition to make this JSON work?
If not, how should I take care of that JSON?
Let me know if you need to take a look at the full oTable definition. Thank you.
oTable
var oTable = $("#table").dataTable({
'bServerSide': true,
'fnServerParams': function (aoData) {
aoData.push({ "name": "startDate", "value": startDate });
aoData.push({ "name": "endDate", "value": endDate });
},
'sAjaxSource': '/getData',
'sAjaxDataProp': '',
"aoColumns": [
{ "sWidth": "25%","sTitle": "field1", "mDataProp": "field1" },
{ "sWidth": "25%","sTitle": "field2", "mDataProp": "field2"},
{ "sWidth": "25%","sTitle": "field3", "mDataProp": "field3"},
{ "sWidth": "25%","sTitle": "field4", "mDataProp": "field4",
"mRender": function ( data, type, full ) {
return data + ' %';
}
}
]
});
EDIT
The JSON I'm posting is the result of a CALL StoredProcedure from a MySQL db using the node-mysql module. Maybe this has occured to someone before and knows how to treat that data in order to DataTables to make it work.
Is it possible to configure DataTables to hanlde array of objects instead of default array of arrays for aaData propery?
Here is my JSON:
{
"sEcho": 1,
"iTotalRecords": "57",
"iTotalDisplayRecords": "57",
"aaData": [
{
"userid": "tM4Ga0zX",
"password": "ZKwMEiIa",
"vpn_password": "K3lS2yn9",
"status_id": 0,
"expire_account": "0000-01-01 00:00:00",
"created": "2014-02-11 09:29:02"
},
{
"userid": "tM4Ga0zX",
"password": "ZKwMEiIa",
"vpn_password": "K3lS2yn9",
"status_id": 0,
"expire_account": "0000-01-01 00:00:00",
"created": "2014-02-11 09:29:02"
},
{
"userid": "tM4Ga0zX",
"password": "ZKwMEiIa",
"vpn_password": "K3lS2yn9",
"status_id": 0,
"expire_account": "0000-01-01 00:00:00",
"created": "2014-02-11 09:29:02"
}
]
}
Yes, add an aoColumns-definition your initialization :
..
"aoColumns": [
{ "mDataProp": "userid" },
{ "mDataProp": "password" },
{ "mDataProp": "vpn_password" },
{ "mDataProp": "status_id" },
{ "mDataProp": "expire_account" },
{ "mDataProp": "created" }
]
..
fiddle with your data as above -> http://jsfiddle.net/6tK7K/
Important : If you use dataTables v 1.9.2 or above, use the property name "mData" instead. As the docs says :
Note that prior to DataTables 1.9.2 mData was called mDataProp. The
name change reflects the flexibility of this property and is
consistent with the naming of mRender. If 'mDataProp' is given, then
it will still be used by DataTables, as it automatically maps the old
name to the new if required.