My Copy Activity is setup to use a REST Get API call as my source. I keep getting Error Code 2200 Invalid PaginationRule RuleKey=supportRFC5988.
I can call the GET Rest URL using the Web Activity, but this isn't optimal as I then have to pass the output to a stored procedure to load the data to the table. I would much rather use the Copy Activity.
Any ideas why I would get an Invalid PaginationRule error on a call?
I'm using a REST Linked Service with the following properties:
Name: Workday
Connect via integration runtime: link-unknown-self-hosted-ir
Base URL: https://wd2-impl-services1.workday.com/ccx/service
Authentication type: Basic
User name: Not telling
Azure Key Vault for password
Server Certificate Validation is enabled
Parameters: Name:format Type:String Default value:json
Datasource:
"name": "Workday_Test_REST_Report",
"properties": {
"linkedServiceName": {
"referenceName": "Workday",
"type": "LinkedServiceReference",
"parameters": {
"format": "json"
}
},
"folder": {
"name": "Workday"
},
"annotations": [],
"type": "RestResource",
"typeProperties": {
"relativeUrl": "/customreport2/company1/person%40company.com/HIDDEN_BI_RaaS_Test_Outbound"
},
"schema": []
}
}
Copy Activity
{
"name": "Copy Test Workday REST API output to a table",
"properties": {
"activities": [
{
"name": "Copy data1",
"type": "Copy",
"dependsOn": [],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"source": {
"type": "RestSource",
"httpRequestTimeout": "00:01:40",
"requestInterval": "00.00:00:00.010",
"requestMethod": "GET",
"paginationRules": {
"supportRFC5988": "true"
}
},
"sink": {
"type": "SqlMISink",
"tableOption": "autoCreate"
},
"enableStaging": false
},
"inputs": [
{
"referenceName": "Workday_Test_REST_Report",
"type": "DatasetReference"
}
],
"outputs": [
{
"referenceName": "Destination_db",
"type": "DatasetReference",
"parameters": {
"schema": "ELT",
"tableName": "WorkdayTestReportData"
}
}
]
}
],
"folder": {
"name": "Workday"
},
"annotations": []
}
}
Well after posting this, I noticed that in the copy activity code there is a nugget about "supportRFC5988": "true" I switched the true to false, and everything just worked for me. I don't see a way to change this in the Copy Activity GUI
Editing source code and setting this option to false helped!
Related
When I am performing restore of an email message via graph API with a Post request, instead of restoring it is creating a new message with the same data. Because in the JSON createDateTime is being updated although I am passing previous createDataTime.
To elaborate more: I want to restore below mail message which got created in 2018 ( "createdDateTime": "2018-12-31T14:49:42Z") but when I am posting same JSON for restore, createDateTime is being updated automatically to the present date. Which is problem because it's not the restore, it is just like creating new message.
{
"#odata.type": "#microsoft.graph.eventMessageResponse",
"#odata.etag": "W/\"DAAAABYAAABjFtMyIejaSbuRSeM/auJwAAGfpJnO\"",
"id": "AAMkAGZiNGI0MWM4LTQ0NjUtNDUyMy1hOTI2LWNopaTZiMGYxZTBkNQBGAAAAAACaBIVNrajXSj6AQcjiAFBwBjFtMyIejaSbuRSeM-auJwAAAAAAEJAABjFtMyIejaSbuRSeM-auJwAAGf4eRfAAA=",
"createdDateTime": "2018-12-31T14:49:42Z",
"lastModifiedDateTime": "2020-12-31T14:49:46Z",
"changeKey": "DopskAkslaAABjFtMyIejaSbuRSeM/auJwAAGfpJnO",
"categories": [],
"receivedDateTime": "2020-12-31T14:49:43Z",
"sentDateTime": "2020-12-31T14:49:42Z",
"hasAttachments": false,
"internetMessageId": "<MA1PR0101MB207oPF15907003958DB7A58BDD60#MA1PR0101MB2070.INDPRD01.PROD.OUTLOOK.COM>",
"subject": "Accepted: New Year Party",
"bodyPreview": "",
"importance": "normal",
"parentFolderId": "AQMkAGZiNGI0MWM4LTQ0ADY1LTQ1MjMtYTkyNi1jZGU2YjBmMWUwZDUALgAAA5oEhU2tqNdKuqPoBByOIAlkallspspspspspppAAAIBCQAAAA==",
"conversationId": "AAQkAGZiNGI0MWM4LTQ0NjUtNDUyMy1hOTI2LWNkZTZiMGYxZTBkNQAQAEJ5AU8Tk1nklXE3E0XGh2w=",
"conversationIndex": "AQHW34QsrZ0Wy3deoU2Bn2byefNABQ==",
"isDeliveryReceiptRequested": null,
"isReadReceiptRequested": false,
"isRead": true,
"isDraft": false,
"inferenceClassification": "focused",
"meetingMessageType": "meetingAccepted",
"type": "singleInstance",
"isOutOfDate": false,
"isAllDay": false,
"isDelegated": false,
"responseType": "accepted",
"recurrence": null,
"body": {
"contentType": "text",
"content": ""
},
"sender": {
"emailAddress": {
"name": "Mark Rober",
"address": "mark#securemigration.in"
}
},
"from": {
"emailAddress": {
"name": "Mark Rober",
"address": "mark#securemigration.in"
}
},
"toRecipients": [
{
"emailAddress": {
"name": "#Class Yammer",
"address": "ClassYammer#securemigration.in"
}
}
],
"ccRecipients": [],
"bccRecipients": [],
"replyTo": [],
"flag": {
"flagStatus": "notFlagged"
},
"startDateTime": {
"dateTime": "2020-12-31T15:00:00.0000000",
"timeZone": "UTC"
},
"endDateTime": {
"dateTime": "2020-12-31T15:30:00.0000000",
"timeZone": "UTC"
}
}
Please help me with it.
In Azure Data Factory, I need to tap into a HTTP requests via URL using the HTTP connector. I was able to do this as well as setup the dataset. Where I'm having issues is on the pipeline. Here's what I need to do. What is the best way to accomplish this?
Call out to the service base URL and retrieve the header returned of TotalPages.
Using the value for TotalPages, make subsequent requests to the URL with the parameter page (e.g., page=1, page=2, etc.) using the value from TotalPages to form those requests.
Thanks.
Ok. So the issue here is that you cannot nest control structures in Data Factory more than 1 time. The solution is to create two or more pipelines (aka Master and Child).
From the Master pipeline retrieve the number of tasks you will need to execute, and pass them to a for loop. Within the for loop launch for each activity pair a new Child pipeline which will then execute the second activity.
If the Activity is simple enough you can skip the Child Pipeline altogether and do it directly inside the first for loop.
As a Json representation of pipelines in question it should look along these lines:
{
"name": "generic_master",
"properties": {
"activities": [
{
"name": "Web1",
"type": "WebActivity",
"dependsOn": [],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"url": "https://jsonplaceholder.typicode.com/posts/1",
"method": "GET"
}
},
{
"name": "ForEach1",
"type": "ForEach",
"dependsOn": [
{
"activity": "Web1",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"items": {
"value": "#activity('Web1').output",
"type": "Expression"
},
"activities": [
{
"name": "Execute Pipeline1",
"type": "ExecutePipeline",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"pipeline": {
"referenceName": "generic_child",
"type": "PipelineReference"
},
"waitOnCompletion": true
}
}
]
}
}
],
"annotations": []
}
}
{
"name": "generic_child",
"properties": {
"activities": [
{
"name": "Web1",
"type": "WebActivity",
"dependsOn": [],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"url": "https://jsonplaceholder.typicode.com/posts/1",
"method": "POST"
}
}
],
"annotations": []
}
}
In order to read the TotalPages values from the HTTP Request's response, you can use a "Lookup" activity to submit the HTTP request and store the TotalPages value in a variable with the "Set variable" activity.
Actions:
Pipeline level:
create a variable called TotalPages
Lookup activity:
tick the first row only box on the Settings tab
As a source dataset, use the data set defined for your HTTP request
Select the GET method.
Set variable activity:
Select the TotalPages variable on the Variables tab
In the value box, click on "Add dynamic content" and enter something like this: #{activity('GetTotalPages').output.firstRow.RegisterSearch['#TotalPages']}
In my case, the lookup activity is called GetTotalPages, and my HTTP request returns the total number of pages in a RegisterSearch array, under a column name #TotalPages
I'm trying to query the TinkerPop server (hosted inside docker container) via CosmosDB client library, which uses under the hood Gremlin.Net. So I managed to connect it and insert the data, here's intercepted WebSocket request:
!application/vnd.gremlin-v1.0+json{
"requestId": "b64bd2eb-46c3-4095-9eef-768bca2a14ed",
"op": "eval",
"processor": "",
"args": {
"gremlin": "g.addV(\"User\").property(\"UserId\",2).property(\"CustomerId\",1)"
}
}
The response:
{
"requestId": "b64bd2eb-46c3-4095-9eef-768bca2a14ed",
"status": {
"message": "",
"code": 200,
"attributes": {
"host": "/172.19.0.1:38848"
}
},
"result": {
"data": [
{
"id": 0,
"label": "User",
"type": "vertex",
"properties": {}
}
],
"meta": {}
}
}
Problem is that I see those properties when I'm connected via gremlin console
gremlin> g.V().hasLabel("User").has("CustomerId",1).has("UserId",2).limit(1).valueMap()
==>{UserId=[2], CustomerId=[1]}
Also, I'm able to query the TinkerPop server with Gremlin.Net:
!application/vnd.gremlin-v1.0+json{
"requestId": "de35909f-4bc1-4aae-aa5f-28361b3c0933",
"op": "eval",
"processor": "",
"args": {
"gremlin": "g.V().hasLabel(\"User\").has(\"CustomerId\",1).has(\"UserId\",2).limit(1)"
}
}
But it returns a payload with zero-valued ID and without any properties included:
{
"requestId": "de35909f-4bc1-4aae-aa5f-28361b3c0933",
"status": {
"message": "",
"code": 200,
"attributes": {
"host": "/172.19.0.1:38858"
}
},
"result": {
"data": [
{
"id": 0,
"label": "User",
"type": "vertex",
"properties": {}
}
],
"meta": {}
}
}
Tried to swap between GraphSON v1, v2, v3 with no luck. Documentation says that script serializers should include all the properties. Do I have to tweak the config somehow to make this work and return properties?
So it seems that with a version of 3.4 of the Gremlin server ReferenceElementStrategy
was added by default to traversals, to preserve compatibility between binary and script serializers. In our case we wanted to mimic the behavior of the CosmosDB, so to adjust and receive desired behavior just remove the strategy from init script (in our case it was empty-sample.groovy
globals << [g : graph.traversal().withStrategies(ReferenceElementStrategy.instance())]
to
globals << [g : graph.traversal()]
I've successfully created my Knowledgebase using API.
But I forgot to add some alternative questions and metadata for one of the pairs.
I've noticed PATH method in the API to update the Knowledebase, so updating kb is supported.
I've created a payload which looked like this:
{
"add": {
},
"delete": {
},
"update": {
"qnaList": [
{
"id": 1,
"answer": "Answer",
"source": "link_to_source",
"questions": [
"Question 1?",
"Question 2?"
],
"metadata": [
{
"name": "oldMetadata",
"value": "oldMetadata"
},
{
"name": "newlyAddedMetaData",
"value": "newlyAddedMetaData"
}
]
}]}
}
I get back the following response HTTP 202 Accepted:
{
"operationState": "NotStarted",
"createdTimestamp": "2018-05-21T07:46:52Z",
"lastActionTimestamp": "2018-05-21T07:46:52Z",
"userId": "user_uuid",
"operationId": "operation_uuid"
}
So, looks like it worked. But in reality, this request doesn't take any affect.
When I check operation details, it returns me the following:
{
"operationState": "Succeeded",
"createdTimestamp": "2018-05-21T07:46:52Z",
"lastActionTimestamp": "2018-05-21T07:46:54Z",
"resourceLocation": "/knowledgebases/kb_uuid",
"userId": "user_uuid",
"operationId": "operation_uuid"
}
What am I doing wrong? And how should I update my kb via API properly?
Please help
I had the same problem, I discovered that it was necessary to have all the data of the json even if they were not used.
In your case you need "name" and "urls" in the "update" section and "Delete" in "update/qnaList/questions" section:
{
"add": {},
"delete": {},
"update": {
"name": "nameofKbBase", //this
"qnaList": [
{
"id": 2370,
"answer": "DemoAnswerEdit",
"source": "CustomSource",
"questions": {
"add": [
"DemoQuestionEdit"
],
"delete": [] //this
},
"metadata": { }
}
],
"urls": [] //this
}
}
Recently I have started learning Loopback. I have tried to add required, min and max.
Here is my remote method in json file with parameters :
{
"name": "registration",
"methods": {
"registrationIn": {
"accepts": [
{
"arg": "firstname",
"type": "string",
"min": 1,
"max": 25, // same for here
"required": true,
"description": "Firstname of the person.",
"http": {
"source": "form"
}
}
{
"arg": "mobile",
"type": "number",
"min": 1, // since required it correct but not checking i removed required
"max": 10, // not working
"required": true, // working
"description": "",
"http": {
"source": "form"
}
}
],
"returns": [],
"description": "This method used to registration.",
"http": [{
"path": "/registrationIn",
"verb": "post"
}]
}
}
required property is working fine but min and max is not working.
Can anyone guide me to where i am doing wrong ?
Thanks
Please refer this link validation
You can add validation on model.
module.exports = function(user) {
user.validatesLengthOf('password', {min: 5, message: {min: 'Password is too short'}});
}