Related
I need to set datatype for Additional Column with Dynamic Content in Sink in ADF
By default its taking nvarchar(max) from Json obj but I need bigInt
Below is a Json Obj which create table with Additional column
{
"source": {
"type": "SqlServerSource",
"additionalColumns": [
{
"name": "ApplicationId",
"value": 3604509277250831000
}
],
"sqlReaderQuery": "SELECT * from Table A",
"queryTimeout": "02:00:00",
"isolationLevel": "ReadUncommitted",
"partitionOption": "None"
},
"sink": {
"type": "AzureSqlSink",
"writeBehavior": "insert",
"sqlWriterUseTableLock": false,
"tableOption": "autoCreate",
"disableMetricsCollection": false
},
"enableStaging": false,
"translator": {
"type": "TabularTranslator",
"typeConversion": true,
"typeConversionSettings": {
"allowDataTruncation": true,
"treatBooleanAsNumber": false
}
}
}
ADF Configuration
After create table Database - column with datatype
If I convert Dynamic content into Int
#int(pipeline().parameters.application.applicationId)
Then getting below warning
Please let me know how can I set Datatype in ADF
I also tried the same and getting same result.
By default its taking nvarchar(max) from Json obj but I need bigInt
To resolve this when you add additional column in your source data set and in Mapping click onimport schema it will import the schema of the source and also give you additional column in schema you have to change the type of the column as Int64 as shown in below image. in below image you can see after name there is additional means it is an additional column.
After this run your pipeline, It will create additional column with data type bigint .
{
"name": "pipeline2",
"properties": {
"activities": [
{
"name": "Copy data1",
"type": "Copy",
"dependsOn": [],
"policy": {
"timeout": "0.12:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"source": {
"type": "JsonSource",
"additionalColumns": [
{
"name": "name",
"value": {
"value": "#pipeline().parameters.demo.age",
"type": "Expression"
}
}
],
"storeSettings": {
"type": "AzureBlobFSReadSettings",
"recursive": true,
"enablePartitionDiscovery": false
},
"formatSettings": {
"type": "JsonReadSettings"
}
},
"sink": {
"type": "AzureSqlSink",
"writeBehavior": "insert",
"sqlWriterUseTableLock": false,
"tableOption": "autoCreate",
"disableMetricsCollection": false
},
"enableStaging": false,
"translator": {
"type": "TabularTranslator",
"mappings": [
{
"source": {
"path": "$['taskId']"
},
"sink": {
"name": "taskId",
"type": "String"
}
},
{
"source": {
"path": "$['taskObtainedScore']"
},
"sink": {
"name": "taskObtainedScore",
"type": "String"
}
},
{
"source": {
"path": "$['multiInstance']"
},
"sink": {
"name": "multiInstance",
"type": "String"
}
},
{
"source": {
"path": "$['name']"
},
"sink": {
"name": "name",
"type": "Int64"
}
}
],
"collectionReference": ""
}
},
"inputs": [
{
"referenceName": "Json1",
"type": "DatasetReference"
}
],
"outputs": [
{
"referenceName": "AzureSqlTable1",
"type": "DatasetReference"
}
]
}
],
"parameters": {
"demo": {
"type": "object",
"defaultValue": {
"name": "John",
"age": 30,
"isStudent": true
}
}
},
"annotations": []
}
}
OUTPUT:
I am working on a Sunburst that would eventually take input data without a configured parent-child structure but a column based data so I employed a nest transform instead of a stratify as in the given example:
https://vega.github.io/editor/#/gist/c1eeb9142fd7f611513f5a4edf7e180e/spec.json
The problem with it, is that it generates internal nodes even if some fields are empty as long as my data object got value available for leaf nodes.
How can i transform my data to get a visual as in the following example
https://vega.github.io/editor/#/gist/112c45cfed3b9c9013ea0b63a318292f/spec.json
The idea is to "transform" the column based data into a parent, child structure in order to apply a Stratify transform.
Starting data:
{
"value": 140,
"level1": "France",
"level2": "Reins",
"level3": "Jean Monnet",
"level4": "rue 3 ",
"level5": "no",
"level6": "a level"
}
First group by all the levels or depth of the sunburst according to your value or quantity.
"transform": [
{
"type": "aggregate",
"groupby": [
"level1",
"level2",
"level3",
"level4",
"level5",
"level6"
],
"fields": [
"value"
],
"ops": [
"sum"
],
"as": [
"value"
]
}
]
Secondly, we need to have a root note for stratify, so based on available data we can transform our data as following to define our root node having level1 elements as child nodes:
{
"name": "isolatedLevel1",
"source": [
"mainData"
],
"transform": [
{
"type": "project",
"fields": [
"level1",
"currentNode",
"value"
],
"as": [
"currentNode",
"parent",
"value"
]
},
{
"type": "aggregate",
"groupby": [
"parent",
"currentNode"
],
"fields": [
"value"
],
"ops": [
"sum"
],
"as": [
"value"
]
},
{
"type": "formula",
"as": "parent",
"expr": "!isDefined(datum.parent)?'rootID':datum.parent"
},
{
"type": "project",
"fields": [
"currentNode",
"parent",
"value"
],
"as": [
"currentNode",
"parent",
"value"
]
},
{
"type": "filter",
"expr": "datum.currentNode"
}
]
},
{
"name": "rootNode",
"source": "isolatedLevel1",
"transform": [
{
"type": "project",
"fields": [
"parent",
"level2"
],
"as": [
"currentNode",
"parent"
]
},
{
"type": "aggregate",
"groupby": ["currentNode","parent"]
},
{
"type": "project",
"fields": [
"currentNode",
"parent"
],
"as": [
"currentNode",
"parent"
]
}
]
},
{
"name": "isolatedLevel1MergedWithParent",
"source": [
"isolatedLevel1",
"rootNode"
]
},
Then, the idea is to determine for the upcoming levels, the parent and child between 2 levels.
We can apply this data transform and repeat it till the last level of our data.
{
"name": "isolatedLevel2",
"source": "mainData",
"transform": [
{
"type": "project",
"fields": [
"level1",
"level2",
"value"
],
"as": [
"parent",
"currentNode",
"value"
]
},
{
"type": "aggregate",
"groupby": [
"parent",
"currentNode"
],
"fields": [
"value"
],
"ops": [
"sum"
],
"as": [
"value"
]
},
{
"type": "project",
"fields": [
"currentNode",
"parent",
"value"
],
"as": [
"currentNode",
"parent",
"value"
]
},
{
"type": "filter",
"expr": "datum.currentNode"
},
{
"type": "filter",
"expr": "datum.parent"
}
]
}
Finally, we merge all these "decomposed" transforms all together to get back our data, transformed into a parent child hierarchy, with values associated to its nodes only. This is a way to suit your data to a Stratify VEGA transform.
{
"name": "mergedlevelsFinal",
"source": [
"isolatedLevel1",
"isolatedLevel2",
"isolatedLevel3",
"isolatedLevel4",
"isolatedLevel5",
"isolatedLevel6",
"rootNode"
],
"transform": [
{
"type": "formula",
"as": "value",
"expr": "!indata('parents','parent',datum.currentNode)? datum.value:''"
}
]
}
The hard part gone, we can now apply our stratify to the transformed data as we want for our sunburst.
{
"name": "tree",
"source": [
"mergedlevelsFinal"
],
"transform": [
{
"type": "stratify",
"key": "currentNode",
"parentKey": "parent"
},
{
"type": "partition",
"field": "value",
"sort": {
"field": "value"
},
"size": [
{
"signal": "2 * PI"
},
{
"signal": "width / 3"
}
],
"as": [
"a0",
"r0",
"a1",
"r1",
"depth",
"children"
]
},
{
"type": "formula",
"expr": "!datum.children?datum.value:''",
"as": "value"
}
]
}
Full Spec Solution available here https://vega.github.io/editor/#/gist/0f66d06894b61f8ba131f13f564e8c1f/spec.json
I have an API returning response based on input type.
For a single filter I get
{
"intervalData": [
{
"timestamp": "2021-05-25T06:00:00.000Z",
"result": {
"Attempts": 309194
}
}
],
"intervalTotals": {
"Attempts": 4719471
}
}
For multiple filter I get
{
"multiFilterData": [
{
"filter": {
"id": 111111
},
"intervalData": {
"timestamp": "2021-05-25T20:41:39.000Z",
"result": {
"Attempts": 7902
}
}
},
{...}
]
}
That means the response will either have (intervalData,intervalTotals) OR multiFilterData.
Question:
Is it possible to have a generic schema to handle this or must I have two separate schemas?
How to make sure that if one combination is present other should not be there?
I tried with below schema on https://www.jsonschemavalidator.net/. The validation fails if there is some other keys listed under the required, unfortunately the validation succeeds if one of the entry from oneOf is missing. EDIT1 below
{
"$schema": "http://json-schema.org/draft-07/schema",
"type": "object",
"oneOf": [
{
"required": [
"intervalData",
"intervalTotals",
]
},
{
"required": [
"multiFilterData",
]
}
],
"properties": {
"intervalData": {
"type": "array",
"additionalItems": false,
"items": {
"type": "object",
"required": [
"timestamp",
"result"
],
"properties": {
...
},
"additionalProperties": false
}
},
"intervalTotals": {
"type": "object",
"required": [
"Attempts"
],
"properties": {
"Attempts": {
"type": "integer"
}
},
"additionalProperties": false
},
"multiFilterData": {
"type": "array",
"additionalItems": false,
"items": {
"type": "object",
"required": [
"filter",
"intervalData"
],
"properties": {
"filter": {
"type": "object",
"required": [
"id"
],
"properties": {
"id": {
"type": "integer"
}
},
"additionalProperties": false
},
"intervalData": {
"type": "object",
"required": [
"timestamp",
"result"
],
"properties": {
"timestamp": {
"type": "string"
},
"result": {
"type": "object",
"required": [
"Attempts"
],
"properties": {
...
},
"additionalProperties": false
}
},
"additionalProperties": false
}
},
"additionalProperties": false
}
}
},
"additionalProperties": false
}
EDIT 1: This schema also passes even though the first required is missing an element on the list.
{
"$schema": "http://json-schema.org/draft-07/schema",
"type": "object",
"oneOf": [
{
"required": [
"intervalData"
]
},
{
"required": [
"multiFilterData",
]
}
],
"properties": {
"intervalData": {...},
"intervalTotals": {...},
"multiFilterData": {...}
},
"additionalProperties": false
}
I want to create a JSON schema for an object in which one of the attributes is restricted to multiple sets of enums.
For example:
{
"data": {
"type": "myObject",
"attributes": {
"states": [
"Washington",
"Oregon",
"California"
]
}
}
}
is a valid JSON object against the schema.
And
{
"data": {
"type": "myObject",
"attributes": {
"states": [
"British Columbia",
"Alberta",
"Ontario"
]
}
}
}
is also a valid JSON object agains the schema
BUT,
{
"data": {
"type": "myObject",
"attributes": {
"states": [
"Washington",
"Oregon",
"Alberta"
]
}
}
}
is NOT a valid JSON object against the schema.
I have tried the following schema definition:
{
"type": "object",
"properties": {
"data": {
"type": "object",
"properties": {
"type": {
"type": "string"
},
"attributes": {
"type": "object",
"properties": {
"states": {
"type": "array",
"items": {
"oneOf": [
{
"enum": ["Washington","Oregon","California"],
"description": "United States"
},
{
"enum": ["British Columbia","Alberta", "Ontario"],
"description": "Canada"
}
]
},
"description": "Filter by states"
}
}
}
}
}
}
}
But with this schema above this is still considered valid:
{
"data": {
"type": "myObject",
"attributes": {
"states": [
"Washington",
"Oregon",
"Alberta"
]
}
}
}
BTW, you can use this for testing whether a JSON object conforms to a schema: https://www.jsonschemavalidator.net/
Thank you!
You need to invert the order of the oneOf and the items keywords, so that the same oneOf clause is used for all items:
...
"states": {
"type": "array",
"oneOf": [
{
"items": {
"enum": ["Washington","Oregon","California"],
"description": "United States"
}
},
{
"items": {
"enum": ["British Columbia","Alberta", "Ontario"],
"description": "Canada"
}
}
]
},
...
I have a JSON that can have either an IBAN account number, in which case a BIC is required, or an account_number or both of them.
So, this is valid (only account_number):
"bankaccount_data": [
{
"bic": null,
"iban": null,
"account_name": "Bankgiro",
"account_number": "12345-6789",
"bank_name": "Bankgiro",
"type": "Bankgiro"
}
]
And this is valid with iban and bic:
"bankaccount_data": [
{
"bic": "BANKBIC",
"iban": "SE0123456789",
"account_name": "Bankgiro",
"account_number": null,
"bank_name": "Bankgiro",
"type": "Bankgiro"
}
]
Even this with both account_number and iban/bic:
"bankaccount_data": [
{
"bic": "BANKBIC",
"iban": "SE0123456789",
"account_name": "Bankgiro",
"account_number": "12345-6789",
"bank_name": "Bankgiro",
"type": "Bankgiro"
}
]
My problem is with requiring BOTH iban and bic if either/or exists. I have this schema which is not doing that but should "illustrate" my need:
"bankaccount_data": {
"type": "array",
"items": [
{
"type": "object",
"properties": {
"account_name": { "type": "string" },
"bank_name": { "type": [ "string", "null" ] },
"type": { "type": "string" }
},
"required": [ "type" ],
"anyOf": [
{
"type": "object",
"properties": {
"bic": { "type": "string" },
"iban": { "type": "string" }
},
"required": [ "bic", "iban" ],
"dependencies": {
"iban": [
"bic"
],
"bic": [
"iban"
]
}
},
{
"type": "object",
"properties": {
"account_number": { "type": "string" }
},
"required": [ "account_number" ]
}
]
}
]
}
I wasn't quite sure, but I think you're asking for, if iban or bic is present, then the other must also be present.
You had the right ideas here. However it's complicated by the fact that your values can be null as opposed to simply not present.
dependencies only checks the properties are present, not anything to do with their value. null is still a value. dependencies cannot help you in your situation, as you always expect the keys to be present in the object.
First let's look at the solution.
{
"anyOf": [
{
"properties": {
"bic": { "type": "string" },
"iban": { "type": "string" },
"account_number": { "enum": [null]}
},
"required": [
"iban",
"bic"
]
},
{
"required": [
"account_number"
],
"properties": {
"bic": { "enum": [null] },
"iban": { "enum": [null] },
"account_number": { "type": "string" }
}
}
]
}
https://jsonschema.dev/s/dg0CY
The reason you need this sort of duplication, is you need to fully express the condition that your checking.
The values of an anyOf array are subschemas. The values are full schemas on their own.
Taking your values for anyOf, anyOf[0] does express the constraint that your looking for, however anyOf[1] says nothing about iban or bic, and so "any of" the schemas is considered valid. Each value in anyOf is not aware of the contense of the other; only the results are combined.
JSON Schema is constraints based, so anything NOT expressed, is allowed.
In the solution I've provided above, each schema in anyOf checks fully for the required condition.
For your example, it works in draft-07 the same as draft-04, so the demo is the same.