VEGA Sunburst using table based raw data - vega

I am working on a Sunburst that would eventually take input data without a configured parent-child structure but a column based data so I employed a nest transform instead of a stratify as in the given example:
https://vega.github.io/editor/#/gist/c1eeb9142fd7f611513f5a4edf7e180e/spec.json
The problem with it, is that it generates internal nodes even if some fields are empty as long as my data object got value available for leaf nodes.
How can i transform my data to get a visual as in the following example
https://vega.github.io/editor/#/gist/112c45cfed3b9c9013ea0b63a318292f/spec.json

The idea is to "transform" the column based data into a parent, child structure in order to apply a Stratify transform.
Starting data:
{
"value": 140,
"level1": "France",
"level2": "Reins",
"level3": "Jean Monnet",
"level4": "rue 3 ",
"level5": "no",
"level6": "a level"
}
First group by all the levels or depth of the sunburst according to your value or quantity.
"transform": [
{
"type": "aggregate",
"groupby": [
"level1",
"level2",
"level3",
"level4",
"level5",
"level6"
],
"fields": [
"value"
],
"ops": [
"sum"
],
"as": [
"value"
]
}
]
Secondly, we need to have a root note for stratify, so based on available data we can transform our data as following to define our root node having level1 elements as child nodes:
{
"name": "isolatedLevel1",
"source": [
"mainData"
],
"transform": [
{
"type": "project",
"fields": [
"level1",
"currentNode",
"value"
],
"as": [
"currentNode",
"parent",
"value"
]
},
{
"type": "aggregate",
"groupby": [
"parent",
"currentNode"
],
"fields": [
"value"
],
"ops": [
"sum"
],
"as": [
"value"
]
},
{
"type": "formula",
"as": "parent",
"expr": "!isDefined(datum.parent)?'rootID':datum.parent"
},
{
"type": "project",
"fields": [
"currentNode",
"parent",
"value"
],
"as": [
"currentNode",
"parent",
"value"
]
},
{
"type": "filter",
"expr": "datum.currentNode"
}
]
},
{
"name": "rootNode",
"source": "isolatedLevel1",
"transform": [
{
"type": "project",
"fields": [
"parent",
"level2"
],
"as": [
"currentNode",
"parent"
]
},
{
"type": "aggregate",
"groupby": ["currentNode","parent"]
},
{
"type": "project",
"fields": [
"currentNode",
"parent"
],
"as": [
"currentNode",
"parent"
]
}
]
},
{
"name": "isolatedLevel1MergedWithParent",
"source": [
"isolatedLevel1",
"rootNode"
]
},
Then, the idea is to determine for the upcoming levels, the parent and child between 2 levels.
We can apply this data transform and repeat it till the last level of our data.
{
"name": "isolatedLevel2",
"source": "mainData",
"transform": [
{
"type": "project",
"fields": [
"level1",
"level2",
"value"
],
"as": [
"parent",
"currentNode",
"value"
]
},
{
"type": "aggregate",
"groupby": [
"parent",
"currentNode"
],
"fields": [
"value"
],
"ops": [
"sum"
],
"as": [
"value"
]
},
{
"type": "project",
"fields": [
"currentNode",
"parent",
"value"
],
"as": [
"currentNode",
"parent",
"value"
]
},
{
"type": "filter",
"expr": "datum.currentNode"
},
{
"type": "filter",
"expr": "datum.parent"
}
]
}
Finally, we merge all these "decomposed" transforms all together to get back our data, transformed into a parent child hierarchy, with values associated to its nodes only. This is a way to suit your data to a Stratify VEGA transform.
{
"name": "mergedlevelsFinal",
"source": [
"isolatedLevel1",
"isolatedLevel2",
"isolatedLevel3",
"isolatedLevel4",
"isolatedLevel5",
"isolatedLevel6",
"rootNode"
],
"transform": [
{
"type": "formula",
"as": "value",
"expr": "!indata('parents','parent',datum.currentNode)? datum.value:''"
}
]
}
The hard part gone, we can now apply our stratify to the transformed data as we want for our sunburst.
{
"name": "tree",
"source": [
"mergedlevelsFinal"
],
"transform": [
{
"type": "stratify",
"key": "currentNode",
"parentKey": "parent"
},
{
"type": "partition",
"field": "value",
"sort": {
"field": "value"
},
"size": [
{
"signal": "2 * PI"
},
{
"signal": "width / 3"
}
],
"as": [
"a0",
"r0",
"a1",
"r1",
"depth",
"children"
]
},
{
"type": "formula",
"expr": "!datum.children?datum.value:''",
"as": "value"
}
]
}
Full Spec Solution available here https://vega.github.io/editor/#/gist/0f66d06894b61f8ba131f13f564e8c1f/spec.json

Related

Validation issue on JSON schema #jsonschema

enter image description hereI am trying to create a JSON schema which is expecting inputs from 3 sources (file/Table/Kafka message), at a time any one of them will get the data from external source.
Issue :
I am using "anyOf" keyword in JSON but when I am applying "anyOf" and testing the negative scenario that is removing one of required field from the input properties, it's not giving any validation error, which ideally should have given that required field is missing.
**Expectation: **
Need to get input from any one of the source (file/table/message). If external party sends data in the file format, file portion from JSON schema should get executed, similarly if gets data from table or kafka message then relevant portion from JSON schema should get executed.
JSON SCHEMA CODE:
{
"$schema": "https://json-schema.org/draft/2019-09/schema",
"type": "object",
"title": "Job",
"properties": {
"unique_name": {
"type": "string",
"format": "regex",
"pattern": "[a-zA-Z0-9_]+"
},
"processing_instructions_file": {
"type": "string",
"examples": [
"/path/to/some/file/processing123.yaml"
]
},
"variables": {
"type": "array",
"additionalItems": true,
"items": {
"type": "object",
"properties": {
"name": {
"type": "string",
"examples": [
"environment_prefix"
]
},
"value": {
"type": "string",
"examples": [
"DEV"
]
}
},
"additionalProperties": true,
"required": [
"name",
"value"
]
}
},
"inputs": {
"type": "array",
"additionalItems": true,
"items": {
"type": "object",
"properties": {
"is_stream": {
"type": "boolean",
"examples": [
true
],
"default": false
}
},
"additionalProperties": true,
"anyOf": [
{
"type": "object",
"properties": {
"file": {
"type": "object",
"properties": {
"folder_structure": {
"type": "string",
"examples": [
"$environment_prefix\\$$yyyy-MMM-dd"
]
},
"naming_convention": {
"type": "string",
"examples": [
"STATIC_FILE_NAME:$$yyyy_MMM-dd"
]
},
"format type": {
"type": "string",
"enum": [
"CSV",
"EBCDIC"
],
"examples": [
"CSV"
]
},
"format_options": {
"type": "array",
"additionalItems": true,
"items": {
"type": "object",
"properties": {
"option_name": {
"type": "string",
"examples": [
"generate_record_id"
]
},
"option_value": {
"type": [
"string",
"boolean"
],
"examples": [
true
]
}
},
"additionalProperties": true,
"required": [
"option_name",
"option_value"
]
}
}
},
"additionalProperties": true,
"if": {
"properties": {
"format type": {
"type": "string",
"enum": [
"CSV",
"EBCDIC"
],
"examples": [
"CSV"
],
"const": "EBCDIC"
}
},
"required": [
"format type"
]
},
"then": {
"properties": {
"schema_location": {
"type": "string",
"examples": [
"some_path"
]
}
},
"required": [
"schema_location"
]
},
"else": {
"properties": {
"schema_location": {
"type": "string",
"examples": [
"some_path"
]
}
}
},
"required": [
"folder_structure",
"naming_convention",
"format type",
"format_options"
]
}
},
"additionalProperties": true
},
{
"type": "object",
"properties": {
"table": {
"type": "object",
"properties": {
"name": {
"type": "string",
"examples": [
"tbl_employees"
]
},
"jdbc_url": {
"type": "string",
"examples": [
"orcl#thinXXX"
]
},
"jdbc_library_jar": {
"type": "string",
"examples": [
"orcl.jar"
]
},
"schema": {
"type": "string",
"examples": [
"xxxxx"
]
},
"info_date_from_column_name": {
"type": "string",
"examples": [
"ASOF_DATE"
]
},
"info_date_to_column_name": {
"type": "string",
"examples": [
"END_DATE"
]
},
"primary_key_column_name": {
"type": "array",
"additionalItems": true,
"items": {
"type": "string"
}
}
},
"additionalProperties": true,
"required": [
"name",
"jdbc_url",
"schema"
]
}
},
"additionalProperties": true
},
{
"type": "object",
"properties": {
"message": {
"type": "object",
"properties": {
"topic_name": {
"type": "string",
"examples": [
"XXXXXX"
]
},
"group_id": {
"type": "string",
"examples": [
"XXXXX"
]
},
"port": {
"type": "integer",
"examples": [
8081
]
},
"kafka_cluster_ip": {
"type": "string",
"examples": [
"1.1.1.1"
]
},
"kafka_cluster_schema_registry": {
"type": "string",
"examples": [
"1.1.1.1"
]
}
},
"additionalProperties": true,
"required": [
"topic_name",
"group_id",
"port",
"kafka_cluster_ip",
"kafka_cluster_schema_registry"
]
}
},
"additionalProperties": true
}
]
}
},
"outputs": {
"type": "array",
"additionalItems": true,
"items": {
"type": "object",
"properties": {
"name": {
"type": "string",
"examples": [
"output_3"
]
},
"is_stream": {
"type": "boolean",
"examples": [
true
]
},
"spark_save_mode": {
"type": "string",
"examples": [
"append"
]
},
"file": {
"type": "object",
"properties": {
"root_path": {
"type": "string",
"examples": [
"some_path"
]
},
"folder_structure": {
"type": "string",
"examples": [
"$environment_prefix\\$$yyyy-MMM-dd"
]
},
"naming_convention": {
"type": "string",
"examples": [
"STATIC_FILE_NAME:$$yyyy_MMM-dd"
]
},
"partition_by": {
"type": "array",
"additionalItems": true,
"items": {
"type": "string"
}
},
"format": {
"type": "object",
"properties": {
"format_type": {
"type": "string",
"examples": [
"CSV"
]
},
"format_options": {
"type": "array",
"additionalItems": true,
"items": {
"type": "object",
"properties": {
"option_name": {
"type": "string",
"examples": [
"include_header"
]
},
"option_value": {
"type": [
"string",
"boolean"
],
"examples": [
true
]
}
},
"additionalProperties": true,
"required": [
"option_name",
"option_value"
]
}
}
},
"additionalProperties": true,
"required": [
"format_type",
"format_options"
]
}
},
"additionalProperties": true,
"required": [
"root_path",
"folder_structure",
"naming_convention",
"partition_by",
"format"
]
},
"mongo": {
"type": "object",
"properties": {
"mongo_spark_connector_options": {
"type": "array",
"additionalItems": true,
"items": {
"type": "object",
"properties": {
"option_name": {
"type": "string",
"examples": [
"format"
]
},
"option_value": {
"type": "string"
}
},
"additionalProperties": true,
"required": [
"option_name",
"option_value"
]
}
}
},
"additionalProperties": true,
"required": [
"mongo_spark_connector_options"
]
},
"table": {
"type": "object",
"properties": {
"name": {
"type": "string",
"examples": [
"tbl_employees"
]
},
"jdbc_url": {
"type": "string",
"examples": [
"orcl#thin:XXX"
]
},
"jdbc_library_jar": {
"type": "string",
"examples": [
"orcl.jar"
]
},
"schema": {
"type": "string",
"examples": [
"xxxxx"
]
},
"info_date_from_column_name": {
"type": "string",
"examples": [
"ASOF_DATE"
]
},
"info_date_to_column_name": {
"type": "string",
"examples": [
"END_DATE"
]
},
"primary_key_column_name": {
"type": "array",
"additionalItems": true,
"items": {
"type": "string"
}
}
},
"additionalProperties": true,
"required": [
"name",
"jdbc_url",
"jdbc_library_jar",
"schema",
"info_date_from_column_name",
"info_date_to_column_name",
"primary_key_column_name"
]
},
"message": {
"type": "object",
"properties": {
"topic_name": {
"type": "string",
"examples": [
"XXXXXX"
]
},
"group_id": {
"type": "string",
"examples": [
"XXXXX"
]
},
"port": {
"type": "integer",
"examples": [
8081
]
},
"kafka_cluster_ip": {
"type": "string",
"examples": [
"1.1.1.1"
]
},
"kafka_cluster_schema_registry": {
"type": "string",
"examples": [
"1.1.1.1"
]
}
},
"additionalProperties": true,
"required": [
"topic_name",
"group_id",
"port",
"kafka_cluster_ip",
"kafka_cluster_schema_registry"
]
}
},
"additionalProperties": true,
"required": [
"name",
"is_stream"
]
}
}
},
"additionalProperties": true,
"required": [
"unique_name",
"processing_instructions_file",
"inputs",
"outputs"
]
}
JSON DATA
{
"unique_name": "aXeTGImcM6PrCalRpGjLigbj1puXXvK",
"processing_instructions_file": "/path/to/some/file/processing123.yaml",
"variables": [
{
"name": "environment_prefix",
"value": "DEV"
}
],
"inputs": [
{
"is_stream": false,
"file": {
"folder_structure": "$environment_prefix\\$$yyyy-MMM-dd",
"naming_convention": "STATIC_FILE_NAME:$$yyyy_MMM-dd",
"format type": "CSV",
"format_options": [
{
"option_name": "generate_record_id",
"option_value": "Lorem"
}
],
"schema_location": "some_path"
}
}
],
"outputs": [
{
"name": "output_3",
"is_stream": true,
"spark_save_mode": "append",
"file": {
"root_path": "some_path",
"folder_structure": "$environment_prefix\\$$yyyy-MMM-dd",
"naming_convention": "STATIC_FILE_NAME:$$yyyy_MMM-dd",
"partition_by": [
"Lorem"
],
"format": {
"format_type": "CSV",
"format_options": [
{
"option_name": "include_header",
"option_value": "Lorem"
}
]
}
},
"mongo": {
"mongo_spark_connector_options": [
{
"option_name": "format",
"option_value": "Lorem"
}
]
},
"table": {
"name": "tbl_employees",
"jdbc_url": "orcl#thin:XXX",
"jdbc_library_jar": "orcl.jar",
"schema": "xxxxx",
"info_date_from_column_name": "ASOF_DATE",
"info_date_to_column_name": "END_DATE",
"primary_key_column_name": [
"Lorem"
]
},
"message": {
"topic_name": "XXXXXX",
"group_id": "XXXXX",
"port": 8081,
"kafka_cluster_ip": "1.1.1.1",
"kafka_cluster_schema_registry": "1.1.1.1"
}
}
]
}

GraphDB - 1 vertex having multiple properties

Could you please help me understand if we can have multiple array properties in 1 vertex.. For example,
"id": "CU10611973PH",
"label": "Phone-Home",
"type": "vertex",
"properties": {
"PhonePreference": [
{
"id": "c63cf5c9-329b-456e-b94e-0c500587780f",
"value": "Primary"
}
],
"PhoneType": [
{
"id": "11f241d8-8e84-479b-b6a4-aeab7e039665",
"value": "Home"
}
],
"PhoneNumber": [
{
"id": "5795e2ec-6d8c-424c-b42d-f01b587caeca",
"value": "22222222222"
}
],
"pk": [
{
"id": "CU10611973PH|pk",
"value": "pk"
}
]
}
},
{
"id": "CU10611973PC",
"label": "Phone-cell",
"type": "vertex",
"properties": {
"PhonePreference": [
{
"id": "488c54b2-f256-4053-adc5-c07ec2e1a629",
"value": "Primary"
}
],
"PhoneType": [
{
"id": "2f5263e7-f42b-487f-a185-338682a68fdd",
"value": "Cell"
}
],
"PhoneNumber": [
{
"id": "a967516d-986c-4eac-929a-b86824677922",
"value": "123456789"
}
],
"pk": [
{
"id": "CU10611973PC|pk",
"value": "pk"
So here we are having 2 vertex, one for Phone-Home and one for Phone-Cell. Is there a way to make it come under 1 vertex with 2 set of properties .

Postgresql using json_build_obj

I'm trying to get a column of geometries from a table with PostgreSQL and PostGIS
SELECT json_build_object(
'type', 'FeatureCollection',
'features', json_agg(ST_AsGeoJSON(t.*)::json)
) FROM (SELECT position FROM permit) as t
This returns an Array with a key json_build_object.
[
{
"json_build_object": {
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"geometry": {
"type": "Point",
"coordinates": [
-97.74318277,
30.25239259
]
},
"properties": {}
}
]
}
}
]
I would like a single FeatureCollection object.
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"geometry": {
"type": "Point",
"coordinates": [
-97.74318277,
30.25239259
]
},
"properties": {}
}
]
}
Is this the expected output of the query? And how can I get a clean FeatureCollection otherwise?

Angular6 JSON schema form for Array of Items

I am trying out Angular6 JSON form for my application and stuck in the issue of having array schema
The basic layout looks like
{
"schema": {
"type": "array",
"properties": {
"type": { "type": "string" },
"number": { "type": "string" },
}
},
"layout": [
{
"type": "array",
"items": [ {
"type": "div",
"displayFlex": true,
"flex-direction": "row",
"items": [
{ "key": "type", "flex": "1 1 50px",
"notitle": true, "placeholder": "Type"
},
{ "key": "number", "flex": "4 4 200px",
"notitle": true, "placeholder": "Phone Number"
}
]
} ]
}
],
"data": [
{ "type": "cell", "number": "702-123-4567" },
{ "type": "work", "number": "702-987-6543" }
]
}
But I am not getting the expected outcome, that is Form is prefilled with the data
[
{ "type": "cell", "number": "702-123-4567" },
{ "type": "work", "number": "702-987-6543" }
]
Refer: https://hamidihamza.com/Angular6-json-schema-form/
Based on your code there are some parts you may need to revisit.
The schema type should be either an object or boolean based on the documentation http://json-schema.org/latest/json-schema-core.html
Within the schema section, it seems that you want the type and number to be your properties of a JSON instance. Having this you can only pass one instance of data to the framework to fill in your properties because the framework cannot decide on which value to use for your property of type string.
In case of looking for having an array of type and number, you can have a property like "phone number" with the type array. below is an example from angular6-json-schema flex layout example which I think you had as your reference.
"schema": {
...
"phone_numbers": {
"type": "array",
"items": {
"type": "object",
"properties": {
"type": { "type": "string", "enum": [ "cell", "home", "work" ] },
"number": { "type": "string" }
},
"required": [ "type", "number" ]
}
And pass your data having something as follows:
"data": {
...
"phone_numbers": [
{ "type": "cell", "number": "702-123-4567" },
{ "type": "work", "number": "702-987-6543" }
],
}

How do I create a role assignment for a website in an Azure resource manager deployment template?

I tried what seemed to be the obvious way with a nested resource:
{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"resources": [
{
"type": "Microsoft.Web/sites",
"apiVersion": "2016-08-01",
"name": "testapp",
"location": "[resourceGroup().location]",
"resources": [
{
"type": "Microsoft.Web/sites/providers/Microsoft.Authorization/roleassignments",
"apiVersion": "2015-07-01",
"name": "<guid>",
"dependsOn": [
"[resourceId('Microsoft.Web/sites/', 'testapp')]"
],
"properties": {
"roleDefinitionId": "[resourceId('Microsoft.Authorization/roleDefinitions', 'b24988ac-6180-42a0-ab88-20f7382dd24c')]",
"principalId": "<guid>",
}
}
]
}
]
}
But that didn't work - it create a role assignment, but at the resource group level rather than at the website level. (Is that a bug?)
If I try explicitly specifying a scope:
{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"resources": [
{
"type": "Microsoft.Web/sites",
"apiVersion": "2016-08-01",
"name": "testapp",
"location": "[resourceGroup().location]",
"resources": [
{
"type": "Microsoft.Web/sites/providers/Microsoft.Authorization/roleassignments",
"apiVersion": "2015-07-01",
"name": "<guid>",
"dependsOn": [
"[resourceId('Microsoft.Web/sites/', 'testapp')]"
],
"properties": {
"roleDefinitionId": "[resourceId('Microsoft.Authorization/roleDefinitions', 'b24988ac-6180-42a0-ab88-20f7382dd24c')]",
"principalId": "<guid>",
"scope": "[resourceId('Microsoft.Web/sites/', 'testapp'))]"
}
}
]
}
]
}
That fails, saying that the scope ID must match the URI of the resource.
I also tried some options with a non-nested resource, but none of them would go through either. Is this functionality not supported, or is there some syntax I'm missing that works?
Found answer here: https://www.henrybeen.nl/creating-an-authorization-rule-using-an-arm-template/
Scope tag is not for individual roleAssignements.
The 'name' tag seems to do the scoping.
"parameters": {
"roleAssignmentsGuidFunctionsReader": {
"type": "string",
"defaultValue": "[newGuid()]"
},
"roleAssignmentsGuidFunctionsContributor": {
"type": "string",
"defaultValue": "[newGuid()]"
}
},
"variables": {
"uniqueId": "[substring(uniqueString(resourceGroup().id),9,4)]",
"functionsName": "[concat('MyfuncApp','Functions',variables('uniqueId') )]",
"readerRole": "[concat('/subscriptions/', subscription().subscriptionId, '/providers/Microsoft.Authorization/roleDefinitions/', 'acdd72a7-3385-48ef-bd42-f606fba81ae7')]",
"contributorRole": "[concat('/subscriptions/', subscription().subscriptionId, '/providers/Microsoft.Authorization/roleDefinitions/', 'b24988ac-6180-42a0-ab88-20f7382dd24c')]"
},
"resources": [
{
"apiVersion": "2017-09-01",
"type": "Microsoft.Web/sites/providers/roleAssignments",
"name": "[concat(variables('functionsName'), '/Microsoft.Authorization/', parameters('roleAssignmentsGuidFunctionsContributor'))]",
"properties": {
"roleDefinitionId": "[variables('contributorRole')]",
"principalId": "[ reference( resourceId('Microsoft.Web/sites', variables('functionsName') ), '2018-11-01', 'Full').identity.principalId]" //,
},
"dependsOn": [
"[resourceId('Microsoft.Web/sites', variables('functionsName'))]"
]
}
]