How can I de-duplicate two similar JSON Schemas? - jsonschema

I have the following service object in schema definitions:
"services": {
"type": "object"
"propertyNames": {
"pattern": "^[A-Za-z_]*$",
"errorMessage": "service names must be non-numeric"
}
"patternProperties": {
"^[A-Za-z_]*$": {
"type": "object"
"properties": {
"source": {
"type": ["string", "object"]
},
"port": {
"type": "number",
"errorMessage": "invalid port value"
},
"cpu": {
"type": "number",
"errorMessage": "invalid cpu value
},
"memory": {
"type": "number",
"errorMessage": "invalid memory value
},
"overrides": { "$ref": "#/definitions/overrides" },
"allOf": [...]
}
}
}
In a separate definition, I am maintaining the following override object:
"overrides": {
"type": "object",
"propertyNames": {
"pattern": "^(dev|int|trn|qa|stag|prod)?$",
"errorMessage": "override stage name must be approved environment (dev, int, trn, qa, stag, prod)"
},
"patternProperties": {
"^(dev|int|trn|qa|stag|prod)?$": {
"type": "object",
"propertyNames": {
"pattern": "^[A-Za-z_]*$",
"errorMessage": "override property name must be non-numeric"
},
"properties": {
"$ref": "..."
},
"allOf": [...]
}
}
}
For some context, this is for a configuration file where a service object is defined with port, source, cpu, mem, and other options. Each service can have an override object which can be used to replace service object properties per environment... e.g.
{
"services": {
"main": {
"source": "https://github.com/foo,
"port": 3000,
"cpu": 256,
"memory": 1,
"overrides": {
"prod": {
"port": 8443
}
}
}
}
}
My goal is to optimize this so that I do not have to maintain two of the same schemas. Essentially right now every property that exists in service definition is then duplicated as a property of the override objects. I am also maintaining separate "allOf" validations as the service object properties are required, but override object properties are all optional.
Is it possible to leverage the $ref feature in order to point to the properties of service definition from the override definition? Furthermore, would it be possible to get rid of the override schema entirely and reuse the service schema instead, or does the differences in their schema patternProperties prevent that?
I appreciate any guidance that might get me over the hump.
Full schema minus the allOf blocks for brevity:
{
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"services": {
"type": "object",
"propertyNames": {
"pattern": "^[A-Za-z_]*$",
"errorMessage": "service names must be non-numeric"
},
"patternProperties": {
"^[A-Za-z_]*$": {
"type": "object",
"propertyNames": {
"pattern": "^[A-Za-z_]*$",
"errorMessage": "service property names must be non-numeric"
},
"properties": {
"source": {
"type": ["string", "object"]
},
"port": {
"type": "number",
"errorMessage": "invalid port value"
},
"cpu": {
"type": "number",
"enum": [256, 512, 1024, 2048, 4096],
"errorMessage": "invalid cpu value - should be one of the following: 256, 512, 1024, 2048, 4096"
},
"memory": {
"type": "number",
"enum": [0.5, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11,
12, 13, 14, 15, 16, 17, 18, 19, 20, 21,
22, 23, 24, 25, 26, 27, 28, 29, 30],
"errorMessage": "memory value should be 0.5 - 30"
},
"environment": {
"type": "object",
"errorMessage": "invalid environment variables defined - must be an object"
},
"overrides": { "$ref": "#/definitions/overrides" }
},
"required": ["source", "port", "cpu", "memory", "min_instances"],
"errorMessage": {
"required": {
"source": "missing service source",
"port": "missing service port",
"cpu": "missing service cpu",
"memory": "missing service memory",
"min_instances": "missing minimum required service instances"
},
"type": "service must have at least one defined property",
"additionalProperties": "invalid property found in service definition"
},
"allOf": [...],
"additionalProperties": false
}
},
"errorMessage": { "type": "must have at least one defined service" }
},
"overrides": {
"type": "object",
"propertyNames": {
"pattern": "^(dev|int|trn|qa|stag|prod)?$",
"errorMessage": "override stage name must be approved environment (dev, int, trn, qa, stag, prod)"
},
"patternProperties": {
"^(dev|int|trn|qa|stag|prod)?$": {
"type": "object",
"propertyNames": {
"pattern": "^[A-Za-z_]*$",
"errorMessage": "override property name must be non-numeric"
},
"properties": {
"port": {
"type": "number",
"errorMessage": "invalid port value"
},
"cpu": {
"type": "number",
"enum": [256, 512, 1024, 2048, 4096],
"errorMessage": "invalid cpu value - should be one of the following: 256, 512, 1024, 2048, 4096"
},
"memory": {
"type": "number",
"enum": [0.5, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11,
12, 13, 14, 15, 16, 17, 18, 19, 20, 21,
22, 23, 24, 25, 26, 27, 28, 29, 30],
"errorMessage": "memory value should be 0.5 - 30"
},
"environment": {
"type": "object",
"errorMessage": "invalid environment variables defined - must be an object"
}
},
"errorMessage": {
"type": "environment must have at least one defined override",
"additionalProperties": "invalid property found in override definition"
},
"allOf": [...],
"additionalProperties": false
}
},
"errorMessage": { "type": "overrides must have at least one defined environment" }
}
},
"type": "object",
"properties": {
"name": {
"type": "string",
"errorMessage": "app name should be string"
},
"account": {
"type": "string",
"errorMessage": "invalid account value"
},
"id": {
"type": "string",
"errorMessage": "invalid id tag value"
},
"services": {
"$ref": "#/definitions/services"
},
"stages": {
"type": "array",
"errorMessage": "invalid stages option - must be an array"
}
},
"required": ["name", "account", "id", "services"],
"errorMessage": {
"required": {
"name": "missing app name",
"account": "missing designated account",
"id": "missing designated id tag",
"services": "no services are defined"
}
}
}

Because you've added fields for some properties which are different, you can't de-duplicate all of it. errorMessage is not part of the JSON Schema specification, so it's use is limited to the library you're using.
Some of the properties you can de-duplicate, and you've already used references ($ref).
You could move the memory component to it's own definition...
...
"definitions": {
"componentMemory": {
"type": "number",
"enum": [0.5, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11,
12, 13, 14, 15, 16, 17, 18, 19, 20, 21,
22, 23, 24, 25, 26, 27, 28, 29, 30],
"errorMessage": "memory value should be 0.5 - 30"
},
...
and then reference it in both subschema locations...
...
"cpu": {
"type": "number",
"enum": [256, 512, 1024, 2048, 4096],
"errorMessage": "invalid cpu value - should be one of the following: 256, 512, 1024, 2048, 4096"
},
"memory": {
"$ref": "#/definitions/services"
},
...
If you cared less about the error messages, it could be much further de-duplicated, but I'm guessing you do care about them.
As such, anywhere that the subschemas are identical, it can be de-duplicated.
The values of a properties object ARE subschemas.

Related

Best Match for Validation error with oneof or anyof

I am trying to get proper validation error from oneof or anyof pattern. I have json schema with two or more oneof/anyof condition as mentioned below:
json_schema = {
"type": "object",
"properties": {
"comment": {
"description": "Server Pool Policy Qualification Comments",
"type": "string",
"default": ""
},
"name": {
"description": "Server Pool Policy Qualification Name",
"type": "string",
"default": "",
"pattern": "^[\\-\\.:_a-zA-Z0-9]{1,16}$"
},
"qualifications": {
"description": "Qualifications of Server Pool Policy Qualification",
"type": "array",
"items": {
"description": "Qualification of Server Pool Policy Qualification",
"type": "object",
"oneOf": [
{
"properties": {
"type": {
"description": "Qualification Type",
"type": "string",
"enum": [
"adapter"
]
},
"adapter_qualification":{
"description": "Adapter Qualifications - Adapter Type",
"type": "array",
"properties": {
"adapter_type": {
"description": "Adapter Qualifications - Adapter Type",
"type": "string",
"enum": [
"virtualized-scsi-if"
]
},
"adapter_pid": {
"description": "Adapter Qualifications - Adapter PID (RegEx)",
"type": "string",
"default": "",
"pattern": "[ !#$%\\(\\)\\*\\+,\\-\\./:;\\?#\\[\\\\\\]\\^_\\{\\|\\}~a-zA-Z0-9]{0,256}"
},
"adapter_maximum_capacity": {
"description": "Adapter Qualifications - Maximum Capacity",
"type": "string",
"default": "unspecified",
"pattern": "^unspecified$|^[0-9]$|^[0-9][0-9]$|^[0-9][0-9][0-9]$|^[0-9][0-9][0-9][0-9]$|^[0-5][0-9][0-9][0-9][0-9]$|^6[0-4][0-9][0-9][0-9]$|^65[0-4][0-9][0-9]$|^655[0-2][0-9]$|^6553[0-5]$"
}
},
"additionalProperties": False,
"required": [
"type",
"adapter_type"
]
}
}
},
{
"properties": {
"type": {
"description": "Qualification Type",
"type": "string",
"enum": [
"server_pid"
]
},
"server_pid": {
"description": "Server PID Qualifications - Server PID",
"type": "string",
"default": "",
"pattern": "^123$"
}
},
"additionalProperties": False,
"required": [
"type",
"server_pid"
]
}
]
}
}
},
"additionalProperties": False,
"required": [
"name"
]
}
I have data which has additional element first_rack_id but best matches 2nd element from oneof.
data = {
"descr": "description",
"name": "domainGroup",
"qualifications": [
{
"server_pid": "B200M5",
"type": "server_pid",
"first_rack_id": "10"
}
]
}
validator = Draft7Validator(json_schema)
best = best_match(validator.iter_errors(data))
My expectation is that the error message thrown by validation will find 2nd element from oneof and throw error saying additional property is not allowed. but i get match for 1st element as mentioned below:
'server_pid' is not one of ['adapter']
Failed validating 'enum' in schema[0]['properties']['type']:
{'description': 'Qualification Type',
'enum': ['adapter'],
'type': 'string'}
On instance['type']:
'server_pid'
how do i specify validator to best match with property "type" which will match with enum "server_pid" instead of enum "adapter"
You can specify which schema to validate against with the if/then keywords. It's a bit verbose and can be error prone, but it's the best way to express this sort of thing. Although popular, oneOf is almost never the right choice.
"allOf": [
{
"if": {
"type": "object",
"properties": {
"type": { "const": "adapter" }
},
"required": ["type"]
},
"then": { "$ref": "#/definitions/adapter" }
},
{
"if": {
"type": "object",
"properties": {
"type": { "const": "server_pid" }
},
"required": ["type"]
},
"then": { "$ref": "#/definitions/server-pid" }
}
],
Since best_match need a sort key to help match errors and the default key is to use most depth errors key, see:
best_match
relevance
So maybe you can use a less depth key to match the errors.(below function is just a draft test, you can use it as reference)
def match_less_path(error):
return len(error.path)
best = best_match(validator.iter_errors(data), match_less_path)
And I test the output like this:
Additional properties are not allowed ('first_rack_id' was unexpected)
Failed validating 'additionalProperties' in schema[1]:
{'additionalProperties': False,
'properties': {'server_pid': {'default': '',
'description': 'Server PID '
'Qualifications - Server '
'PID',
'pattern': '^123$',
'type': 'string'},
'type': {'description': 'Qualification Type',
'enum': ['server_pid'],
'type': 'string'}},
'required': ['type', 'server_pid']}
On instance:
{'first_rack_id': '10', 'server_pid': 'B200M5', 'type': 'server_pid'}

Concatenate/ build Json objects From PostgresSQl database

Im trying to build Json array from data existing in a database.
I shall build a Json file that shall match the following Json-file with PostgresSQL.
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Student iformation",
"type": "object",
"required": [
"student",
"name",
"login",
"program",
"branch",
"finished",
],
"properties": {
"student": {
"type": "string",
"minLength": 10,
"maxLength": 10,
"title": "A national identification number, 10 digits"
},
"name": {
"type": "string",
"title": "The name of the student"
},
"login": {
"type": "string",
"title": "The univerity issued computer login"
},
"program": {
"type": "string",
},
"branch": {
"anyOf":[{"type": "string"},{"type": "null"}],
},
"finished": {
"type": "array",
"title": "A list of read courses",
"items": {
"type": "object",
"required": [
"course",
"code",
"credits",
"grade"
],
"properties": {
"course": {
"type": "string",
"title": "Course name"
},
"code": {
"type": "string",
"minLength": 6,
"maxLength": 6,
"title": "Course code"
},
"credits": {
"type": "number",
"title": "Academic credits"
},
"grade": {
"enum" : ["U", "3", "4", "5"]
}
}
}
}
I have tried to do the following to get a better understanding of how to concatenate, build and arrange data that exists in the database:
SELECT array_to_json(array_agg(row_to_json(t))) FROM (
SELECT idnr, name, login, program from students) t;
and
select json_build_object('properties',
json_build_object('student',
json_build_object('idnr',idnr),'name',
json_build_object('name',name),'login',
json_build_object('login',login),'program',
json_build_object('program',program),'branch',
json_build_object('branch',branch)))
from Basicinformation;
How do I build, concatenate objects with PostgresSQL?

Azure ARM template "VmAgentNotRunning"

I am trying to build a template for SQL IaaS deployments including the VM, the SQL IaaS extension, and the SQL IaaS machine. I am getting an error when the SQL IaaS machine deploys because the VM has not fully finished deploying I think.
My current template looks like this:
{
"name": "[parameters('virtualMachineName')]",
"type": "Microsoft.Compute/virtualMachines",
"apiVersion": "2018-10-01",
"location": "[variables('location')]",
"dependsOn": [
"[resourceId('Microsoft.Network/networkInterfaces/', variables('nicName'))]"
],
"properties": {
"hardwareProfile": {
"vmSize": "[parameters('virtualMachineSize')]"
},
"osProfile": {
"computerName": "[parameters('virtualMachineName')]",
"adminUsername": "",
"adminPassword": "",
"windowsConfiguration": {
"provisionVMAgent": true
}
},
"storageProfile": {
"imageReference": {
"publisher": "MicrosoftSQLServer",
"offer": "[variables('sqlOffer')]",
"sku": "[parameters('sqlEdition')]",
"version": "latest"
},
"osDisk": {
"createOption": "FromImage",
"osType": "Windows",
"caching": "ReadWrite",
"name": "[variables('OSDiskName')]",
"managedDisk": {
"storageAccountType": "Premium_LRS"
}
},
"dataDisks": [
{
"name": "[concat(parameters('virtualMachineName'),'-DATA1')]",
"createOption": "Empty",
"managedDisk": {
"storageAccountType": "StandardSSD_LRS"
},
"lun": 0,
"diskSizeGB": "[parameters('sizeOfUserDBDiskInGB')]",
"caching": "ReadOnly"
},
{
"name": "[concat(parameters('virtualMachineName'),'-DATA2')]",
"createOption": "Empty",
"managedDisk": {
"storageAccountType": "Standard_LRS"
},
"lun": 1,
"diskSizeGB": 128,
"caching": "ReadOnly"
},
{
"name": "[concat(parameters('virtualMachineName'),'-DATA3')]",
"createOption": "Empty",
"managedDisk": {
"storageAccountType": "Standard_LRS"
},
"lun": 2,
"diskSizeGB": 1023,
"caching": "ReadOnly"
},
{
"name": "[concat(parameters('virtualMachineName'),'-DATA4')]",
"createOption": "Empty",
"managedDisk": {
"storageAccountType": "Standard_LRS"
},
"lun": 4,
"diskSizeGB": 512,
"caching": "ReadOnly"
},
{
"name": "[concat(parameters('virtualMachineName'),'-DATA5')]",
"createOption": "Empty",
"managedDisk": {
"storageAccountType": "Standard_LRS"
},
"lun": 5,
"diskSizeGB": 512,
"caching": "None"
},
{
"name": "[concat(parameters('virtualMachineName'),'-DATA6')]",
"createOption": "Empty",
"managedDisk": {
"storageAccountType": "Standard_LRS"
},
"lun": 6,
"diskSizeGB": 512,
"caching": "None"
}
]
},
"networkProfile": {
"networkInterfaces": [
{
"id": "[resourceId('Microsoft.Network/networkInterfaces', variables('nicName'))]"
}
]
},
"diagnosticsProfile": {
"bootDiagnostics": {
"enabled": true,
"storageUri": "[reference(resourceId(resourceGroup().name, 'Microsoft.Storage/storageAccounts', parameters('diagnosticsStorageAccountName')), '2015-06-15').primaryEndpoints['blob']]"
}
}
},
"tags": {
"Application Stack": "[parameters('Application Stack')]",
"Business Contact": "[parameters('Business Contact')]",
"Associated to": "[resourceGroup().name]",
"Business Owner": "[parameters('Business Owner')]",
"BusinessUnit": "[parameters('Business Unit')]",
"Department": "[parameters('Department')]",
"Region": "[parameters('Region')]",
"Type": "[variables('VMType')]"
}
},
{
"apiVersion": "2015-06-15",
"type": "Microsoft.Compute/virtualMachines/extensions",
"name": "[concat(parameters('virtualMachineName'), '/SqlIaasExtension')]",
"location": "[variables('location')]",
"dependsOn": [
"[concat('Microsoft.Compute/virtualMachines/', parameters('virtualMachineName'))]"
],
"properties": {
"type": "SqlIaaSAgent",
"publisher": "Microsoft.SqlServer.Management",
"typeHandlerVersion": "1.2",
"autoUpgradeMinorVersion": "true",
"settings": {
"AutoTelemetrySettings": {
"Region": "[variables('location')]"
},
"AutoPatchingSettings": {
"PatchCategory": "WindowsMandatoryUpdates",
"Enable": true,
"DayOfWeek": "Saturday",
"MaintenanceWindowStartingHour": "2",
"MaintenanceWindowDuration": "180"
},
"ServerConfigurationsManagementSettings": {
"SQLConnectivityUpdateSettings": {
"ConnectivityType": "Private",
"Port": "1433"
},
"SQLWorkloadTypeUpdateSettings": {
"SQLWorkloadType": "General"
},
"SQLStorageUpdateSettings": {
"DiskCount": 1,
"NumberOfColumns": 1,
"StartingDeviceID": "2",
"DiskConfigurationType": "NEW"
},
"AdditionalFeaturesServerConfigurations": {
"IsRServicesEnabled": "false"
}
}
},
"protectedSettings": {
"SQLAuthUpdateUserName": "",
"SQLAuthUpdatePassword": ""
}
}
},
{
"type": "Microsoft.SqlVirtualMachine/sqlVirtualMachines",
"apiVersion": "2017-03-01-preview",
"name": "[parameters('virtualMachineName')]",
"location": "[variables('location')]",
"dependsOn": [
"[resourceId('Microsoft.Compute/virtualMachines', parameters('virtualMachineName'))]"
],
"properties": {
"virtualMachineResourceId": "[resourceId('Microsoft.Compute/virtualMachines', parameters('virtualMachineName'))]",
"sqlImageOffer": "[variables('sqlOffer')]",
"sqlServerLicenseType": "AHUB",
"sqlManagement": "Full",
"sqlImageSku": "[parameters('sqlEdition')]"
}
}
This last portion deploys the Azure SQL IaaS machine, which is dependent on the VM Agent running. I am getting an error saying,
"error": {
"code": "VmAgentNotRunning",
"message": "The VM agent in Virtual Machine: '/subscriptions/{GUID}/resourceGroups/{ResourceGroup}/providers/Microsoft.Compute/virtualMachines/{VmName}' is not in running state. Please make sure it is installed and in running state and try again later."
}
Is there a way I can add a waitfor or add a dependency on the agent running before deploying the SQL IaaS machine? Without deploying this, the template works great. After the machine is deployed, i can run a powershell script to create the object, but i'm trying to avoid that. Thanks in advance.
Randomly discovered this article, which explains my issue.
https://learn.microsoft.com/en-us/azure/virtual-machines/windows/sql/virtual-machines-windows-sql-register-with-resource-provider?tabs=azure-cli%2Cpowershell#management-modes
The trick was to change "sqlManagement": "Full", to "sqlManagement": "LightWeight",

Podio Api: Why does PUT request /app/{app_id}/field/{field_id} delete category field options for contact apps?

I'm tring to modify category fields of a contact app using Podio API.
I get the following response for the GET request (https://api.podio.com/app/22768616/field/189304190):
(Previously I created the field with a POST request and everything works fine)
{
"status": "active",
"type": "category",
"field_id": 189304190,
"label": "myCategories",
"config": {
"default_value": null,
"unique": false,
"description": null,
"hidden_create_view_edit": false,
"required": false,
"mapping": null,
"label": "myCategories",
"visible": true,
"delta": 9,
"hidden": false,
"settings": {
"multiple": false,
"options": [
{
"status": "active",
"text": "Cat1",
"id": 1,
"color": "DCEBD8"
},
{
"status": "active",
"text": "Cat2",
"id": 2,
"color": "DCEBD8"
},
{
"status": "active",
"text": "Cat3",
"id": 3,
"color": "DCEBD8"
},
{
"status": "active",
"text": "Cat4",
"id": 4,
"color": "DCEBD8"
}
],
"display": "dropdown"
}
},
"external_id": "mycategories-2"
}
If I send a PUT request to https://api.podio.com/app/22768616/field/189304190 with the same response, the dropdown category field changes to an inline category field and all the options are deleted. (I expected nothing would happen to my field, I also tried to modify the text of the options, but got the same result).
{
"status": "active",
"type": "category",
"field_id": 189304190,
"label": "myCategories",
"config": {
"default_value": null,
"unique": false,
"description": null,
"hidden_create_view_edit": false,
"required": false,
"mapping": null,
"label": "myCategories",
"visible": true,
"delta": 0,
"hidden": false,
"settings": {
"multiple": false,
"options": [
{
"status": "deleted",
"text": "Cat1",
"id": 1,
"color": "DCEBD8"
},
{
"status": "deleted",
"text": "Cat2",
"id": 2,
"color": "DCEBD8"
},
{
"status": "deleted",
"text": "Cat3",
"id": 3,
"color": "DCEBD8"
},
{
"status": "deleted",
"text": "Cat4",
"id": 4,
"color": "DCEBD8"
}
],
"display": "inline"
}
},
"external_id": "mycategories-2"
}
Could you please help with any example to update a category fields correctly?
Can you add what the body is when you are using the PUT endpoint?
My guess is that you are somehow not mapping the "settings" parameter correctly. Per the API documentation the settings parameter for a category field should follow this format:
{
"options": The list of options for the question
[
{
"id": The id of the option, only use this for existing options,
"status": The current status of the option, "active" or "deleted", only use this to delete options,
"text": The text for the option (required)
},
... (more options)
],
"multiple": True if multiple answers should be allowed, False otherwise
}

How to make the root element mandatory in JSONSchema

I have the below JSONSchema, I want the root tag envelope to be mandatory.
Any help would be appreciated.
{
"id": "envelope",
"$schema": "http://json-schema.org/schema#",
"tittle": "Root schema",
"description": "Root schema for all services",
"apiVersion": "1.0",
"type": "object",
"required": [
"metadata",
"data"
],
"properties": {
"metadata": {
"description": "The meta data of the data field",
"type": "object",
"required": [
"sourceSystem",
"deliveryCount",
"retryPeriod",
"correlationId"
],
"properties": {
"sourceSystem": {
"description": "The source System ",
"type": "string"
},
"deliveryCount": {
"description": "Number of times the request tried",
"type": "number",
"default": 0,
"maxLength": 5
},
"retryPeriod": {
"description": "Time set to retry",
"type": "number"
},
"correlationId": {
"description": "Unique id for reference",
"type": "string"
}
}
},
"data": {
"description": "The actual content",
"type": "object"
},
"response": {
"description": "Response",
"type": "string"
}
}
}
The output is
{
"metadata": {
"sourceSystem": "",
"deliveryCount": 1,
"retryPeriod": 30,
"correlationId": ""
},
"data": {}
}
expected output is
{ "envelope" : {
"metadata": {
"sourceSystem": "",
"deliveryCount": 1,
"retryPeriod": 30,
"correlationId": ""
},
"data": {} } }
The "id" attribute does not define any root element, it is used for different purposes.
All you need to do is to define your root schema as an object with a single "envelope" property:
{
"type" : "object"
"properties" : {
"envelope" : {
// here comes your (current) schema
}
}
}