Related
Im trying to build Json array from data existing in a database.
I shall build a Json file that shall match the following Json-file with PostgresSQL.
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Student iformation",
"type": "object",
"required": [
"student",
"name",
"login",
"program",
"branch",
"finished",
],
"properties": {
"student": {
"type": "string",
"minLength": 10,
"maxLength": 10,
"title": "A national identification number, 10 digits"
},
"name": {
"type": "string",
"title": "The name of the student"
},
"login": {
"type": "string",
"title": "The univerity issued computer login"
},
"program": {
"type": "string",
},
"branch": {
"anyOf":[{"type": "string"},{"type": "null"}],
},
"finished": {
"type": "array",
"title": "A list of read courses",
"items": {
"type": "object",
"required": [
"course",
"code",
"credits",
"grade"
],
"properties": {
"course": {
"type": "string",
"title": "Course name"
},
"code": {
"type": "string",
"minLength": 6,
"maxLength": 6,
"title": "Course code"
},
"credits": {
"type": "number",
"title": "Academic credits"
},
"grade": {
"enum" : ["U", "3", "4", "5"]
}
}
}
}
I have tried to do the following to get a better understanding of how to concatenate, build and arrange data that exists in the database:
SELECT array_to_json(array_agg(row_to_json(t))) FROM (
SELECT idnr, name, login, program from students) t;
and
select json_build_object('properties',
json_build_object('student',
json_build_object('idnr',idnr),'name',
json_build_object('name',name),'login',
json_build_object('login',login),'program',
json_build_object('program',program),'branch',
json_build_object('branch',branch)))
from Basicinformation;
How do I build, concatenate objects with PostgresSQL?
It is possible to export a bigquery table schema to a JSON file but the resulting JSON file is a bigquery table schema and not a JSON schema.
I am looking for a way to generate a JSON schema using a bigquery table based on the standard available here: https://json-schema.org/
This looks something like this:
{
"definitions": {},
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "http://example.com/root.json",
"type": "object",
"title": "The Root Schema",
"required": [
"glossary"
],
"properties": {
"glossary": {
"$id": "#/properties/glossary",
"type": "object",
"title": "The Glossary Schema",
"required": [
"title",
"GlossDiv"
],
"properties": {
"title": {
"$id": "#/properties/glossary/properties/title",
"type": "string",
"title": "The Title Schema",
"default": "",
"examples": [
"example glossary"
],
"pattern": "^(.*)$"
},
"GlossDiv": {
"$id": "#/properties/glossary/properties/GlossDiv",
"type": "object",
"title": "The Glossdiv Schema",
"required": [
"title",
"GlossList"
],
"properties": {
"title": {
"$id": "#/properties/glossary/properties/GlossDiv/properties/title",
"type": "string",
"title": "The Title Schema",
"default": "",
"examples": [
"S"
],
"pattern": "^(.*)$"
},
"GlossList": {
"$id": "#/properties/glossary/properties/GlossDiv/properties/GlossList",
"type": "object",
"title": "The Glosslist Schema",
"required": [
"GlossEntry"
],
"properties": {
"GlossEntry": {
"$id": "#/properties/glossary/properties/GlossDiv/properties/GlossList/properties/GlossEntry",
"type": "object",
"title": "The Glossentry Schema",
"required": [
"ID",
"SortAs",
"GlossTerm",
"Acronym",
"Abbrev",
"GlossDef",
"GlossSee"
],
"properties": {
"ID": {
"$id": "#/properties/glossary/properties/GlossDiv/properties/GlossList/properties/GlossEntry/properties/ID",
"type": "string",
"title": "The Id Schema",
"default": "",
"examples": [
"SGML"
],
"pattern": "^(.*)$"
},
"SortAs": {
"$id": "#/properties/glossary/properties/GlossDiv/properties/GlossList/properties/GlossEntry/properties/SortAs",
"type": "string",
"title": "The Sortas Schema",
"default": "",
"examples": [
"SGML"
],
"pattern": "^(.*)$"
},
"GlossTerm": {
"$id": "#/properties/glossary/properties/GlossDiv/properties/GlossList/properties/GlossEntry/properties/GlossTerm",
"type": "string",
"title": "The Glossterm Schema",
"default": "",
"examples": [
"Standard Generalized Markup Language"
],
"pattern": "^(.*)$"
},
"Acronym": {
"$id": "#/properties/glossary/properties/GlossDiv/properties/GlossList/properties/GlossEntry/properties/Acronym",
"type": "string",
"title": "The Acronym Schema",
"default": "",
"examples": [
"SGML"
],
"pattern": "^(.*)$"
},
"Abbrev": {
"$id": "#/properties/glossary/properties/GlossDiv/properties/GlossList/properties/GlossEntry/properties/Abbrev",
"type": "string",
"title": "The Abbrev Schema",
"default": "",
"examples": [
"ISO 8879:1986"
],
"pattern": "^(.*)$"
},
"GlossDef": {
"$id": "#/properties/glossary/properties/GlossDiv/properties/GlossList/properties/GlossEntry/properties/GlossDef",
"type": "object",
"title": "The Glossdef Schema",
"required": [
"para",
"GlossSeeAlso"
],
"properties": {
"para": {
"$id": "#/properties/glossary/properties/GlossDiv/properties/GlossList/properties/GlossEntry/properties/GlossDef/properties/para",
"type": "string",
"title": "The Para Schema",
"default": "",
"examples": [
"A meta-markup language, used to create markup languages such as DocBook."
],
"pattern": "^(.*)$"
},
"GlossSeeAlso": {
"$id": "#/properties/glossary/properties/GlossDiv/properties/GlossList/properties/GlossEntry/properties/GlossDef/properties/GlossSeeAlso",
"type": "array",
"title": "The Glossseealso Schema",
"items": {
"$id": "#/properties/glossary/properties/GlossDiv/properties/GlossList/properties/GlossEntry/properties/GlossDef/properties/GlossSeeAlso/items",
"type": "string",
"title": "The Items Schema",
"default": "",
"examples": [
"GML",
"XML"
],
"pattern": "^(.*)$"
}
}
}
},
"GlossSee": {
"$id": "#/properties/glossary/properties/GlossDiv/properties/GlossList/properties/GlossEntry/properties/GlossSee",
"type": "string",
"title": "The Glosssee Schema",
"default": "",
"examples": [
"markup"
],
"pattern": "^(.*)$"
}
}
}
}
}
}
}
}
}
}
}
BigQuery does not use the json-schema standard for the tables schema. I found two projects that have the code available to go from json-schema to BigQuery schema:
jsonschema-bigquery
jsonschema-transpiler
You could try using those projects as reference to create the opposite transformation. Also, you could create a feature request to the BigQuery team, asking to include the json-schema standard as an output format option.
No this is not possible without writing a program to do so for you.
There is a feature request made by me that requests this functionality.
https://issuetracker.google.com/issues/145308573
Given the following two JSON Schema definitions
Schema A
{
"$schema": "http://json-schema.org/draft-04/hyper-schema",
"title": "Member Resource",
"description": "A Member at a group",
"id": "schemata/member",
"definitions": {
"first_name": {
"description": "the first name",
"example": "Severus",
"minLength": 1,
"maxLength": 255,
"type": "string"
},
"last_name": {
"description": "the last name",
"example": "Snape",
"minLength": 1,
"maxLength": 255,
"type": "string"
},
"member_response": {
"description": "Successful response to a show or search request",
"type": "object",
"additionalProperties": false,
"properties": {
"member": { "$ref": "/schemata/member" }
},
"required": ["member"]
},
"error_response": {
"description": "Error response to a show or search request",
"type": "object",
"additionalProperties": false,
"properties": {
"reference_id": {"$ref": "#/definitions/reference_id"},
"errors": {"$ref": "#/definitions/errors"}
},
"required": [errors"]
}
},
"links": [
{
"description": "Retrieve a member",
"href": "/members/{(%23%2Fschemata%member%2Fdefinitions%2Fidentity)}",
"method": "GET",
"rel": "instance",
"title": "Show",
"targetSchema": {
"description": "Result of a get request. Can be either a success or a failure.",
"type": ["object"],
"oneOf": [
{ "$ref": "#/definitions/member_response" },
{ "$ref": "#/definitions/error_response" }
]
}
},
{
"description": "Search for a member",
"href": "/members/search",
"method": "GET",
"rel": "instance",
"schema": {
"description": "The expected payload for a search request",
"type": "object",
"additionalProperties": false,
"properties": {
"first_name": {
"$ref": "#/definitions/first_name"
},
"last_name": {
"$ref": "#/definitions/last_name"
}
},
"required": ["first_name", "last_name"]
},
"targetSchema": {
"description": "Result of a get request. Can be either a success or a failure.",
"type": ["object"],
"oneOf": [
{ "$ref": "#/definitions/member_response" },
{ "$ref": "#/definitions/error_response" }
]
}
"title": "Search"
}
],
}
Schema B
{
"$schema": "http://json-schema.org/draft-04/hyper-schema",
"title": "Member Resource",
"description": "A Member at a group",
"id": "schemata/member",
"definitions": {
"first_name": {
"description": "the first name",
"example": "Severus",
"minLength": 1,
"maxLength": 255,
"type": "string"
},
"last_name": {
"description": "the last name",
"example": "Snape",
"minLength": 1,
"maxLength": 255,
"type": "string"
},
"search_payload": {
"description": "The expected payload for a search request",
"type": "object",
"additionalProperties": false,
"properties": {
"first_name": {
"$ref": "#/definitions/first_name"
},
"last_name": {
"$ref": "#/definitions/last_name"
}
},
"required": ["first_name", "last_name"]
},
"member_response": {
"description": "Successful response to a show or search request",
"type": "object",
"additionalProperties": false,
"properties": {
"member": { "$ref": "/schemata/member" }
},
"required": ["member"]
},
"error_response": {
"description": "Error response to a show or search request",
"type": "object",
"additionalProperties": false,
"properties": {
"reference_id": {"$ref": "#/definitions/reference_id"},
"errors": {"$ref": "#/definitions/errors"}
},
"required": [errors"]
},
"get_response": {
"description": "Result of a get request. Can be either a success or a failure.",
"type": ["object"],
"oneOf": [
{ "$ref": "#/definitions/member_response" },
{ "$ref": "#/definitions/error_response" }
]
}
},
"links": [
{
"description": "Retrieve a member",
"href": "/members/{(%23%2Fschemata%member%2Fdefinitions%2Fidentity)}",
"method": "GET",
"rel": "instance",
"title": "Show",
"targetSchema": {
"$ref": "#/definitions/get_response"
}
},
{
"description": "Search for a member",
"href": "/members/search",
"method": "GET",
"rel": "instance",
"schema": {
"$ref": "#/definitions/search_payload"
},
"targetSchema": {
"$ref": "#/definitions/get_response"
},
"title": "Search"
}
],
}
Both schemas are functionally the same. The difference is that targetSchema is defined inline in schema A but as a ref in schema B.
I use a library to validate the input and output to and API endpoint. For example when testing my APIs I want to validate that the response to each request returns a JSON object that conforms with targetSchema for that API.
JSON::Validator.fully_validate(
schema,
object_to_test,
:fragment => "/path/to/fragment"
)
In order to validate against the targetSchema for the /members/search API defined above I need to be able to reference its targetSchema.
In schema B I can do
JSON::Validator.fully_validate(
schema,
object_to_test,
:fragment => "#/definitions/get_response"
)
Is it possible to do the above for schema A too? i.e. can I reference the actual targetSchema of the search link directly. Perhaps it might look like the following
JSON::Validator.fully_validate(
schema,
object_to_test,
:fragment => "#/links[1]/targetSchema"
)
or
JSON::Validator.fully_validate(
schema,
object_to_test,
:fragment => "#/links/[SOME_WAY_OF_SPECIFYING_THAT_TITLA_EQL_SEARCH"]/targetSchema"
)
Given your schema, you can reference the search link's targetSchema with the following JSON Pointer(1).
#/links/1/targetSchema
Here 1 is the index of the desired item in the links array. This is the only way to reference an item in an array. To precisely answer the question -- there is no way of specifying the item in the array where title equals "search".
Obviously, referencing the targetSchema using an index is fragile. If you add a link to the schema in the wrong place, your code will break. You would be better off if you looped through the links in code and chose the one you need.
You might ask, "Why is it so difficult to reference a link's targetSchema for validation?" The answer is that targetSchema is not intended for validation. targetSchema is intended to be informational only. It's documentation. The only schema that the response should be responsible for conforming to is the one it declares in the response(2). This is one of the core ideas of REST. The client and server are decoupled. The client doesn't make any assumptions about the response it will get. The response itself should have all the information needed to interpret the response and what you can do next.
https://www.rfc-editor.org/rfc/rfc6901
http://json-schema.org/latest/json-schema-core.html#anchor33
I'm trying to create a BigQuery table using Python. Other operations (queries, retrieving table bodies etc.) are working fine, but when trying to create a table I'm stuck with an error:
apiclient.errors.HttpError: https://www.googleapis.com/bigquery/v2/projects/marechal-consolidation/datasets/marechal_results/tables?alt=json
returned "Output field used as input">
Here's the command I'm executing:
projectId = 'xxxx'
dataSet = 'marechal_results'
with open(filePath+'tableStructure.json') as data_file:
structure = json.load(data_file)
table_result = tables.insert(projectId=projectId, datasetId=dataSet, body=structure).execute()
JSON table:
{
"kind": "bigquery#table",
"tableReference": {
"projectId": "xxxx",
"tableId": "xxxx",
"datasetId": "xxxx"
},
"type": "table",
"schema": {
"fields": [
{
"mode": "REQUIRED",
"type": "STRING",
"description": "Company",
"name": "COMPANY"
},
{
"mode": "REQUIRED",
"type": "STRING",
"description": "Currency",
"name": "CURRENCY"
}
// bunch of other fields follow...
]
}
}
Why am I receiving this error?
EDIT: Here's the JSON object I'm passing as parameter:
{
"kind": "bigquery#table",
"type": "TABLE",
"tableReference": {
"projectId": "xxxx",
"tableId": "xxxx",
"datasetId": "xxxx"
},
"schema": {
"fields": [
{
"type": "STRING",
"name": "COMPANY"
},
{
"type": "STRING",
"name": "YEAR"
},
{
"type": "STRING",
"name": "COUNTRY_ISO"
},
{
"type": "STRING",
"name": "COUNTRY"
},
{
"type": "STRING",
"name": "COUNTRY_GROUP"
},
{
"type": "STRING",
"name": "REGION"
},
{
"type": "STRING",
"name": "AREA"
},
{
"type": "STRING",
"name": "BU"
},
{
"type": "STRING",
"name": "REFERENCE"
},
{
"type": "FLOAT",
"name": "QUANTITY"
},
{
"type": "FLOAT",
"name": "NET_SALES"
},
{
"type": "FLOAT",
"name": "GROSS_SALES"
},
{
"type": "STRING",
"name": "FAM_GRP"
},
{
"type": "STRING",
"name": "FAMILY"
},
{
"type": "STRING",
"name": "PRESENTATION"
},
{
"type": "STRING",
"name": "ORIG_FAMILY"
},
{
"type": "FLOAT",
"name": "REF_PRICE"
},
{
"type": "STRING",
"name": "CODE1"
},
{
"type": "STRING",
"name": "CODE4"
}
]
}
}
This is probably too late to help you but hopefully it helps the next poor soul like me. It took me a while figure out what "Output field used as input" meant.
Though the API specifies the same object for the request (input) and response (output), some fields are only allowed in the response. In the docs you will see their descriptions prefixed with "Output only". From looking at your table definition I see that you have "type": "TABLE" and "type" is listed as an "Output only" property. So I would gander that if you remove it then that error will go away. Here is the link to the docs: https://cloud.google.com/bigquery/docs/reference/rest/v2/tables
It would help if they told you what field the violation was on.
Facing an issue with schema validation.
schema :
{
"type": "object",
"$schema": "http://json-schema.org/draft-03/schema",
"id": "#",
"required": true,
"patternProperties": {
"^[A-Z0-9._%+-]+#[A-Z0-9.-]+\.[A-Z]{2,6}$": {
"type": "object",
"required": true,
"properties": {
"_from": {
"id": "_from",
"type": "string",
"required": true
},
"message": {
"type": "object",
"id": "message",
"properties": {
"detail": {
"type": "string",
"id": "detail",
"required": true
},
"from": {
"type": "string",
"id": "from",
"required": true
}
}
}
}
}
}
}
json :
{
"tom#example.com": {
"_from": "giles#gmail.com",
"message": {
"from": "Giles#gmail.com",
"detail": "AnyonewanttomeetmeinParis"
}
},
"harry#example.com": {
"_from": "giles#gmail.com",
"message": {
"from": "Giles#gmail.com",
"detail": "AnyonewanttomeetmeinParis"
}
}
}
Here the key email address is dynamic, somehow it doesn't validate regex for email validation.
Can you please advise me to correct the schema.
I am validating using : http://json-schema-validator.herokuapp.com/index.jsp
I see in your pattern that you seem to have forgotten to escape some characters or didn't do it correctly:
"^[A-Z0-9._%+-]+#[A-Z0-9.-]+\.[A-Z]{2,6}$"
and it causes the error that you can see when you hover the mouse over the link at the top of the validator:
it should be:
"^[A-Z0-9\\._%\\+-]+#[A-Z0-9\\.-]+\\.[A-Z]{2,6}$"
or without escaping the inner/class characters but I'd use the first pattern because I think its intention is clearer:
"^[A-Z0-9._%+-]+#[A-Z0-9.-]+\\.[A-Z]{2,6}$"
You need to have two \ because the first \ is an escape for the second \. With a single one it wouldn't work because there is no escape sequence like \. or \+ in javascript. You want to have a \in the pattern itself.
However json schema patternProperties are case sensitive by default so you need to extend your email pattern by adding a-z to it:
"^[A-Za-z0-9\\._%\\+-]+#[A-Za-z0-9\\.-]+\\.[A-Za-z]{2,6}$"
(I didn't find any other way to make it case insensitive)
You also need to exclude any other property names by adding "additionalProperties": false next to the patternProperties or otherwise it catches everything else that does not match the pattern.
The working schema should then look like this:
{
"type": "object",
"$schema": "http://json-schema.org/draft-03/schema",
"id": "#",
"required": true,
"patternProperties": {
"^[A-Za-z0-9\\._%\\+-]+#[A-Za-z0-9\\.-]+\\.[A-Za-z]{2,6}$": {
"type": "object",
"required": true,
"properties": {
"_from": {
"id": "_from",
"type": "string",
"required": true
},
"message": {
"type": "object",
"id": "message",
"properties": {
"detail": {
"type": "string",
"id": "detail",
"required": true
},
"from": {
"type": "string",
"id": "from",
"required": true
}
}
}
}
}
},
"additionalProperties": false
}
I've tested it on: http://jsonschemalint.com/
Changed the schema as per draft 04 :
{
"type": "object",
"$schema": "http://json-schema.org/draft-04/schema",
"patternProperties": {
"^[A-Za-z0-9\\._%\\+-]+#[A-Za-z0-9\\.-]+\\.[A-Za-z]{2,6}$": {
"type": "object",
"properties": {
"__from": {
"type": "string"
},
"message": {
"type": "object",
"properties": {
"from": {
"type": "string"
},
"detail": {
"type": "string"
}
},
"required": [ "from","detail"]
}
},
"required": [ "__from","message"]
}
},
"additionalProperties": false
}