How to handle dynamic schema in bigquery - google-bigquery

Question

My data looks like this:
row 1 - {"id": "1", "object": "user","metadata": {"name": "1234"}}
rows 2 - {"id": "1", "object": "user","metadata": {"name": "1234","email": "abc#abc.com"}}
I created the table using row 1
metadata RECORD NULLABLE
metadata.tenant STRING NULLABLE
object STRING NULLABLE
id STRING NULLABLE
But my insert will fail on row 2. what should my schema look like so that it can handle changes in metadata field?

Answer 1

For the example shown in your question - I would go with below schema
[
{
"name": "id",
"type": "INTEGER",
"mode": "NULLABLE"
},
{
"name": "object",
"type": "STRING",
"mode": "NULLABLE"
},
{
"name": "metadata",
"type": "STRING",
"mode": "NULLABLE"
}
]
And below is an example of how I would process it
#standardSQL
WITH `yourProject.yourDataset.yourTable` AS (
SELECT 1 AS id, 'user' AS object, '{"name": "BI Architect", "email": "abc#abc.com"}' AS metadata UNION ALL
SELECT 2, 'expert', '{"name": "Elliott Brossard"}'
)
SELECT
id,
object,
JSON_EXTRACT_SCALAR(metadata, '$.name') AS name,
JSON_EXTRACT_SCALAR(metadata, '$.email') AS email
FROM `yourProject.yourDataset.yourTable`
ORDER BY id
resulted in below output
id object name email
1 user BI Architect abc#abc.com
2 expert Elliott Brossard null