Bigquery add columns to table schema - google-bigquery

Question

I am trying to add new column to BigQuery existing table. I have tried bq command tool and API approach. I get following error when making call to Tables.update().
I have tried with providing full schema with additional field and that also gives me same error as shown below.
With API I get following Error:
{
"schema": {
"fields": [{
"name": "added_column",
"type": "integer",
"mode": "nullable"
}]
}
}
{
"error": {
"errors": [{
"domain": "global",
"reason": "invalid",
"message": "Provided Schema does not match Table [blah]"
}],
"code": 400,
"message": "Provided Schema does not match Table [blah]"
}
}
With BQ tool I get following error:
./bq update -t blah added_column:integer
BigQuery error in update operation: Provided Schema does not match Table [blah]

Answer 1

Try this:
bq --format=prettyjson show yourdataset.yourtable > table.json
Edit table.json and remove everything except the inside of "fields" (e.g. keep the [ { "name": "x" ... }, ... ]). Then add your new field to the schema.
Or pipe through jq
bq --format=prettyjson show yourdataset.yourtable | jq .schema.fields > table.json
Then run:
bq update yourdataset.yourtable table.json
You can add --apilog=apilog.txt to the beginning of the command line which will show exactly what is sent / returned from the bigquery server.

Answer 2

In my case I was trying to add a REQUIRED field to a template table, and was running into this error. Changing the field to NULLABLE , let me update the table.
Also more recent version on updates for anybody stumbling from Google.
#To create table
bq mk --schema domain:string,pageType:string,source:string -t Project:Dataset.table
#Or using schema file
bq mk --schema SchemaFile.json -t Project:Dataset.table
#SchemaFile.json format
[{
"mode": "REQUIRED",
"name": "utcTime",
"type": "TIMESTAMP"
},
{
"mode": "REQUIRED",
"name": "domain",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "testBucket",
"type": "STRING"
},
{
"mode": "REQUIRED",
"name": "isMobile",
"type": "BOOLEAN"
},
{
"mode": "REQUIRED",
"name": "Category",
"type": "RECORD",
"fields": [
{
"mode": "NULLABLE",
"name": "Type",
"type": "STRING"
},
{
"mode": "REQUIRED",
"name": "Published",
"type": "BOOLEAN"
}
]
}]
# TO update
bq update --schema UpdatedSchema.json -t Project:Dataset.table
# Updated Schema contains old and any newly added columns
Some docs for template tables

Answer 3

Example using the BigQuery Node JS API:
const fieldDefinition = {
name: 'nestedColumn',
type: 'RECORD',
mode: 'REPEATED',
fields: [
{name: 'id', type: 'INTEGER', mode: 'NULLABLE'},
{name: 'amount', type: 'INTEGER', mode: 'NULLABLE'},
],
};
const table = bigQuery.dataset('dataset1').table('source_table_name');
const metaDataResult = await table.getMetadata();
const metaData = metaDataResult[0];
const fields = metaData.schema.fields;
fields.push(fieldDefinition);
await table.setMetadata({schema: {fields}});

Answer 4

I was stuck trying to add columns to an existing table in BigQuery using the Python client and found this post several times. I'll then let the piece of code that solved it for me, in case someone's having the same problem:
# update table schema
bigquery_client = bigquery.Client()
dataset_ref = bigquery_client.dataset(dataset_id)
table_ref = dataset_ref.table(table_id)
table = bigquery_client.get_table(table_ref)
new_schema = list(table.schema)
new_schema.append(bigquery.SchemaField('LOLWTFMAN','STRING'))
table.schema = new_schema
table = bigquery_client.update_table(table, ['schema']) # API request

Answer 5

You can add Schema to your table through GCP console Easier and Clear:-

Answer 6

Here's a quick snippet I wrote that will dynamically add schema columns if the data coming in (from a server, etc) doesn't match what exists currently in a BigQuery Table:
def verify_schema(client, table, data_dict):
schema = list(table.schema)
existing_schema_names = [schema.name for schema in schema]
validation_list = [True if schema_field in existing_schema_names else schema.append(
bigquery.SchemaField(name=schema_field, field_type='STRING', mode='NULLABLE')) for schema_field in data_dict.keys()]
if None in validation_list:
table.schema = schema
client.update_table(table, ['schema'])