How to add timestamp column when loading file to table - google-bigquery

Question

I'm loading batch files to a table.
I want to add a timestamp column to the table so I can know the insertion times
on the record. I'm loading in append mode, so not all records insert at the same time.
Unfortunately, I didn't find a way to it in big query. When loading a file to a table, I didn't find an option to add padding the insertion with additional columns. I just want to calculate timestamp in my code and put it as constant field for all the insertion process.
The solution that I'm doing now, is to load to temp table and then query the table + new timestamp field into the target table. It works, but it's another step and I have multiple loadings and the full process takes too much time due to the latency of another step.
Does anyone know about another solution with only 1 step?

Answer 1

That's a great feature request for https://code.google.com/p/google-bigquery/issues/list. Unfortunately, there is no automated way to do it today. I like the way you are doing it though :)

Answer 2

If you are willing to make a new table to house this information, I recommend making a new table with the following settings:
table with _PARTITIONTIME field based on insertion
If you make a table using the default _PARTITIONTIME partitioning field, it does exactly what you are asking based on time of insertion

Answer 3

You can add a timestamp column/value using Pandas data-frame:
from datetime import datetime
import pandas as pd
from google.cloud import bigquery
insertDate = datetime.utcnow()
bigqueryClient = bigquery.Client()
tableRef = bigqueryClient.dataset("dataset-name").table("table-name")
dataFrame = pd.read_json("file.json")
dataFrame['insert_date'] = insertDate
bigqueryJob = bigqueryClient.load_table_from_dataframe(dataFrame, tableRef)
bigqueryJob.result()

Answer 4

You can leverage the "hive partitioning" functionality of BigQuery load jobs to accomplish this. This feature is normally used for "external tables" where the data just sits in GCS in carefully-organized folders, but there's no law against using it to import data into a native table.
When you write your batch files, include your timestamp as part of the path. For example, if your timestamp field is called "added_at" then write your batch files to gs://your-bucket/batch_output/added_at=1658877709/file.json
Load your data with the hive partitioning parameters so that the "added_at" value comes from the path instead of from the contents of your file. Example:
bq load --source_format=JSON \
--hive_partitioning_mode=AUTO \
--hive_partitioning_source_uri_prefix=gs://your-bucket/batch_output/ \
dataset-name.table-name \
gs://your-bucket/output/added_at=1658877709/*
The python API has equivalent functionality.