Create bigquery table using google cloud deployment manager YAML file - google-bigquery

Question

I am trying to create a big query table using the deployment manager by following YAML file :
imports:
- path: schema.txt
resources:
- name: test
type: bigquery.v2.table
properties:
datasetId: test_dt
tableReference:
datasetId: test_dt
projectId: test_dev
tableId: test
schema:
fields: {{ imports["schema.txt"] }}
However, when I try to give the table schema definition via .txt file I get a parsing error. If I give the schema definition instead of .txt file then the script runs successfully. This method of importing the text file is given in the google cloud help. Can anyone help me with this?

Answer 1

I think the way the deployment manager is formatting the contents of the .txt file might be incorrect. A good way to debug this would be to collect HTTP request traces and comparing the difference between the two requests.

Answer 2

this is yaml we could get working using nested or repeated field in bigquery deployment manager.
# Example of the BigQuery (dataset and table) template usage.
#
# Replace `<FIXME:my_account#email.com>` with your account email.
imports:
- path: templates/bigquery/bigquery_dataset.py
name: bigquery_dataset.py
- path: templates/bigquery/bigquery_table.py
name: bigquery_table.py
resources:
- name: dataset_name_here
type: bigquery_dataset.py
properties:
name: dataset_name_here
location: US
access:
- role: OWNER
userByEmail: my_account#email.com
- name: table_name_here
type: bigquery_table.py
properties:
name: table_name_here
datasetId: $(ref.dataset_name_here.datasetId)
timePartitioning:
properties:
field:
type: DAY
schema:
- name: column1
type: STRUCT
fields:
- name: column2
type: string
- name: test1
type: RECORD
mode: REPEATED
fields:
- name: test2
type: string

Answer 3

Sample YAML for Creating View in BigQuery using Deployment manager:
Note: This YAML also shows how to create partitioning(_PARTITIONTIME) on a table (hello_table)
# Example of the BigQuery (dataset and table) template usage.
# Replace `<FIXME:my_account#email.com>` with your account email.
imports:
- path: templates/bigquery/bigquery_dataset.py
name: bigquery_dataset.py
- path: templates/bigquery/bigquery_table.py
name: bigquery_table.py
resources:
- name: dataset_name
type: bigquery_dataset.py
properties:
name: dataset_name
location: US
access:
- role: OWNER
userByEmail: my_account#email.com
- name: hello
type: bigquery_table.py
properties:
name: hello_table
datasetId: $(ref.dataset_name.datasetId)
timePartitioning:
type: DAY
schema:
- name: partner_id
type: STRING
- name: view_step
type: bigquery_table.py
properties:
name: hello_view
datasetId: $(ref.dataset_name.datasetId)
view:
query: select partner_id from `project_name.dataset_name.hello_table`
useLegacySql: False