Inserting data into a BigQuery table through Python - sql

Question

I am trying to insert data into an existing BigQuery Table. But I'm struggling. I am sorry but I am new to BigQuery so I am surely missing something. I am using the BigQuery API and I want to append/insert the data through Python.
from google.cloud import bigquery
from google.api_core.exceptions import NotFound
import logging
import os
run_transformation_query("A_DAILY_ORDER_NORMALIZE.sql", pipeline_settings)
def run_transformation_query(query_file, pipeline_settings):
result = invoke_transformation_queries(query_file, pipeline_settings)
error_value = result.error_result
def invoke_transformation_queries(sql_file_name, pipeline_settings):
logging.basicConfig(filename='error.log', level=logging.ERROR)
client = bigquery.Client(project=pipeline_settings['project'])
print("check")
#debug_string = f"{pipeline_settings['repo_path']}sql/{sql_file_name}"
# sql = get_query_text(
# f"{pipeline_settings['repo_path']}sql/{sql_file_name}")
file_path = os.path.join('sql', sql_file_name)
sql = get_query_text (file_path)
def get_query_text(file_path):
with open(file_path, 'r') as file:
query_text = file.read()
return query_text
My SQL File is as follows:
DECLARE project_name STRING DEFAULT 'xxx-dl-cat-training';
DECLARE dataset_name STRING DEFAULT 'xxxxxxx_bootcamp_dataset_source';
DECLARE table_name_source STRING DEFAULT 'xxxxxxx-bootcamp-table-source';
DECLARE table_name_target STRING DEFAULT 'xxxxxxx-bootcamp-table-target';
DECLARE todays_date STRING;
SET todays_date = FORMAT_DATE("%Y%m%d", CURRENT_DATE());
WITH ORDERS AS (
SELECT
date_order,
area,
customer_name,
SPLIT(order_details, ',') AS items_list,
total_transaction
FROM
`${project_name}.${dataset_name}.${table_name_source}`
), TRANSFORMED_ORDERS AS (
SELECT
date_order,
area,
customer_name,
TRIM(IFNULL(
REGEXP_REPLACE(
item,
r'-\s*\d+(\.\d+)?$',
''
),
item
)) AS item_name,
CAST(NULLIF(TRIM(REGEXP_EXTRACT(item, r'-\s*\d+(\.\d+)?')), '') AS FLOAT64) AS item_price,
total_transaction
FROM ORDERS, UNNEST(items_list) as item
WHERE CAST(NULLIF(TRIM(REGEXP_EXTRACT(item, r'-\s*\d+(\.\d+)?')), '') AS FLOAT64) IS NOT NULL
)
CREATE OR REPLACE TABLE `${project_name}.${dataset_name}.${table_name_target}`
SELECT *
FROM TRANSFORMED_ORDERS;
Once my Subquery ends , whatever I get this error
Expected "(" or "," or keyword SELECT but got keyword CREATE at [36:1]'}
I am not sure where I ma messing up. Any help will be appreciated. I have run the transformation in the BigQuery UI and i am happy with the transformation. It all works ok
I shall be grateful if someone can help