As per the Google Cloud BQ Documentation, bq show --job=true myproject:US.bquijob_123x456_123y123z123c command should return the status of a BQ Job. But in my case I am getting the following error :-
BigQuery error in show operation: Not found: Job my-project:US.bqjob_r4bc4365eb9z97aa8_000001855ca75006_1
What could be the reason behind this error ? Also, I checked that I do have all the necessary permissions namely :-
roles/bigquery.admin
roles/bigquery.user
roles/bigquery.jobUser
Thanks!
Related
I was running into troubles today while running Airflow and airflow-dbt-python. I tried to debug a bit using the logs and the error shown in the logs was this one:
[2022-12-27, 13:53:53 CET] {functions.py:226} ERROR - [0m12:53:53.642186 [error] [MainThread]: Encountered an error:
Database Error
Expecting value: line 2 column 5 (char 5)
Quite a weird one.
Possibly check your credentials file that allows DBT to run queries on your DB (in our case we run DBT with BigQuery), in our case the credentials file was empty. We even tried to run DBT directly in the worker instead of running it through airflow, giving as a result exactly the same error. Unfortunately this error is not really explicit.
I am executing dbt run -s model_name on CLI and the task completes successfully. However, when I run the exact same command on dbt cloud, I get this error:
Syntax or semantic analysis error thrown in server while executing query.
Error message from server: org.apache.hive.service.cli.HiveSQLException:
Error running query: org.apache.spark.sql.AnalysisException: cannot
resolve '`pv.meta.uuid`' given input columns: []; line 6 pos 4;
\n'Project ['pv.meta.uuid AS page_view_uuid#249595,
'pv.user.visitorCookieId AS (80) (SQLExecDirectW)")
it looks like it fails recognizing 'pv.meta.uuid' syntax which extract data from a json format. It is not clear to me what is going on. Any thoughts? Thank you!
I'm following this documentation on how to use the 'bq load' command to load data into BigQuery from a CSV in Google Cloud: https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-csv
However, I keep getting the same error: "Syntax error: Expected end of input but got identifier "bq" at [12:1]"
This is the very first part of my code, why is it expecting to get an end of input identifier? It's the beginning of my code. I have researched all over the internet and cannot find somebody with a similar issue.
Thank you in advance for your help.
Here's the code:
bq load \
--source_format=CSV \
cpo_metrics.12mo_hc \
gs://unity-ai-people-prd-test/12mo_hc.csv \
--autodetect=TRUE;
Here's what it looks like when I'm trying to run in the BQ console:
[1]: https://i.stack.imgur.com/oIDut.png
I'm running the following bq command
bq load --source_format=CSV --skip_leading_rows=1 --max_bad_records=1000 --replace raw_data.order_20150131 gs://raw-data/order/order/2050131/* order.json
and
getting the following message when loading data into bq .
*************************************
Waiting on bqjob_r4ca10491_0000014ce70963aa_1 ... (412s) Current status: DONE
BigQuery error in load operation: Error processing job
'orders:bqjob_r4ca10491_0000014ce70963aa_1': Too few columns: expected
11 column(s) but got 1 column(s). For additional help: http://goo.gl/RWuPQ
Failure details:
- File: 844 / Line:1: Too few columns: expected 11 column(s) but got
1 column(s). For additional help: http://goo.gl/RWuPQ
**********************************
The message display only the file number .
checked the files content most of them are good .
gsutil ls and the cloud console on the other hand display file names .
how can I know which file is it according to the file number?
There seems to be some weird spacing introduced in the question, but if the desired path to ingest is "/order.json" - that won't work: You can only use "" at the end of the path when ingesting data to BigQuery.
I am facing a "Connection Error" issue with the table_query function, on one of my datasets
Example 1)
bq query "select * from (table_query([MYDOMAIN:MYPROJECT:MYDATASET], 'table_id contains \"20140522\"')) limit 1"
The above command always(100% reproducible) fails with error "BigQuery error in query operation: Connection error. Please try again.", after a minute or so.
But if I execute "bq ls -j",
I can see that it actually launched bqjob_r1c258fa26faab850_00000147f479ed5d_1
and that it really completed in 1 second.
Using "bq --format=prettyjson show -j bqjob_r1c258fa26faab850_00000147f479ed5d_1" and "bq head ....",
I can read the output of the job as well.
Example 2:
I am seeing 100% reproducible failures in querying the underlying TABLES table in this dataset as well.
bq query "select * from [MYDOMAIN:MYPROJECT:MYDATASET.__TABLES__] limit 1"
This also reports "BigQuery error in query operation: Connection error. Please try again."
But "bq ls -j" shows me that it succesfully completed job bqjob_r7e6eabcd0a39541_00000147f482cf90_1 in 0 seconds.
Like in previous example, I can use "bq head" to read the job output.
The questions I have are:
1) Why do the "command line tool(version 2.0.22), and the browser" always wait a long time after the job completed, and then lie to me that the job failed with "Connection error"?
2) Why does this "connection error" issue happen only with this particular dataset?