I am facing a "Connection Error" issue with the table_query function, on one of my datasets
Example 1)
bq query "select * from (table_query([MYDOMAIN:MYPROJECT:MYDATASET], 'table_id contains \"20140522\"')) limit 1"
The above command always(100% reproducible) fails with error "BigQuery error in query operation: Connection error. Please try again.", after a minute or so.
But if I execute "bq ls -j",
I can see that it actually launched bqjob_r1c258fa26faab850_00000147f479ed5d_1
and that it really completed in 1 second.
Using "bq --format=prettyjson show -j bqjob_r1c258fa26faab850_00000147f479ed5d_1" and "bq head ....",
I can read the output of the job as well.
Example 2:
I am seeing 100% reproducible failures in querying the underlying TABLES table in this dataset as well.
bq query "select * from [MYDOMAIN:MYPROJECT:MYDATASET.__TABLES__] limit 1"
This also reports "BigQuery error in query operation: Connection error. Please try again."
But "bq ls -j" shows me that it succesfully completed job bqjob_r7e6eabcd0a39541_00000147f482cf90_1 in 0 seconds.
Like in previous example, I can use "bq head" to read the job output.
The questions I have are:
1) Why do the "command line tool(version 2.0.22), and the browser" always wait a long time after the job completed, and then lie to me that the job failed with "Connection error"?
2) Why does this "connection error" issue happen only with this particular dataset?
Related
As per the Google Cloud BQ Documentation, bq show --job=true myproject:US.bquijob_123x456_123y123z123c command should return the status of a BQ Job. But in my case I am getting the following error :-
BigQuery error in show operation: Not found: Job my-project:US.bqjob_r4bc4365eb9z97aa8_000001855ca75006_1
What could be the reason behind this error ? Also, I checked that I do have all the necessary permissions namely :-
roles/bigquery.admin
roles/bigquery.user
roles/bigquery.jobUser
Thanks!
I have 14 scheduled queries that run hourly from Google Sheets but they fail half of the time. I don't understand the error status though as the queries do run successfully half of the time. The error reads:
Error status: Invalid value Error while reading table: tester-253410.test1.Table_View_2_Agent_Targets, error message: Failed to read the spreadsheet. Errors: Deadline=118.888051456s; JobID: tester-253410:5e59a150-0000-2421-b469-001a1144591c
Is there anything that I can try?
Initially took me a while to figure out that ">" causes an issue in cmd and learned from one of the answers that escaping with ^ works. So after changing my query to below format it works fine.
bq query --use_legacy_sql=false --destination_table= (SELECT * FROM `dataset_ID.table_nm` GROUP BY AAA HAVING SUM(BBB)^>0 )
But now I am trying to run the same query through a file.
bq query --use_legacy_sql=false --destination_table= --flagfile="Y:query.sql"
However above it gives me an error if I have "^>".
Error in query string: Error processing job Syntax error: Unexpected ">" at [1:227]
If I don't have "^" in front of the ">" the query doesn't return an error but the result is empty..
Hoping someone could help out with the issue above.
Thanks!
I'm running the following bq command
bq load --source_format=CSV --skip_leading_rows=1 --max_bad_records=1000 --replace raw_data.order_20150131 gs://raw-data/order/order/2050131/* order.json
and
getting the following message when loading data into bq .
*************************************
Waiting on bqjob_r4ca10491_0000014ce70963aa_1 ... (412s) Current status: DONE
BigQuery error in load operation: Error processing job
'orders:bqjob_r4ca10491_0000014ce70963aa_1': Too few columns: expected
11 column(s) but got 1 column(s). For additional help: http://goo.gl/RWuPQ
Failure details:
- File: 844 / Line:1: Too few columns: expected 11 column(s) but got
1 column(s). For additional help: http://goo.gl/RWuPQ
**********************************
The message display only the file number .
checked the files content most of them are good .
gsutil ls and the cloud console on the other hand display file names .
how can I know which file is it according to the file number?
There seems to be some weird spacing introduced in the question, but if the desired path to ingest is "/order.json" - that won't work: You can only use "" at the end of the path when ingesting data to BigQuery.
I have a directory with a file named file1.txt
And I run the command:
bq query "SELECT * FROM [publicdata:samples.shakespeare] LIMIT 5"
In my local machine it works fine but in Compute Engine I receive this error:
Waiting on bqjob_r2aaecf624e10b8c5_0000014d0537316e_1 ... (0s) Current status: DONE
BigQuery error in query operation: Error processing job 'my-project-id:bqjob_r2aaecf624e10b8c5_0000014d0537316e_1': Field 'file1.txt' not found.
If the directory is empty it works fine. I'm guessing the asterisk is expanding the file(s) into the query but I don't know why.
Apparently the bq command which is located at /usr/bin/bq has the following script:
#!/bin/sh
exec /usr/lib/google-cloud-sdk/bin/bq ${#}
which expands the asterisk.
As a current workaround I'm calling /usr/lib/google-cloud-sdk/bin/bq directly.