I am running simple select query in BigQuery GCP Console and it works perfectly fine. But, when I run the same query using BQ CLI, it fails.
When I run the same query without the "WHERE" clause, it works.
SELECT field_path FROM `GCP_PROJECT_ID.MY_DATASET.INFORMATION_SCHEMA.COLUMN_FIELD_PATH`
WHERE table_name="MY_TABLE_NAME"
Below is the error message
Error in query string: Error processing job 'GCP_project_ID:jobidxxxxx': Unrecognized name:
MY_DATASET at [1:1xx]
I have tried the following "WHERE" clauses as well. None of these work as well.
... WHERE table_name IN ("MY_TABLE_NAME")
... WHERE table_name like "%MY_TABLE_NAME%"
I have reproduced your query on my own datasets using the Command Line tool and it worked fine. This is the command I ran:
bq query --use_legacy_sql=false 'SELECT field_path FROM `<projectName>.<DatasetName>.INFORMATION_SCHEMA.COLUMN_FIELD_PATHS`where table_name="<TableName>"'
Related
Beginner
Im trying to use sqoop import from Oracle to HDFS
I get error message -
SQL command not properly ended , Import Failed : java:IOexception : No >column to generate for ClassWriter
my query is
--query "select a,b,c from db.table where date = to_date('9999-12-31','yyyy-MM-dd')" where \$conditions"
Your query contains two WHERE clause, this is syntax error. Use AND to add $conditions to the same WHERE, whole query should be double-quoted, including $CONDITIONS, it is the part of the same WHERE, and in your query double-quotes are broken.
Like this:
--query "select a,b,c from db.table where date = to_date('9999-12-31','yyyy-MM-dd') and \$CONDITIONS"
Check what exactly sqoop executes, does it resolve correctly or not (conditions passed correctly), use --verbose to get more information and fix query accordingly.
I am trying to run a very simple query that just select all the rows based upon some multiple criteria and all are inclusive i.e. I am using AND in select HiveQL statement. The table is an external table in Hive and the storage handler is phoenix, so I checked in phoenix also about that query and it is working fine, but in Hive, it is showing some java IO exception error which I am not able to get where I am wrong. The query I am using is:
SELECT * FROM msme_api.msme_jk WHERE gender='Male' AND socialcategory='General';
The complete error message is:
Error: java.io.IOException: org.apache.hadoop.hive.ql.exec.UDFArgumentLengthException: The operator 'AND' accepts at least 2 argument. (state=,code=0)
I am trying for external and for internal Hive tables, In both, the issues are still the same but when I give order by, OR statement then it's surprising that it works.
SELECT * FROM msme_api.msme_jk WHERE gender='Male' AND socialcategory='General' order by aid;
SELECT * FROM msme_api.msme_jk WHERE gender='Male' OR socialcategory='General';
Both works fine but with AND, I am getting the error.
I still confused about how hive is taking and processing the above queries and why I am not able to execute simple select statement. Any help will be appreciated.
I'm running the BigQuery command line shell and I'm not able to successfully run multi-line queries (aka queries with line breaks) because whenever I paste the query into the shell each line gets run individually instead of the whole thing together.
For example,
select * from table
works fine because it is in one line but if I try to run
select
*
from
table
it does not work because each line gets run separately.
Is there any way to get this to work?
The query command creates a query job that runs the supplied SQL query. in the documentation Using the bq command-line tool you can find some examples like:
bq query --nouse_legacy_sql \
'SELECT
COUNT(*)
FROM
`bigquery-public-data`.samples.shakespeare'
Honestly I too couldn't find decent help on this, so I created a bit of a workaround using the CONCAT function. Notice that I have a space before the FROM clause. Hope his helps.
DECLARE table_name STRING DEFAULT '20220214';
DECLARE query STRING;
SET query = CONCAT("""select user_pseudo_id, user_id""",
""" FROM `app.analytics.events_""" || table_name || """` where user_id IS NOT NULL group by user_pseudo_id, user_id""");
EXECUTE IMMEDIATE query;
I'm trying to run queries in standard mode via command line. I'm using recently version of this bigquery-2.0.17.
firstly I tried to use bq query command:
bq query --use_legacy_sql=False "SELECT string_col FROM `source_name`"
And get exception
FATAL Flags parsing error: Unknown command line flag 'use_legacy_sql'
After, I've run in shell mode
project_name> query --use_legacy_sql=False "SELECT string_col FROM `source_name`"
And get:
float() argument must be a string or a number
Could you advise me, how can I run query in command line.
Thanks
The commands you are running are all correct. The only thing I do differently is I read the queries from a file so I don't have problems with my bash trying to interpret the ` sign, like so:
cat query.sql | bq query --use_legacy_sql=False
(You won't have this issue in the bq shell interpreter).
Supposing your queries are all correct and following the Standard syntax (you can also run a simple query like "select [1, 2]" to see if it works, if it does, then maybe your source_name has some issue going on), the only thing that comes to mind is maybe if you try to reinstall gcloud and even update it (currently we are at bq version 2.0.24), as this seems to be more related to environment rather than command syntax.
Need to use gcloud installation instead of alone bq.
Thanks
I am using bq command line to read data from multiple tables with similar names, and have subsidiary query problem.
Simple example:
bq query --append=true --destination_table=xxxxxxxxxxxx:my_table.result
SELECT udid FROM (TABLE_QUERY(xxxxxxxxxxxx:my_table,'table_id
CONTAINS "data_2014_05_05"'))
When I run that query in the BQ GUI I get the results. However, when I do it from
the command line I get: "Error evaluating subsidiary query".
In addition, if I test only the subsidiary query from the command line:
bq query "SELECT * FROM xxxxxxxxxxxx:my_table.__TABLES__
WHERE table_id CONTAINS 'data_2014_05_05'"
it works fine and I get the tables' info.
So why is there "Error evaluating subsidiary query" in the main query?
Is there a problem with subsidiaries in bq command line?
There is no example whatsoever online or in the documentation.
Remove/escape the special characters, such as the quotes, in your query when passing it to the command line tool.