Hello i have the next code and when i run it inside Bigquery is giving me the "correct" answer but when i put it into a sh script and execute the script inside google shell vm gives me the next error. Any thoughts?
I suppose the error lies in the need to multiply case/when statements results inside another case/when statement..
This is an example of what my code looks like:
SELECT CASE WHEN (
(CASE WHEN TABLE1.COL1 = 'X' THEN 0 ELSE 1 END) *
(CASE WHEN TABLE2.COL2 = 'Y' THEN 1 ELSE 0 END) *
(CASE WHEN (SELECT 0
FROM TABLE3
WHERE TABLE3.ID = TABLE2.ID) = 0 THEN 0 ELSE 1 END)) = 1
THEN (SELECT '111111') ELSE NULL END
FROM TABLE1
INNER JOIN TABLE2
ON TABLE1.ID = TABLE2.ID
FULL JOIN (SELECT COL1,'TRUE' FROM TABLE4) AS XX
ON XX.COL1 = TABLE1.COL1 AND XX.COL1 IS NULL
WHERE
TABLE1.COL3 = 'YY'
I cant provide the expected result but the result of that query gives me the next error: Parenthesized expression cannot be parsed as an
expression, struct constructor, or subquery at...
I'll post debugging tips as an answer since I think you'll probably find the problem this way. From a command-line prompt, list recent jobs:
bq ls -j --all
The failed query job will probably be at the top. Copy the job ID and use it with the next command:
bq --prettyformat=json show -j YOUR_JOB_ID
This will print out the complete job configuration as well as the error message. What I suspect you'll see is that the query is garbled; the quotes or some other character may have caused unexpected behavior when interpreted by the shell. When executing queries from the command-line, it's a good idea to put the contents in a file, then pipe it as input to the bq tool, e.g.
bq query --use_legacy_sql=false < query.sql
This prevents the shell from intercepting any part of the query as a command.
Thanks #Elliott Brossard, i resolved the problem with literally no change in the code hah. So what i did was rewrite the bq query sentence like: bq query --destination_table=.. --use_legacy_sql=false --replace 'QUERY' instead of assign the query to a variable and then execute it with an echo | bq query sentence..
Thanks
Related
I'm getting an unexpected error when using a case statement with a query using termsql. Is there something I'm doing wrong with it, perhaps invalid sql, or might this be an issue with the termsql parser (sqlparse version 0.4.2)?
Here's a simplified example of what I'm seeing (with newlines to help readability):
echo "Fred|10\nJulie|15\nFatima|42\nJulie|10\nFatima|18" | \
termsql -c 'name,activities' -t test -d'|' \
"select case name when 'Fatima' then 'Primary' else 'Secondary' end primary_normalized,sum(activities) from test group by primary_normalized"
That yields and error of:
Error: near line 1: near "'Fatima'": syntax error
The goal there is to group things by "Primary" and "Secondary" based on a chosen name.
This same query will do it, but the output isn't quite what I want:
echo "Fred|10\nJulie|15\nFatima|42\nJulie|10\nFatima|18" | \
termsql -c 'name,activities' -t test -d'|' \
"select name,sum(activities) from test group by case name when 'Fatima' then 'Primary' else 'Secondary' end"
The output from that is
Fatima|60
Fred|35
...which is kind of weird since the select is operating on the name field, but the group is operating on the case statement output. Not unexpected, but not desirable, where I'd really like the first option to work.
When I run the following query from the BQ UI, it runs fine and gives the desired output.
But when I run the same from the command line, it gives the following error
bq query --destination_table
Chicago_Traffic_Sensor_Data.Latest_Traffic_Data_with_Geocoding
--replace --use_legacy_sql=false 'SELECT segmentid, _lif_lat, start_lon, _lit_lat, _lit_lon, _traffic, _last_updt, CASE WHEN
_traffic < 20 THEN '#FF0000' WHEN _traffic >= 20 and _traffic < 40 THEN '#FFFF00' WHEN _traffic >= 40 THEN '008000' ELSE '#666666' END as
strokeColor FROM (SELECT *, ROW_NUMBER() OVER (PARTITION BY segmentid
ORDER BY _last_updt DESC) col FROM
Chicago_Traffic_Sensor_Data.traffic_sensor_data) x WHERE x.col = 1
ORDER BY segmentid' Error in query string: Error processing job
'bedrock-gcp-testing:bqjob_r5d944587b08c4e54_000001626fa3f61d_1':
Syntax error: Unexpected end of statement at [1:412]
You need to escape any characters in your SQL that the command line is trying to interpret. What I find much easier & quicker is to put my SQL in a file, and pipe it instead. For example:
bq query --destination_table grey-sort-challenge:partitioning_magic.foobarred --use_legacy_sql=false "$(cat data.sql)"
how to execute a HIVE query in background when the query looks like below
Select count(1) from table1 where column1='value1';
I am trying to write it using a script like below
#!/usr/bin/ksh
exec 1> /home/koushik/Logs/`basename $0 | cut -d"." -f1 | sed 's/\.sh//g'`_$(date +"%Y%m%d_%H%M%S").log 2>&1
ST_TIME=`date +%s`
cd $HIVE_HOME/bin
./hive -e 'SELECT COUNT(1) FROM TABLE1 WHERE COLUMN1 = ''value1'';'
END_TIME=`date +%s`
TT_SECS=$(( END_TIME - ST_TIME))
TT_HRS=$(( TT_SECS / 3600 ))
TT_REM_MS=$(( TT_SECS % 3600 ))
TT_MINS=$(( TT_REM_MS / 60 ))
TT_REM_SECS=$(( TT_REM_MS % 60 ))
printf "\n"
printf "Total time taken to execute the script="$TT_HRS:$TT_MINS:$TT_REM_SECS HH:MM:SS
printf "\n"
but getting error like
FAILED: SemanticException [Error 10004]: Line 1:77 Invalid table alias or column reference 'value1'
let me know exactly where I am doing mistake.
Create a document named example
vi example
Enter the query in the document and save it.
create table sample as
Select count(1) from table1 where column1='value1';
Now run the document using the following command:
hive -f example 1>example.error 2>example.output &
You will get the result as
[1]
Now disown the process :
disown
Now the process will run in the background. If you want to know the status of the output, you may use
tail -f example.output
True #Koushik ! Glad that you found the issue.
In the query, bash was unable to form the hive query due to ambiguous single quotes.
Though SELECT COUNT(1) FROM Table1 WHERE Column1 = 'Value1' is valid in hive,
$hive -e 'SELECT COUNT(1) FROM Table1 WHERE Column1 = 'Value1';' is not valid.
The best solution would be to use double quotes for the Value1 as
hive -e 'SELECT COUNT(1) FROM Table1 WHERE Column1 = "Value1";'
or use a quick and dirty solution by including the single quotes within double quotes.
hive -e 'SELECT COUNT(1) FROM Table1 WHERE Column1 = "'"Value1"'";'
This would make sure that the hive query is properly formed and then executed accordingly. I'd not suggest this approach unless you've a desperate ask for a single quote ;)
I am able to resolve it replacing single quote with double quote. Now the modified statement looks like
./hive -e 'SELECT COUNT(1) FROM Table1 WHERE Column1 = "Value1";'
I am trying to migrate data from mysql to hive.I am not able to write a subquery case when statement with IN clause.This is my query. Can you Please help in this regard. AM i not following the proper syntax .
CREATE TABLE HIVE_TPCE_TEMP.TMP_CDMA_CD AS
SELECT A.DRI,C.BOUND_ID,A.CT_ID,A.CD_ID,A.CID,
A.TID,A.TASK_SEQ_ID,A.DIV_ID,C.BLOCK_GROUP_ID,C.ZIP_CODE,C.ROAD_CATEGORY_ID,A.RXPOWER,"${hiveconf:C_CDMA_DEVICE_ONLINE_RXPOWER_METRIC_ID}" METRIC_ID,
CASE WHEN
((A.DRI,A.DIV_ID,A.RFID) in (SELECT DRI,DIV_ID,HOME_RFID FROM HIVE_TPCE_TEMP.TMP_HOME_NETWORKS)) THEN
CASE WHEN MODE IN ('A','N') THEN "${hiveconf:HAD}" ELSE "${hiveconf:HD}" END
WHEN (COALESCE(A.RFID,0) = 0) AND ((A.DRI,A.DIV_ID,D.FR,D.SUBBAND) IN (SELECT DRI,DIV_ID,HOME_FR,
HOME_SUBBAND FROM HIVE_TPCE_TEMP.TMP_HOME_NETWORKS))
THEN CASE WHEN MODE IN ('A','N') THEN "${hiveconf:HAD}" ELSE "${hiveconf:HD}" END
ELSE CASE WHEN MODE IN ('A','N') THEN "${hiveconf:PAI}" ELSE "${hiveconf:PDI}" END END HPDA_ID
FROM HIVE_TPCE.VW_CDMA_CD A INNER JOIN HIVE_TPCE.STG_CURRENT_FILES B
ON A.DRI = B.DRI AND A.SOURCE_FILE_ID = B.SOURCE_FILE_ID
INNER JOIN
HIVE_TPCE.WRK_LOCATION C
ON A.DRI = C.DRI AND A.LOCATION_ID = C.LOCATION_ID
INNER JOIN
HIVE_TPCE.LU_RADIO D
ON A.RADIO_ID = D.RADIO_ID WHERE A.CID > 0 AND D.MODE IN ('A','N') AND A.RXPOWER IS NOT NULL AND A.CALL_RESULT_ID BETWEEN 1 AND 16;
My error signature is
FAILED: ParseException line 10:42 mismatched input ',' expecting ) near 'DRI' in expression specification
According to the Hive Language Manual: "Hive supports subqueries only in the FROM clause".
Your CASE WHEN is part of the SELECT clause, but it includes includes a SELECT subquery. Seems like that is not supported, so your syntax is not correct (in Hive).
Perhaps you could stage the data in MySQL using the query you have and then load it into Hive using a simple SELECT without CASE WHEN?
See official document.
It says
Assumptions
We plan to limit the scope with the following assumptions and limitations.
Subqueries could only be top-level expressions in SELECT. That is, subqueries in complex expressions, aggregates, UDFs, etc. will not be supported for now
I am running the following query, but no rows are returned even though a record exists that should match the query.
SELECT
*
FROM
tblsignup
WHERE
usr_email='amir#gmail.com'
AND
(status=1 or status=2)
You should try by simplifying the query (yeah...even if it's so simple)
try this
Select * from tblsignup
then
Select * from tblsignup where
usr_email = 'amir#gmail.com'
then
Select * from tblsignup where
usr_email='amir#gmail.com' and
status > 0
//I know you won't use > 0 at the end, but we want to eliminate the most cause of error we simplify by > 0 only to be easier to read
Tell us from where you start getting 0 line, this could lead us to the problem, I know I already had a problem like that with a field named "date", because date is already used by MySQL, funny MySQL still let me use that fieldname tho.
Try this:
select * from `tblsignup` where `usr_email`='amir#gmail.com' and (`status`=1 or `status`=2)
I have a feeling "status" might be reserved for something special. It might be worth a shot changing it to `status`.
Try wrapping brackets around the status column name:
SELECT *
FROM tblsignup
WHERE usr_email = 'amir#gmail.com'
AND ([status] = 1
OR [status] = 2);
EDIT
After reading your comment, why not use:
SELECT *
FROM tblsignup
WHERE usr_email = 'amir#gmail.com'
AND [status] > 0;
May it be that your column or table has case sensitive collation and the address is typed different ('Amir...')? As your query is correct SQL. You can find that with:
EXEC sp_help DatabaseName