I want to create a hive script that uses as database one of two given parameters, whichever is not null.
My hive-test.sql is this:
set db_name = coalesce(${hiveconf:dbOne}, ${hiveconf:dbTwo});
use ${hiveconf:db_name};
show tables;
and I run it with:
hive -hiveconf dbOne=my_database -f hive-test.sql
and I am getting:
FAILED: ParseException line 2:12 missing EOF at '(' near 'coalesce'
I should note that if I change the first line in script to:
set db_name = my_database;
it works.
I can't figure out what I did wrong. Your assistance is appreciated.
This feature is not available in Hive.
Do variable assignment in the shell, for example like here: setting-a-shell-variable-in-a-null-coalescing-fashion and pass it to the Hive.
Related
We have a query in which we have defined more than 50 variables.
we call this hql via shell script, most of the times i get into syntax issue where i have not defined hive variables properly in the query.
Example
set hive var0=value0;
set hive var1=value1;
set hive var2=value2;
select * from ${hiveconf:var0} where col1=${hiveconf:var1} and col2=${hiveconf:var2};
I want to to check the above query result after replacing hive variables,
So is there a way to check if the variables are parsed in the right way or are there any syntax errors.
Please let me know for any alternatives as well.
Better use hivevar namespace for the same.
You can print all variable using ! echo command:
set hivevar:var0=value0;
hive> ! echo Variable hivevar:var0 is ${hivevar:var0};
Result:
Variable hivevar:var0 is value0
Also use explain extended <query> - it will print detailed query plan with predicates and fail if it is syntax error.
Update:
Also you can use SELECT for doing the same and Hive can execute simple queries without MR started if hive.fetch.task.conversion is set to more or minimal. If you are using Qubole, add also limit 1 to the query:
set hive.fetch.task.conversion=more;
select 'Variable hivevar:var0 is', '${hivevar:var0}' limit 1;
Why you may need to do this using SELECT? For example for easy checking parameter using casting or some UDF. If you need to check if parameter is of type DATE, use
set hive.fetch.task.conversion=more;
select 'Variable hivevar:var0 is', date '${hivevar:var0}' limit 1;
In this case if ${hivevar:var0} is not date, then type cast exception will be thrown and script execution terminated.
along with hivevar namespace, we can use one more property hive.root.logger=INFO,console.
this will display the query after replacing the variable value, from which we can find out the issue.
cat test.hql
set hivevar:var1=${hivevar:var11};
set hivevar:var2=2345;
select ${hivevar:var11};
select ${hivevar:var2};
hive command - hive --hiveconf hive.root.logger=INFO,console --hivevar var11=1234 -f test.hql
output on console
select 1234
2018-10-17T08:23:31,632 INFO [main] ql.Driver: Completed executing command(queryId=-4dd6-493f-88be-03810f847fe7); Time taken: 0.003 seconds
OK
2018-10-17T08:23:31,632 INFO [main] ql.Driver: OK
2018-10-17T08:23:31,670 INFO [main] io.NullRowsInputFormat$NullRowsRecordReader: Using null rows input format
1234
I've got a param which is like "This is a param", and I'm going to pass it to below hiveQL:
hive -hivevar sys_nm="This is a param" -e 'select * from rd_sys where rd_sys_nm=${hivevar:sys_nm}'
But Hive returned below error message:
Logging initialized using configuration in jar:file:/opt/mapr/hive/hive-0.13/lib/hive-common-0.13.0-mapr-1409.jar!/hive-log4j.properties
FAILED: ParseException line 1:49 missing EOF at 'is' near 'This'
g4t7491_[mgr#g4t7491 ~]$
Does anyone know how to pass it normally?
Hive var don't work like hiveconf where you need to apply "hiveconf:somthing" in the code
when declaring hivevar just add the var name like this -> ${var_name}
for example:
through command line:
hive -hivevar MONTH_VAR='11' -e "select * from table where month=${MONTH_VAR};"
you can also declair through the script:
set hivevar:MONTH_VAR=11;
-- so query would look like this (no hiveconf):
set hivevar:MONTH_VAR=11;
SELECT * from table where month=${MONTH_VAR};
You need to put the string in single quotes for it to parse correctly as a string inside the sql after interpolation.
hive -hivevar sys_nm="'This is a param'" -e 'select * from rd_sys where rd_sys_nm=${hivevar:sys_nm}'
I'm currently using bash on CentOS. DB2 is installed and db2 is on my path.
I have a few lines in a script which are supposed to update my db2 database, but they aren't working. As a minimal reproduction, I can do the exact same thing right in the bash command line and get the same error. Here's that reproduction:
$ db2 connect to PLT02345 user uni using uni; db2 update USM_USER set STATUS = 1 where NAME = 'asm_admin'
I expect this to set STATUS to 1 for everything in PLT02345.USM_USER where the NAME is currently asm_admin.
Instead, I get an error about "ASM_ADMIN" not being valid in the context where it's used. Here's the full output:
Database Connection Information
Database server = DB2/LINUXX8664 10.1.2
SQL authorization ID = UNI
Local database alias = PLT02345
DB21034E The command was processed as an SQL statement because it was not a
valid Command Line Processor command. During SQL processing it returned:
SQL0206N "ASM_ADMIN" is not valid in the context where it is used.
SQLSTATE=42703
I'm confused - what about this makes it not valid? Is bash somehow mutilating the command and not passing everything as it should to db2?
If you're running this from the command line, Bash will drop the 's off 'asm_admin' because it simply assumes you're passing a string. The end result is the SQL becoming WHERE name = asm_admin which is invalid.
To correct this, you need to quote your whole command:
db2 "update USM_USER set STATUS = 1 where NAME = 'asm_admin'"
I am using Apache-Hadoop and Hive as a setup. The hive do get connected with the Hadoop,tables are also created. But with the command show tables this exception occurs:Failed with the exception java.io.IOException:org.apache.hadoop.mapred.InvalidInputException:Input Pattern file:/tmp/${hduser}/034cbea3-2b60-49f5-8284-d6fba957dda3/hive_2015-06-18_05-10-04_183_5811447541305606525-1/-local-10000 matches 0 files
What is the exception and how should i solve it. Please help me.
So please check out the file: vim $HIVE_HOME/conf/hive-site.xml, and you should check the <name>system:user.name, it's value should be hduser not ${hduser}.
please take the right/correct name for the user.
I am currently trying to integrate an sql statement into a shell script, But facing major syntax issue:
My statement in the script:
su - <sid>adm -c 'hdbsql -U SYSTEM export "'SCHEMA'"."'*'" as binary into "'Export Location'" with reconfigure'
I get the following error:
* 257: sql syntax error: incorrect syntax near "*": line 1 col 16 (at pos 16) SQLSTATE: HY000
Would really appreciate if anyone could help me with this.
Thanks and Regards,
AK
Your command line doesn't make much sense to me. It starts with
su - <sid>adm
which means that you are redirecting the contents of the file "sid" into "su" and then redirecting the result of that operation into the file "adm".
Second problem is that in the command you are giving to adm, the single quotes end right before the "" which means, that the "" will get interpreted by the shell as a file glob:
-c 'hdbsql -U SYSTEM export "'SCHEMA'"."'*'" as binary into "'Export Location'" with reconfigure'
You'll need to escape those single quotes like this: "\'".
But I think your problem solving approach is not good. Try to reduce to problem and only then start adding additional things to it. So first try to execute the SQL statement from the "hdbsql" shell. Does it work?
$ hdbsql
> YOUR SQL STATEMENT HERE
Once that works, try to execute the SQL statement from the unix shell as a user:
$ hdbsql -U SYSTEM export ...
Once that works, try to execute it via su
$ su - ...