I am trying to pass beeline variable. source file name command is not able to replace the variable with the value. Same command working using hive -f.Please find commands below.
Note:
cat >dev_env.hql
set hivevar:l1_gcb_trxn_db_path="hdfs://quickstart.cloudera:8020/user/hive/warehouse/l1_app_gcb/process/l1_gcb_trxn";
SET hivevar:myvar=dev;
**source /data/1/appgcb/config/"${myvar}"_env.hql;**
create database l1_gcb_trxn
location ${l1_gcb_trxn_db_path} ;
Error
jdbc:hive2://quickstart.cloudera:10000/def> source /data/1/appgcb/config/"${myvar}"_env.hql;
0: jdbc:hive2://quickstart.cloudera:10000/def> create database l1_gcb_trxn
. . . . . . . . . . . . . . . . . . . . . . .> location ${l1_gcb_trxn_db_path} ;
Error: Error while compiling statement: FAILED: ParseException line 2:9 mismatched input '$' expecting StringLiteral near 'location' in database location specification (state=42000,code=40000)
But hive -f is able to replace the variable.
hive -f "${PROJECT_DIR}/tmp/create_db_l1_gcb_trxn.hql" --hivevar db_env=${db_env}
I was able to connect to hive using beeline but don't have access to manipulate anything there. below works for to get access to hive.
#!/bin/bash
echo "Executing the hive query - starts"
beeline --showHeader=true --outputformat=csv2 -u jdbc:hive2://hostname.com:10000 -n username -p password -e "select * from tablename limit 5;"
echo "Executing the hive query - ends"
Related
I am working on a program in bash that checks for valid JSON files before loading them into a table. The process first runs f_check_valid_json to verify the JSON. This process runs f_exe_sql_stmnt() that returns a column of bad files, stored in variable bad_fl_list. I would like to be able to input bad_fl_list in the WHERE clause of my update and delete sections of the function.
Right now, the SQL fails when there is more than one JSON file ID in bad_fl_list
f_exe_sql_stmnt(){
db=$1
sql_str=$2
psql -d "$db" -Atc "$sql_str"
if [ $? -gt 0 ]
then
echo "======================================================================="
echo "***Error: Database error while executing sql statement($sql_str)..."
exit 123
fi
}
f_check_valid_json() {
echo "*** checking for valid JSON format***"
sql_stmnt="Select json_fl_id from json_stgng where is_valid_json(json_datarec_fl) = false;"
bad_fl_list=$(f_exe_sql_stmnt "$t_db" "${sql_stmnt}")
echo "BAD FILE LIST: ${bad_fl_list}"
if [ ! -z "$bad_fl_list" ]
then
echo "BAD JSON LIST IS NOT EMPTY"
echo "*** updating balancing table to reflect bad file ***"
updt_bal_log_str="UPDATE ${bal_log_tbl} SET trgt_load_stus_cd ='F' where json_fl_id in ($bad_fl_list);"
f_exe_sql_stmnt "$DB" "$updt_bal_log_str"
echo "*** deleting bad JSON file record from staging with file ID: ${bad_fl_list}"
delete_stmnt="delete from ${stg_tbl} where json_fl_id in ($bad_fl_list);"
f_exe_sql_stmnt "$t_db" "${delete_stmnt}"
fi
}
Here is some example output from the logs:
+ psql -d dedw -Atc 'UPDATE json_load_bal_dtl_log SET trgt_load_stus_cd ='\''F'\'' where json_fl_id in (O21181043417
O21181043417
O21181003641);'
ERROR: syntax error at or near "O21181043417"
LINE 2: O21181043417
^
I'm trying to load a tab separated file to a HIVE text file table using hiveconf parameters as below -
load data local inpath '${hiveconf:TEXT_FILE}' into table ${hiveconf:HIVE_TABLE};
But when I run this .hql file as below
hive -hiveconf DB=$DB TEXT_FILE="$text_file_name" HIVE_TABLE=$HIVE_TABLE -f file_load.hql
I get the below error -
NoViableAltException(16#[202:1: tableName : (db= identifier DOT tab= identifier -> ^( TOK_TABNAME $db $tab) |tab= identifier -> ^( TOK_TABNAME $tab) );])
at org.antlr.runtime.DFA.noViableAlt(DFA.java:158)
......
......
FAILED: ParseException line X:YY cannot recognize input near '$' '{' 'hiveconf' in table name
I searched on google and understood that it's due to hive keyword but I have already created the table successfully and when I load the file by hardcoding the file name and table name then the data gets loaded! Please help me here!
Thank you!
You passing context variables incorrectly. it should be -hiveconf before each variable:
hive -hiveconf DB=$DB -hiveconf TEXT_FILE="$text_file_name" -hiveconf HIVE_TABLE=$HIVE_TABLE -f file_load.hql
I've got a param which is like "This is a param", and I'm going to pass it to below hiveQL:
hive -hivevar sys_nm="This is a param" -e 'select * from rd_sys where rd_sys_nm=${hivevar:sys_nm}'
But Hive returned below error message:
Logging initialized using configuration in jar:file:/opt/mapr/hive/hive-0.13/lib/hive-common-0.13.0-mapr-1409.jar!/hive-log4j.properties
FAILED: ParseException line 1:49 missing EOF at 'is' near 'This'
g4t7491_[mgr#g4t7491 ~]$
Does anyone know how to pass it normally?
Hive var don't work like hiveconf where you need to apply "hiveconf:somthing" in the code
when declaring hivevar just add the var name like this -> ${var_name}
for example:
through command line:
hive -hivevar MONTH_VAR='11' -e "select * from table where month=${MONTH_VAR};"
you can also declair through the script:
set hivevar:MONTH_VAR=11;
-- so query would look like this (no hiveconf):
set hivevar:MONTH_VAR=11;
SELECT * from table where month=${MONTH_VAR};
You need to put the string in single quotes for it to parse correctly as a string inside the sql after interpolation.
hive -hivevar sys_nm="'This is a param'" -e 'select * from rd_sys where rd_sys_nm=${hivevar:sys_nm}'
I have written a simple sh file to retrieve data from Oracle SQL but getting error. Following is my code:
. $HOME/.profile
function assignVariables
{
ID="finapp"
PASS="finapp"
MAIL_BODY_PATH="/rbluat/BACKEND/Finacle/FC10.2.9/app/CDCI_LOGS/"
}
echo $ID
echo $PASS
function getDatatrans
{
TRANID=`sqlplus -s $ID/$PASS#rbluat <<EOF
SELECT DISTINCT TRAN_ID,DTH_INIT_SOL_ID,TRAN_DATE,DEL_FLG FROM TBAADM.DTD WHERE PSTD_FLG='N' AND ENTRY_USER_ID='FIVUSR' and del_flg='N' and tran_date=(select db_stat_date from tbaadm.gct)AND REF_NUM IN (SELECT PYMT_REF_NUM FROM TBAADM.PORD WHERE STATUS IN ('A','H'));
exit;
EOF`
}
assignVariables
getDatatrans
echo $TRANID
I am getting output as :
[YOU HAVE NEW MAIL]
SELECT DISTINCT TRAN_ID,DTH_INIT_SOL_ID,TRAN_DATE,DEL_FLG FROM TBAADM.DTD WHERE PSTD_FLG='N' AND ENTRY_USER_ID='FIVUSR' and del_flg='N' and tran_date=(select db_stat_date from tbaadm.gct)AND REF_NUM IN (SELECT PYMT_REF_NUM FROM TBAADM.PORD WHERE STATUS IN ('A','H')) few.sh test.sh ERROR at line 1: ORA-00942: table or view does not exist
`
Here few.sh and test.sh are the file names present in the current working directory. few.sh is the file where I have written this code. I have no idea how these names are coming. I am working in KSH. I tried googling about it but found no clue.
The output of sqlplus in evaluated in the command
echo $TRANID
When TRANID has a * in it, ksh will show the files it can find.
You should use quotes to avoid evaluation:
echo "$TRANID"
When you are editing you code, you might as well add {} (not needed here, good habit):
echo "${TRANID}"
Likewise:
echo "${ID}"
echo "${PASS}"
...
-s "${ID}/${PASS}#rbluat"
Is it possible to run something like this in Hive CLI?
I am trying to pass file contents as a variable to another query.
set column_list=!cat /home/user/filename.lst ;
create table tabname as select $column_list from ...
if you have a query file you pass the variables as hiveconf
hive -hiveconf var1=abcd -f file.txt
or you can construct your query and then pass it to hive cli using -e
hive -e "create table ..."
file filename.lst
line
make a file test.sh,
temp=$(cat /home/user/filename.lst)
hive -f test.hql -hiveconf var=$temp
make a another file test.hql
create table test(${hiveconf:var} string);
on terminal
sh -x test.sh
It will pass the line to the test.hql and it will create a table with line as column;
note- all files should be in same directory .This script is passing only one variable.