I have some insert statements to run in Hive. I am planning to put them in .hql file and run it through beeline -f option. Is there a way I can echo some log messages in between inserts so that I know the progress. Like :
echo "Starting the inserts ........."
insert1
echo "Insert 1 complete"
insert2
echo "Insert script is complete"
I tried putting echo statements by using linux shell command echo as
!echo ""
But it's not recognizing echo as a command
!sh echo ...
!sh echo "Starting the inserts ........."
insert ...
!sh echo "Insert 1 complete"
insert ...
!sh echo "Insert script is complete"
set msg = "Starting Insert";
set msg;
insert into .... ;
set msg = "Insert complete";
set msg;
Related
I am trying to create a bash script that will generate an SQL CREATE TABLE statement from a CSV file.
#!/bin/bash
# Check if the user provided a CSV file
if [ $# -eq 0 ]
then
echo "No CSV file provided."
exit 1
fi
# Check if the CSV file exists
if [ ! -f $1 ]
then
echo "CSV file does not exist."
exit 1
fi
# Get the table name from the CSV file name
table_name=$(basename $1 .csv)
# Extract the header row from the CSV file
header=$(head -n 1 $1)
# Split the header row into column names
IFS=',' read -r -a columns <<< "$header"
# Generate the PostgreSQL `CREATE TABLE` statement
echo "CREATE TABLE $table_name ("
for column in "${columns[#]}"
do
echo " $column TEXT,"
done
echo ");"
If I have a CSV file with three columns(aa,bb,cc), the generated statement does not have the last column for some reason.
Any idea what could be wrong?
If I do:
for a in "${array[#]}"
do
echo "$a"
done
I am getting:
aaa
bbb
ccc
But when add something into the string:
for a in "${array[#]}"
do
echo "$a SOMETHING"
done
I get:
aaa SOMETHING
bbb SOMETHING
SOMETHING
Thanks.
Your csv file has a '\r`
Try the next block for reproducing the problem.
printf -v header "%s,%s,%s\r\n" "aaa" "bbb" "ccc"
IFS=',' read -r -a columns <<< "$header"
echo "Show array"
for a in "${columns[#]}"; do echo "$a"; done
echo "Now with something extra"
for a in "${columns[#]}"; do echo "$a SOMETHING"; done
You should remove the '\r', what can be done with
IFS=',' read -r -a columns < <(tr -d '\r' <<< "${header}")
I have tried to assign as SQL query value to a variable so that based on its value I will trigger my ETL job in the shell script.
I tried something like below.
echo off
status='sqlcmd -S {host} -d {db} -U {user} -P {password} -Q "SELECT CASE WHEN fc.config_val< fc.config_val_dev_mysql AND fc.config_val < fc.config_val_prod_mysql THEN 1 ELSE 0 END AS mssql_status FROM flex_configs fc;"';
echo $status
But this is not working.
If the status = 1 then I will trigger the ETL job in the same shell script if not I do nothing. Please suggest me the answer as I am unable to get this script.
status=$(sqlcmd -S {host} -d {db} -U {user} -P {password} -Q "SELECT CASE WHEN fc.config_val< fc.config_val_dev_mysql AND fc.config_val < fc.config_val_prod_mysql THEN 1 ELSE 0 END AS mssql_status FROM flex_configs fc;")
echo $status
You need to put code inside $() like $(my code) or `my code`
I am trying to run select statement within case statement in Unix Shell script, but getting unexpected end of file error.
I want to run particular select statement depending on the output of previous sql script ran in shell script. The output from previous sql script is spooled to a log, required pattern is fetched into a variable, which is used in case statement.
My script
#!/usr/bin/sh
exec > check_company_details.log 2>&1
sqlplus username/password#database << EOF
#check_company_details.sql $1
exit;
EOF
pool=$(cat company.log | grep dbPool | awk {'print $5'})
#everything is working till above steps
#if sqlplus command is removed from below case statements, correct output of echo is returned.
case $pool in
dbpool1)
echo "DBPool is POOL1"
sqlplus username/password#database<<EOF
select name from v\$database;
exit;
EOF
;;
dbpool2)
echo "DBPool is POOL1"
sqlplus username/password#database<<EOF
select name from v\$database;
exit;
EOF
;;
dbpool3)
echo "DBPool is DC4POOL1"
sqlplus username/password#database<<EOF
select name from v\$database;
exit;
EOF
;;
*)
echo No Results
;;
esac
Error message:
*./check_company_details.sh: line 37: syntax error: unexpected end of file*
A here doc end string should not have leading whitespace. This means you should rewrite
dbpool3)
echo "DBPool is DC4POOL1"
sqlplus username/password#database<<EOF
select name from v\$database;
exit;
EOF
as
dbpool3)
echo "DBPool is DC4POOL1"
sqlplus username/password#database<<EOF
select name from v\$database;
exit;
EOF
and the same goes for the other cases.
You should also say fgrep dbPool company.log instead of needlessly using cat and instead of using grep when you are not feeding in a regex. You also have the quotes around your awk script in a weird place; it works but it's not what it should be.
pool=$(cat company.log | grep dbPool | awk {'print $5'})
becomes
pool=$(fgrep dbPool company.log | awk '{print $5}')
You should not expand $pool without quoting it, e.g. it should be case "$pool" in. Even if you think it won't have spaces in the variable you should do this for safety.
You should get in to the habit of checking all of your shell scripts with shellcheck whether they work or not.
I think you don't require a case block. You could use an if else statement with pool variable.
if [ "$pool" = "dbpool1" ] || [ "$pool" = "dbpool2" ] || [ "$pool" = "dbpool3" ]
then
echo "DBPool is ${pool}"
sqlplus username/password#database<<EOF
select name from v\$database;
exit
EOF
else
echo "No Results"
fi
I have 2 shell scripts. One with data connection like that (script1.sh):
#!/usr/bin/ksh
query="$1"
whenever sqlerror exit 3
connect $user/$pass#sid
${query}
EOF
echo $?
if [ 0 -ne "$?" ]; then
exit 1
fi
and other is a shell script bigger where I execute sql commands like these:
#!/usr/bin/ksh
set -x
$PATH/script1.sh "
--set serveroutput on
--set feedback off
insert into table (column) values ('$1');
commit;
"
if [[ $? != 0 ]]
then
echo "Error"
exit 3
else
echo "Ok"
fi
............
..............
The problem is that these second script won't detect error in sql commands and always continues with all code. I put traces and I check that rc is always 0.
Could you help me to can detect errors if the sql failed? Thanks!
How can I send email based on a hive query output. Say I have a table where I want to check if the number is between two other numbers of a different table. I can check that in a sql query and return sql output as 0 or 1.
Now the question is how can I send email using mailx or equivalent from the same script based on that sql output.
$ var=hive -S -e "select '0' from test;"
$ echo $var
0
$ var=hive -S -e "select '1' from test;"
$ echo $var
1
Option : Use a shell action in oozie to run a shell script which will execute the hive command inline and capture the output as 0/1 in a variable. Use the variable in the shell to call mailx.
Workflow is your choice to run, you can use oozie or third-party tools or the famous cron job. You can leverage the below shell script for sending the emails based on the output of the Beeline/hive.
#!bin/bash
#Output variable from hive ql
Output=beeline -u ${hiveConnectionSTRING} --silent=true -e "your query that pulls the output as 0 or 1"
#Condition check and sending the email with mailx utility
if [ $Output -gt 0 ];
then
echo "output is zero"
#Email need to be added
#Username=From address(This is the name before your domain EX: Stack#domain name)
echo -e 'your email message should be here \n\n\n\nThank you,' | mailx -r $UserName -s 'Your Subject' -c stackoverflow#gmail.com(your email cc address) -- stackoverflow#gmail.com(your email to address)0
else
echo -e 'your email message should be here \n\n\n\nThank you,' | mailx -r $UserName -s 'Your Subject' -c stackoverflow#gmail.com(your email cc address) -- stackoverflow#gmail.com(your email to address)0
echo "output is 1"
fi