How to replace SQL with bash variable of SQL command output - sql

I am working on a program in bash that checks for valid JSON files before loading them into a table. The process first runs f_check_valid_json to verify the JSON. This process runs f_exe_sql_stmnt() that returns a column of bad files, stored in variable bad_fl_list. I would like to be able to input bad_fl_list in the WHERE clause of my update and delete sections of the function.
Right now, the SQL fails when there is more than one JSON file ID in bad_fl_list
f_exe_sql_stmnt(){
db=$1
sql_str=$2
psql -d "$db" -Atc "$sql_str"
if [ $? -gt 0 ]
then
echo "======================================================================="
echo "***Error: Database error while executing sql statement($sql_str)..."
exit 123
fi
}
f_check_valid_json() {
echo "*** checking for valid JSON format***"
sql_stmnt="Select json_fl_id from json_stgng where is_valid_json(json_datarec_fl) = false;"
bad_fl_list=$(f_exe_sql_stmnt "$t_db" "${sql_stmnt}")
echo "BAD FILE LIST: ${bad_fl_list}"
if [ ! -z "$bad_fl_list" ]
then
echo "BAD JSON LIST IS NOT EMPTY"
echo "*** updating balancing table to reflect bad file ***"
updt_bal_log_str="UPDATE ${bal_log_tbl} SET trgt_load_stus_cd ='F' where json_fl_id in ($bad_fl_list);"
f_exe_sql_stmnt "$DB" "$updt_bal_log_str"
echo "*** deleting bad JSON file record from staging with file ID: ${bad_fl_list}"
delete_stmnt="delete from ${stg_tbl} where json_fl_id in ($bad_fl_list);"
f_exe_sql_stmnt "$t_db" "${delete_stmnt}"
fi
}
Here is some example output from the logs:
+ psql -d dedw -Atc 'UPDATE json_load_bal_dtl_log SET trgt_load_stus_cd ='\''F'\'' where json_fl_id in (O21181043417
O21181043417
O21181003641);'
ERROR: syntax error at or near "O21181043417"
LINE 2: O21181043417
^

Related

Run sql query from If else block in shell script

I want to connect sql db and execute sql query from shell script. I tried using this
if float_cmp "$size1 > 7.50"; then
echo "### THE DATA SIZE IS GREATER THAN 7.5 GB ###"
echo "############### DROPPING CREATED $IMPUSER USER ###########################"
${PATH_TO_CLIENT}sqlplus $EXPUSER/$EXPPWD#$ENDPOINT<< EOF
drop user $IMPUSER cascade;
exit;
EOF
exit 1
else
echo "### THE DATA SIZE IS OKAY ###"
fi
The statement is working outside if block but throwing error when executing in IF block
line 80: warning: here-document at line 72 delimited by end-of-file (wanted `EOF')
line 81: syntax error: unexpected end of file
Can anyone please tell what am I doing wrong and what is the solution?
Try removing the indentation inside of the here-document:
if float_cmp "$size1 > 7.50"; then
echo "### THE DATA SIZE IS GREATER THAN 7.5 GB ###"
echo "############### DROPPING CREATED $IMPUSER USER ###########################"
"${PATH_TO_CLIENT}"sqlplus $EXPUSER/$EXPPWD#$ENDPOINT<< EOF
drop user $IMPUSER cascade;
exit;
EOF
exit 1
else
echo "### THE DATA SIZE IS OKAY ###"
fi

File name is coming in SQL query output from unix

I have written a simple sh file to retrieve data from Oracle SQL but getting error. Following is my code:
. $HOME/.profile
function assignVariables
{
ID="finapp"
PASS="finapp"
MAIL_BODY_PATH="/rbluat/BACKEND/Finacle/FC10.2.9/app/CDCI_LOGS/"
}
echo $ID
echo $PASS
function getDatatrans
{
TRANID=`sqlplus -s $ID/$PASS#rbluat <<EOF
SELECT DISTINCT TRAN_ID,DTH_INIT_SOL_ID,TRAN_DATE,DEL_FLG FROM TBAADM.DTD WHERE PSTD_FLG='N' AND ENTRY_USER_ID='FIVUSR' and del_flg='N' and tran_date=(select db_stat_date from tbaadm.gct)AND REF_NUM IN (SELECT PYMT_REF_NUM FROM TBAADM.PORD WHERE STATUS IN ('A','H'));
exit;
EOF`
}
assignVariables
getDatatrans
echo $TRANID
I am getting output as :
[YOU HAVE NEW MAIL]
SELECT DISTINCT TRAN_ID,DTH_INIT_SOL_ID,TRAN_DATE,DEL_FLG FROM TBAADM.DTD WHERE PSTD_FLG='N' AND ENTRY_USER_ID='FIVUSR' and del_flg='N' and tran_date=(select db_stat_date from tbaadm.gct)AND REF_NUM IN (SELECT PYMT_REF_NUM FROM TBAADM.PORD WHERE STATUS IN ('A','H')) few.sh test.sh ERROR at line 1: ORA-00942: table or view does not exist
`
Here few.sh and test.sh are the file names present in the current working directory. few.sh is the file where I have written this code. I have no idea how these names are coming. I am working in KSH. I tried googling about it but found no clue.
The output of sqlplus in evaluated in the command
echo $TRANID
When TRANID has a * in it, ksh will show the files it can find.
You should use quotes to avoid evaluation:
echo "$TRANID"
When you are editing you code, you might as well add {} (not needed here, good habit):
echo "${TRANID}"
Likewise:
echo "${ID}"
echo "${PASS}"
...
-s "${ID}/${PASS}#rbluat"

Calling SQL*PLUS from UNIX Shell script and print the status message about the query

I want to run a SQL code using shell script and return the message whether the SQL query executed successfully or not. For this I have used unix script given below.
#!/bin/sh
sqlplus -S hr/hr#xe<<EOF
#emp.sql
EOF
var1=$(cat /cygdrive/d/scripts/output.txt | grep -c 'COUNT')
if [ $var1 -ge 1 ];
then
echo "success"
else
echo "failure"
fi
exit;
and emp.sql(called sql file) as
SET ECHO OFF
SPOOL D:\scripts\output.txt
SET LINESIZE 100
SET PAGESIZE 50
SELECT count(*) FROM employees;
SPOOL OFF;
EXIT 0;
When I execute the script I am getting output as
COUNT(*)
----------
107
./script1.sh: line 13: syntax error: unexpected end of file.
I don't know where I should put EOF statement exactly. Also I am not getting the status message whether it is success or failure which I want as output. Please help. Thanks in advance
SPOOL D:\scripts\output.txt Isnt this windows way of referring to a file where as in the shell script you referred to the file as /cygdrive/d/scripts/output.txt. I assume you are using linux shell to execute so I executed your script changing the spool line in sql file. It worked fine.
Edit: Also the \ that you used, for the spooled output.txt path, will cause the sqlplus to terminate. Hence the error line 13: syntax error: unexpected end of file. Perhaps add quotes to the path or use the same file path as you used in shell

Getting value from Oracle database table to UNIX variable

I am trying to create a simple script that connects to a oracle database, executes a select query and store the return value into a Unix variable. Below is the script I have created by following this post:
#!/bin/sh
VALUE=`sqlplus -silent $DB_USERNAME/"$PASSWORD"#"(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=$HOST_NAME)(PORT=$DB_PORT)))(CONNECT_DATA=(SID=$DB_SID)))" <<END
set pagesize 0 feedback off verify off heading off echo off
SELECT ID FROM TEST_USERS WHERE USER_NAME=$SAMPLE_USER;
exit;
END`
if [ -z "$VALUE" ]; then
echo "No rows returned from database"
exit 0
else
echo $VALUE
fi
Now when I run this script I am facing error as :
ERROR: ORA-12533: TNS:illegal ADDRESS parameters SP2-0306: Invalid
option. Usage: CONN[ECT] [{logon|/|proxy} [AS {SYSDBA|SYSOPER|SYSASM}]
[edition=value]] where ::=
[/][#] ::=
[][/][#] SP2-0306:
Invalid option. Usage: CONN[ECT] [{logon|/|proxy} [AS
{SYSDBA|SYSOPER|SYSASM}] [edition=value]] where ::=
[/][#] ::=
[][/][#] SP2-0157:
unable to CONNECT to ORACLE after 3 attempts, exiting SQL*Plus
Please let me know where I am doing mistake?
Try with this VALUE variable:
VALUE=`sqlplus $DB_USERNAME/$PASSWORD#//$HOST_NAME:$DB_PORT/$DB_SID`
And, of course, you need to define all variables, you use there before this line

Error executing shell command in pig script

I have a pig script where in the beginning I would like to generate a string of the dates of the past 7 days from a certain date (later used to retrieve log files for those days).
I attempt to do this with this line:
%declare CMD7 input= ; for i in {1..6}; do d=$(date -d "$DATE -i days" "+%Y-%m-%d"); input="\$input\$d,"; done; echo \$input
I get an error :
" ERROR 2999: Unexpected internal error. Error executing shell command: input= ; for i in {1..6}; do d=$(date -d "2012-07-10 -i days" "+%Y-%m-%d"); input="$input$d,"; done;. Command exit with exit code of 127"
however the shell command runs perfectly fine outside of pig. I am really not sure what is going wrong here.
Thank you!
I have got a working solution but not as streamlined as you want, essentially I don't manage to get Pig to execute a complex shell statement in the declare.
I first wrote a shell script (let's call it 6-days-back-from.sh):
#!/bin/bash
DATE=$1
for i in {1..6}; do d=$( date -d "$DATE -$i days" +%F ) ; echo -n "$d "; done
Then a pig script as follow (let's call it days.pig):
%declare my_date `./6-days-back-from.sh $DATE`
A = LOAD 'dual' USING PigStorage();
B = FOREACH A GENERATE '$my_date';
DUMP B
note that dual is a directory containing a text file with a single line of text, for the purpose of displaying our variable
I called the script as follow:
pig -x local -param DATE="2012-08-03" days.pig
and got the following output:
({(2012-08-02),(2012-08-01),(2012-07-31),(2012-07-30),(2012-07-29),(2012-07-28)})