Within a bash script I am running a sql query via 'psql -c '. Based off of the arguements given to the bash script, the where claus of the select command will be different. So basically I need to know if its possible to do something like this:
psql -c "select statement here until we get to the where clause at which point we break out of statement and do"
if (arg1 was given)
concatenate "where arg1" to the end of the above select statement
if (arg2 was given)
concatenate "where arg2" to the end of the above select statement
and so on for as many arguments. I know I could do this much easier in a sql function if I just passed the arguments but that really isnt an option. Thanks!
Edit: 5 seconds after posting this I realize I could just create a string before calling the psql command and then call the psql command on that. Doh!
psql -c "SELECT columns FROM table ${1:+WHERE $1} ${2:+WHERE $2}"
This uses the "use alternate value" substitution - ${VAR:+alternate} - where alternate is substituted if $VAR is set and not empty. If $VAR is empty, nothing will be substituted.
Save ascript, e.g. query.sh:
#!/bin/bash
query="select statement here until we get to the where clause at which point we break out of statement and do"
if [ $# -gt 0 ]
then
query+=" where $1"
shift
fi
while [ $# -gt 0 ]
then
query+=" and $1"
shift
fi
psql -c "$query"
Call it like
chmod +x ./query.sh
./query.sh "id in (1,2,3)" "modified_by='myname'"
SQL="select x,y,z FROM foobar"
if [ "$1" != "" ]
then
SQL="$SQL where $1"
fi
psql "$SQL"
stmt="select statement here until we get to the where clause at which point we break out of statement and do"
if (( $# > 0 ))
then
stmt="$stmt where $1"
fi
if (( $# > 1 ))
then
stmt="$stmt where $2"
fi
psql -c "$stmt"
Related
I am trying to store SQL query result into an array in a shell script but I face an invalid identifier error when I run my .sh
Could you please check what matter in my code?
#!/usr/bin/ksh
echo Start Executing SQL commands
array=$(sqlplus -s apps/apps << eof
SET PAGESIZE 0;
SELECT directory_name from all_directories where directory_name like '%XXBP%';
eof)
printf '%s\n' "${array[#]}"
This is the error I get:
I know the problem comes from my operator % but I need it to restrict the result of my query.
This sounds like this very specific ksh bug where here-documents silently convert single-quotes to double-quotes. You could try the workaround in that answer, e.g.
#!/usr/bin/ksh
echo Start Executing SQL commands
# put the single-quotes in a variable to prevent the here-document from converting them to double
STR="'%XXBP%'"
array=$(sqlplus -s apps/apps << eof
SET PAGESIZE 0;
SELECT directory_name from all_directories where directory_name like $STR;
eof)
printf '%s\n' "${array[#]}"
I'm trying to create a script that removes my images that are not in DB
There is my code (Updated):
I have 1 problems:
Problem with the like syntax like '%$f%'
#!/bin/bash
db="intranet_carc_development"
user="benjamin"
for f in public/uploads/files/*
do
if [[ -f "$f" ]]
then
psql $db $user -t -v "ON_ERROR_STOP=1" \
-c 'select * from public.articles where content like "%'"$(basename "$f")"'%"' | grep . \
&& echo "exist" \
|| echo "doesn't exist"
fi
done
And I have the following error :
ERROR: column "%1YOLV3M4-VFb2Hydb0VFMw.png%" does not exist
LINE 1: select * from public.articles where content like "%1YOLV3M4-...
^
doesn't exist
ERROR: column "%wnj8EEd8wuJp4TdUwqrJtA.png%" does not exist
LINE 1: select * from public.articles where content like "%wnj8EEd8w...
EDIT : if i use \'%$f%\' for the like :
/purge_files.sh: line 12: unexpected EOF while looking for matching `"'
./purge_files.sh: line 16: syntax error: unexpected end of file
There are several issues with your code :
$f is public/uploads/files/FILENAME and i want only the FILENAME
You can use basename to circumvent that, by writing :
f="$(basename "$f")"
psql $db $user -c "select * from public.articles where content like '%$f%'"...
(The extra quotes are here to prevent issues if you have spaces and special characters in your file name)
your psql request will always return true even if no rows are found
your psql command will return true even if the request fails, unless you set the variable 'ON_ERROR_STOP' to 1
As shown in the linked questions, you can use the following syntax :
#!/bin/bash
set -o pipefail #needed because of the pipe to grep later on
db="intranet_carc_development"
user="benjamin"
for f in public/uploads/files/*
do
if [[ -f "$f" ]]
then
f="$(basename "$f")"
psql $db $user -t -v "ON_ERROR_STOP=1" \
-c "select * from public.articles where content like '%$f%'" | grep . \
&& echo "exist" \
|| echo "doesn't exist"
fi
done
I am trying to run select statement within case statement in Unix Shell script, but getting unexpected end of file error.
I want to run particular select statement depending on the output of previous sql script ran in shell script. The output from previous sql script is spooled to a log, required pattern is fetched into a variable, which is used in case statement.
My script
#!/usr/bin/sh
exec > check_company_details.log 2>&1
sqlplus username/password#database << EOF
#check_company_details.sql $1
exit;
EOF
pool=$(cat company.log | grep dbPool | awk {'print $5'})
#everything is working till above steps
#if sqlplus command is removed from below case statements, correct output of echo is returned.
case $pool in
dbpool1)
echo "DBPool is POOL1"
sqlplus username/password#database<<EOF
select name from v\$database;
exit;
EOF
;;
dbpool2)
echo "DBPool is POOL1"
sqlplus username/password#database<<EOF
select name from v\$database;
exit;
EOF
;;
dbpool3)
echo "DBPool is DC4POOL1"
sqlplus username/password#database<<EOF
select name from v\$database;
exit;
EOF
;;
*)
echo No Results
;;
esac
Error message:
*./check_company_details.sh: line 37: syntax error: unexpected end of file*
A here doc end string should not have leading whitespace. This means you should rewrite
dbpool3)
echo "DBPool is DC4POOL1"
sqlplus username/password#database<<EOF
select name from v\$database;
exit;
EOF
as
dbpool3)
echo "DBPool is DC4POOL1"
sqlplus username/password#database<<EOF
select name from v\$database;
exit;
EOF
and the same goes for the other cases.
You should also say fgrep dbPool company.log instead of needlessly using cat and instead of using grep when you are not feeding in a regex. You also have the quotes around your awk script in a weird place; it works but it's not what it should be.
pool=$(cat company.log | grep dbPool | awk {'print $5'})
becomes
pool=$(fgrep dbPool company.log | awk '{print $5}')
You should not expand $pool without quoting it, e.g. it should be case "$pool" in. Even if you think it won't have spaces in the variable you should do this for safety.
You should get in to the habit of checking all of your shell scripts with shellcheck whether they work or not.
I think you don't require a case block. You could use an if else statement with pool variable.
if [ "$pool" = "dbpool1" ] || [ "$pool" = "dbpool2" ] || [ "$pool" = "dbpool3" ]
then
echo "DBPool is ${pool}"
sqlplus username/password#database<<EOF
select name from v\$database;
exit
EOF
else
echo "No Results"
fi
I've the following file queries.sql that contains a number of queries, structured like this:
/* Query 1 */
SELECT cab_type_id,
Count(*)
FROM trips
GROUP BY 1;
/* Query 2 */
SELECT passenger_count,
Avg(total_amount)
FROM trips
GROUP BY 1;
/* Query 3 */
SELECT passenger_count,
Extract(year FROM pickup_datetime),
Count(*)
FROM trips
GROUP BY 1,
2;
Then I've written a regex, that finds all those queries in the file:
/\*[^\*]*\*/[^;]*;
What I'd like to achieve is the following:
Select all the queries with the regex.
Prefix each query with EXPLAIN ANALYZE
Execute each query and output the results to a new file. That means, query 1 will create a file q1.txt with the corresponding output, query 2 create q2.txt etc.
One of my main challenges (there are no problems, right? ;-)) is, that I'm rather unfamiliar with the linux bash I've to use.
I tried cat queries.sql | grep '/\*[^\*]*\*/[^;]*;' but that doesn't return anything.
So a solution could look like:
count = 0
for query in (cat queries.sql | grep 'somehow-here-comes-my-regex') do
count = $count+1
query = 'EXPLAIN ANALYZE '+query
psql -U postgres -h localhost -d nyc-taxi-data -c query > 'q'$count'.txt'
Except from: that doesn't work and I don't know how to make it work.
You have to omit spaces for variable assignments.
The following script would help. Save it in a file eg.: explain.sh, make it executable using chmod 0700 explain.sh and run in the following way: ./explain.sh query.sql.
#!/bin/bash
qfile="$1"
# number of queries
n="$(grep -oP '(?<=Query )[0-9]+ ' $qfile)"
count=1
for q in $n; do
# Corrected solution, modified after the remarks of #EdMorton
qn="EXPLAIN ANALYZE $(awk -v n="Query $q" 'flag; $0 ~ n {flag=1} /;/{flag=0}' $qfile)"
#qn="EXPLAIN ANALYZE $(awk -v n=$q "flag; /Query $q/{flag=1} /;/{flag=0}" $qfile)"
# psql -U postgres -h localhost -d nyc-taxi-data -c "$qn" > q$count.txt
echo "$qn" > q$count.txt
count=$(( $count + 1 ))
done
First of all, the script accounts for one argument (your example input query.sql file). It reads out the number of queries and save into a variable n. Then in a for loop it iterates through the query numbers and uses awk to extract the number n query and append EXPLAIN ANALYZE to the beginning. Then you can run your psql with the desired query. Here I commented out the psql part. This example script only creates qN.txt files for each explain query.
UPDATE:
The awk part: It is possible to use a shell variable in awk using the -v flag. Here we creates an awk variable n with the value of the q shell variable. n is used to create the starter pattern ie: Query 1. awk -v n="Query $q" 'flag; $0 ~ n {flag=1} /;/{flag=0}' $qfile matches everything between Query 1 and the first occurence of a semi-colon (;) excluding the line of Query 1 from query.sql. The $(...) means command-substitution in bash, thus we can save the output of a shell command into a variable. Here we save the output of awk and prefix it with the EXPLAIN ANALYZE string.
Here is a great answer about awk pattern matching.
It sounds like this is what you're looking for:
awk -v RS= -v ORS='\0' '{print "EXPLAIN ANALYZE", $0}' queries.sql |
while IFS= read -r -d '' query; do
psql -U postgres -h localhost -d nyc-taxi-data -c "$query" > "q$((++count)).txt"
do
The awk statement outputs each query as a NUL-terminated string, the shell loop reads it as such one at a time and calls psql on it. Simple, robust, efficient, etc...
How do you hide the column names and row count in the output from psql?
I'm running a SQL query via psql with:
psql --user=myuser -d mydb --output=result.txt -c "SELECT * FROM mytable;"
and I'm expecting output like:
1,abc
2,def
3,xyz
but instead I get:
id,text
-------
1,abc
2,def
3,xyz
(3 rows)
Of course, it's not impossible to filter the top two rows and bottom row out after the fact, but it there a way to do it with only psql? Reading over its manpage, I see options for controlling the field delimiter, but nothing for hiding extraneous output.
You can use the -t or --tuples-only option:
psql --user=myuser -d mydb --output=result.txt -t -c "SELECT * FROM mytable;"
Edited (more than a year later) to add:
You also might want to check out the COPY command. I no longer have any PostgreSQL instances handy to test with, but I think you can write something along these lines:
psql --user=myuser -d mydb -c "COPY mytable TO 'result.txt' DELIMITER ','"
(except that result.txt will need to be an absolute path). The COPY command also supports a more-intelligent CSV format; see its documentation.
You can also redirect output from within psql and use the same option. Use \o to set the output file, and \t to output tuples only (or \pset to turn off just the rowcount "footer").
\o /home/flynn/queryout.txt
\t on
SELECT * FROM a_table;
\t off
\o
Alternatively,
\o /home/flynn/queryout.txt
\pset footer off
. . .
usually when you want to parse the psql generated output you would want to set the -A and -F ...
# generate t.col1, t.col2, t.col3 ...
while read -r c; do test -z "$c" || echo , $table_name.$c | \
perl -ne 's/\n//gm;print' ; \
done < <(cat << EOF | PGPASSWORD=${postgres_db_useradmin_pw:-} \
psql -A -F -v -q -t -X -w -U \
${postgres_db_useradmin:-} --port $postgres_db_port --host $postgres_db_host -d \
$postgres_db_name -v table_name=${table_name:-}
SELECT column_name
FROM information_schema.columns
WHERE 1=1
AND table_schema = 'public'
AND table_name =:'table_name' ;
EOF
)
echo -e "\n\n"
You could find example of the full bash call here: