I have tried to assign as SQL query value to a variable so that based on its value I will trigger my ETL job in the shell script.
I tried something like below.
echo off
status='sqlcmd -S {host} -d {db} -U {user} -P {password} -Q "SELECT CASE WHEN fc.config_val< fc.config_val_dev_mysql AND fc.config_val < fc.config_val_prod_mysql THEN 1 ELSE 0 END AS mssql_status FROM flex_configs fc;"';
echo $status
But this is not working.
If the status = 1 then I will trigger the ETL job in the same shell script if not I do nothing. Please suggest me the answer as I am unable to get this script.
status=$(sqlcmd -S {host} -d {db} -U {user} -P {password} -Q "SELECT CASE WHEN fc.config_val< fc.config_val_dev_mysql AND fc.config_val < fc.config_val_prod_mysql THEN 1 ELSE 0 END AS mssql_status FROM flex_configs fc;")
echo $status
You need to put code inside $() like $(my code) or `my code`
Related
I'm trying to create a script that removes my images that are not in DB
There is my code (Updated):
I have 1 problems:
Problem with the like syntax like '%$f%'
#!/bin/bash
db="intranet_carc_development"
user="benjamin"
for f in public/uploads/files/*
do
if [[ -f "$f" ]]
then
psql $db $user -t -v "ON_ERROR_STOP=1" \
-c 'select * from public.articles where content like "%'"$(basename "$f")"'%"' | grep . \
&& echo "exist" \
|| echo "doesn't exist"
fi
done
And I have the following error :
ERROR: column "%1YOLV3M4-VFb2Hydb0VFMw.png%" does not exist
LINE 1: select * from public.articles where content like "%1YOLV3M4-...
^
doesn't exist
ERROR: column "%wnj8EEd8wuJp4TdUwqrJtA.png%" does not exist
LINE 1: select * from public.articles where content like "%wnj8EEd8w...
EDIT : if i use \'%$f%\' for the like :
/purge_files.sh: line 12: unexpected EOF while looking for matching `"'
./purge_files.sh: line 16: syntax error: unexpected end of file
There are several issues with your code :
$f is public/uploads/files/FILENAME and i want only the FILENAME
You can use basename to circumvent that, by writing :
f="$(basename "$f")"
psql $db $user -c "select * from public.articles where content like '%$f%'"...
(The extra quotes are here to prevent issues if you have spaces and special characters in your file name)
your psql request will always return true even if no rows are found
your psql command will return true even if the request fails, unless you set the variable 'ON_ERROR_STOP' to 1
As shown in the linked questions, you can use the following syntax :
#!/bin/bash
set -o pipefail #needed because of the pipe to grep later on
db="intranet_carc_development"
user="benjamin"
for f in public/uploads/files/*
do
if [[ -f "$f" ]]
then
f="$(basename "$f")"
psql $db $user -t -v "ON_ERROR_STOP=1" \
-c "select * from public.articles where content like '%$f%'" | grep . \
&& echo "exist" \
|| echo "doesn't exist"
fi
done
How can I send email based on a hive query output. Say I have a table where I want to check if the number is between two other numbers of a different table. I can check that in a sql query and return sql output as 0 or 1.
Now the question is how can I send email using mailx or equivalent from the same script based on that sql output.
$ var=hive -S -e "select '0' from test;"
$ echo $var
0
$ var=hive -S -e "select '1' from test;"
$ echo $var
1
Option : Use a shell action in oozie to run a shell script which will execute the hive command inline and capture the output as 0/1 in a variable. Use the variable in the shell to call mailx.
Workflow is your choice to run, you can use oozie or third-party tools or the famous cron job. You can leverage the below shell script for sending the emails based on the output of the Beeline/hive.
#!bin/bash
#Output variable from hive ql
Output=beeline -u ${hiveConnectionSTRING} --silent=true -e "your query that pulls the output as 0 or 1"
#Condition check and sending the email with mailx utility
if [ $Output -gt 0 ];
then
echo "output is zero"
#Email need to be added
#Username=From address(This is the name before your domain EX: Stack#domain name)
echo -e 'your email message should be here \n\n\n\nThank you,' | mailx -r $UserName -s 'Your Subject' -c stackoverflow#gmail.com(your email cc address) -- stackoverflow#gmail.com(your email to address)0
else
echo -e 'your email message should be here \n\n\n\nThank you,' | mailx -r $UserName -s 'Your Subject' -c stackoverflow#gmail.com(your email cc address) -- stackoverflow#gmail.com(your email to address)0
echo "output is 1"
fi
I want to export all data from sql server table to a csv, I know I can get the desired result by:
sqlcmd -S . -d database -E -s, -W -Q "SELECT * FROM TABLENAME" > file.csv
I have many tables, so I want to create a .bat file that do the work for me, I have this:
set "list = A B C D"
for %%x in (%list%) do (
sqlcmd -S . -d database -E -s, -W -Q "SELECT * FROM %%x" > %%x.csv
)
But I am getting errors I don't know (I am not an expert in bat files). Why this does not work? How can I do what I want?
Spacing is important when using set (unless you're doing math with the /A switch). As written, the variable you're setting isn't %list%. It's %list %. Change your set command as follows:
set "list=A B C D"
I am using below command line to run a SQL query using SQLCMD
sqlcmd -S Server -Q "select top 100 * From people" -d people -t 10
The table has 20 columns and when i look at output command line window wraps the text and makes it difficult to read.
I want my results to be displayed the same way it displays in SQL Server Management Studio (properly formatted). I am not looking for any grids, but i need all my columns to be displayed in row 1 and the results properly beneath.
Thanks in advance.
Answer
We can set the width of each column.
C:/> sqlcmd -S my_server
> :setvar SQLCMDMAXVARTYPEWIDTH 30
> :setvar SQLCMDMAXFIXEDTYPEWIDTH 30
> SELECT * from my_table
> go
We can also set it like this: sqlcmd -S my_server -y 30 -Y 30.
Details
SQLCMDMAXVARTYPEWIDTH (-y)
It limits the number of characters that are returned for the large variable length data type
SQLCMDMAXFIXEDTYPEWIDTH (-Y)
Limits the number of characters that are returned for the following data types
Note: setting -y has serious performance implications.
See https://learn.microsoft.com/en-us/sql/tools/sqlcmd-utility
Formatting issues usually pop up due to your console window.
One solution is to output to the file and use notepad/your favorite editor:
sqlcmd -S myServer -d myDB -E -Q "select top 100 * From people"
-o "output.txt"
This is how I isolated a scalar.
sqlcmd -S xxx.xxx.xxx.xxx,xxxxx -d MyDb -U myUser -P MyPassword -h -1 -W -Q "set NOCOUNT ON; select a from b where b.id='c'"
Updated :
Initial issue :
Having a while read loop printing every line that is read
Answer : Put a done <<< "$var"
Subsequent issue :
I may need some explanations about some SHELL code :
I have this :
temp_ip=$($mysql --skip-column-names -h $db_address -u $db_user -p$db_passwd $db_name -e "select ip_routeur,code_site from $db_vtiger_table where $db_vtiger_table.ip_routeur NOT IN (select ip from $db_erreur_table);")
That gets results looking like this :
<ip1> <site1>
<ip2> <site2>
<ip3> <site3>
<ip4> <site4>
up to 5000 ip_address
I did a "while loop" :
while [ `find $proc_dir -name snmpproc* | wc -l` -ge "$max_proc_snmpget" ];do
{
echo "sleeping, fping in progress";
sleep 1;
}
done
temp_ip=$($mysql --skip-column-names -h $db_address -u $db_user -p$db_passwd $db_name -e "select ip_routeur,code_site from $db_vtiger_table where $db_vtiger_table.ip_routeur NOT IN (select ip from $db_erreur_table);")
while read ip codesite;do
{
sendSNMPGET $ip $snmp_community $code_site &
}
done<<<"$temp_ip"
And the sendSNMPGET function is :
sendSNMPGET() {
touch $procdir/snmpproc.$$
hostname=`snmpget -v1 -c $2 $1 sysName.0`
if [ "$hostname" != "" ]
then
echo "hi test"
fi
rm -f $procdir/snmpproc.$$
The $max_proc_snmpget is set to 30
At the execution, the read is ok, no more printing on screen, but child processes seems to be disoriented
hi
hi
hi
hi
hi
hi
hi
hi
hi
hi
hi
hi
./scan-snmp.sh: fork: Resource temporarily unavailable
./scan-snmp.sh: fork: Resource temporarily unavailable
./scan-snmp.sh: fork: Resource temporarily unavailable
./scan-snmp.sh: fork: Resource temporarily unavailable
Why can't it handle this ?
If temp_ip contains the name of a file that you want to read, then use:
done<"$temp_ip"
In your case, it appears that temp_ip is not a file name but contains the actual data that you want. In that case, use:
done<<<"$temp_ip"
Take care that the variable is placed inside double-quotes. That protects the data against the shell's word splitting which would result in the replacement of new line characters with spaces.
More details
In bash, an expression like <"$temp_ip" is called redirection. In this case in means that the while loop will get its standard input from the file called $temp_ip.
The expression <<<"$temp_ip" is called a here string. In this case, it means that the while loop will get its standard input from the data in the variable $temp_ip.
More information on both redirection and here strings in man bash.
Or you can parse the output of your initial command directly:
$mysql --skip-column-names -h $db_address -u $db_user -p$db_passwd $db_name -e "select ip_routeur,code_site from $db_vtiger_table where $db_vtiger_table.ip_routeur NOT IN (select ip from $db_erreur_table) | \
while read ip codesite
do
...
done
If you want to improve the performance and run some of the 5,000 SNMPGETs in parallel, I would recommend using GNU Parallel (here) like this:
$mysql --skip-column-names -h $db_address -u $db_user -p$db_passwd $db_name -e "select ip_routeur,code_site from $db_vtiger_table where $db_vtiger_table.ip_routeur NOT IN (select ip from $db_erreur_table) | parallel -k -j 20 -N 2 sendSNMPGET {1} $snmp_community {2}
The -k will keep the parallel output in order. The -j 20 will run up to 20 SNMPGETs in parallel at a time. The -N 2 means take 2 parameters from the mysql output per job (i.e. ip and codesite). {1} and {2} are your ip and codesite parameters.
http://www.gnu.org/software/parallel/
I propose to not store the result value but use it directly:
while read ip codesite
do
sendSNMPGET "$ip" "$snmp_community" "$code_site" &
done < <(
"$mysql" --skip-column-names -h "$db_address" -u "$db_user" -p"$db_passwd" "$db_name" \
-e "select ip_routeur,code_site from $db_vtiger_table where $db_vtiger_table.ip_routeur NOT IN (select ip from $db_erreur_table);")
This way you start the mysql command in a subshell and use its output as input to the while loop (similar to piping which here also is an option).
But I see some problems with that code: If you really start each sendSNMPGET command in the background, you very quickly will put a massive load on your computer. For each line you read another active background process is started. This can slow down your machine to the point where it is rendered useless.
I propose to not run more than 20 background processes at a time.
As you don't seem to have liked my answer with GNU Parallel, I'll show you a very simplistic way of doing it in parallel without needing to install that...
#!/bin/bash
MAX=8
j=0
while read ip code
do
(sleep 5; echo $ip $code) & # Replace this with your SNMPGET
((j++))
if [ $j -eq $MAX ]; then
echo -n Pausing with $MAX processes...
j=0
wait
fi
done < file
wait
This starts up to 8 processes (you can change it) and then waits for them to complete before starting another 8. You have already been shown how to feed your mysql stuff into the loop by other respondents in the second to last line of the script...
The key to this is the wait which will wait for all started processes to complete.