I am new to programming world. I have a SQL script which needs to be automated. The automation required is as follows :
1) Script should run every sunday
2) Automatically dump the results in to DUMP_YYYYMMDDHH24MISS.txt
3) Result set is tar gziped
4) upload to SFTP URL with provided username and password.
I am using :
UNIX,
Vertica DB
Can the Gurus here please help ?
This is really 4 questions and should probably asked as such. To answer in the current format though:
1) Schedule a Task Automatically - Crontab
In the terminal, type crontab -e.
If you want something every Sunday at 1am, add the following line:
0 1 * * * 0 /path/to/script/script.sh
This will execute the script every Sunday.
2) Setting the output of the command
I'm only familiar with oracle. The format is probably similar. In order to get the filename as you want it, you'd use the date function as follows. (This is how I would do it in with Oracle):
d=$(date +%Y%M%D%H%M)
var=$(sqlplus -s / as blahblahblah
select * from stuff;
exit
EOF
)
file_name=DUMP_${d}MISS.txt
echo "${var}" >> ${file_name}
Note that your date command is probably different, if you do a man page on date it will tell you which parameters you'd need to get the date formatted as you like.
3) Taring the output
tar -xvf ${file_name}
4) Send over SFTP
You'd have to authenticate the sftp, that is beyond the scope of what anyone can answer without more details. Once you have the machines setup to authenticate, you would do:
sftp username#server<<EOF
put ${file_name}
EOF
Related
I am running PS 4.0 and the following command in interaction with a Veritas Netbackup master server on a Unix host via plink:
PS C:\batch> $testtest = c:\batch\plink blah#blersniggity -pw "blurble" "/usr/openv/netbackup/bin/admincmd/nbpemreq -due -date 01/17/2014" | Format-Table -property Status
As you can see, I attempted a "Format-Table" call at the end of this.
The resulting value of the variable ($testtest) is a string that is laid out exactly like the table in the Unix console, with Status, Job Code, Servername, Policy... all that listed in order. But, it populates the variable in Powershell as just that: a vanilla string.
I want to use this in conjunction with a stored procedure on a SQL box, which would be TONS easier if I could format it into a table. How do I use Powershell to tabulate it exactly how it is extracted from the Unix prompt via Plink?
You'll need to parse it and create PS Objects to be able to use the format-* cmdlets. I do enough of it that I wrote this to help:
http://gallery.technet.microsoft.com/scriptcenter/New-PSObjectFromMatches-87d8ce87
You'll need to be able to isolate the data and write a regex to capture the bits you want.
Give this unix script, which is scheduled batch run:
isql -U$USR -S$SRVR -P$PWD -w2000 < $SCRIPTS/sample_report.sql > $TEMP_DIR/sample_report.tmp_1
sed 's/-\{3,\}//g' $TEMP_DIR/sample_report.tmp_1 > $TEMP_DIR/sample_report.htm_1
uuencode $TEMP_DIR/sample_report.htm_1 sample_report.xls > $TEMP_DIR/sample_report.mail_1
mailx -s "Daily Sample Report" email#example.com < $TEMP_DIR/sample_report.mail_1
There are occasionally cases where the sample_report.xls attached in the mail, is empty, zero lines.
I have ruled out the following:
not command processing timeout - by adding the -t30 to isql, I get the xls and it contains the error, not empty
not sql error - by forcing an error in the sql, I get the xls and it contains the error, not empty
not sure of login timeout - by adding -l1, it does not timeout, but I can't specify a number lower than 1 second, so I can't say
I cannot reproduce this, as I do not know the cause. Has anyone else experienced this or have way to address this? Any suggestions how to find the cause? Is it the unix or the Sybase isql?
I found the cause. Since this is scheduled, and this particular report takes a long time to generate. Other scheduled scripts, I found have this line of code:
rm -f $TEMP_DIR/*
If the this long running report, overlaps with one of the scheduled scripts with the line above, the .tmp_1 can possibly be deleted, hence blank by the time it is mailed. I replicated this by manually deleting the .tmp_1 while the report was still writing the sql in there.
I have an expect script which I need to run every 3 mins on my management node to collect tx/rx values for each port attached to DCX Brocade SAN Switch using the command #portperfshow#
Each time I try to use crontab to execute the script every 3 mins, the script does not work!
My expect script starts with #!/usr/bin/expect -f and I am calling the script using the following syntax under cron:
3 * * * * /usr/bin/expect -f /root/portsperfDCX1/collect-all.exp sanswitchhostname
However, when I execute the script (not under cron) it works as expected:
root# ./collect-all.exp sanswitchhostname
works just fine.
Please Please can someone help! Thanks.
The script collect-all.exp is:
#!/usr/bin/expect -f
#Time and Date
set day [timestamp -format %d%m%y]
set time [timestamp -format %H%M]
#logging
set LogDir1 "/FPerf/PortsLogs"
et timeout 5
set ipaddr [lrange $argv 0 0]
set passw "XXXXXXX"
if { $ipaddr == "" } {
puts "Usage: <script.exp> <ip address>\n"
exit 1
}
spawn ssh admin#$ipaddr
expect -re "password"
send "$passw\r"
expect -re "admin"
log_file "$LogDir1/$day-portsperfshow-$time"
send "portperfshow -tx -rx -t 10\r"
expect timeout "\n"
send \003
log_file
send -- "exit\r"
close
I had the same issue, except that my script was ending with
interact
Finally I got it working by replacing it with these two lines:
expect eof
exit
Changing interact to expect eof worked for me!
Needed to remove the exit part, because I had more statements in the bash script after the expect line (calling expect inside a bash script).
There are two key differences between a program that is run normally from a shell and a program that is run from cron:
Cron does not populate (many) environment variables. Notably absent are TERM, SHELL and HOME, but that's just a small proportion of the long list that will be not defined.
Cron does not set up a current terminal, so /dev/tty doesn't resolve to anything. (Note, programs spawned by Expect will have a current terminal.)
With high probability, any difficulties will come from these, especially the first. To fix, you need to save all your environment variables in an interactive session and use these in your expect script to repopulate the environment. The easiest way is to use this little expect script:
unset -nocomplain ::env(SSH_AUTH_SOCK) ;# This one is session-bound anyway
puts [list array set ::env [array get ::env]]
That will write out a single very long line which you want to put near the top of your script (or at least before the first spawn). Then see if that works.
Jobs run by cron are not considered login shells, and thus don't source your .bashrc, .bash_profile, etc.
If you want that behavior, you need to add it explicitly to the crontab entry like so:
$ crontab -l
0 13 * * * bash -c '. .bash_profile; etc ...'
$
I assume that all I need to do is to:
Create an sql file e.g. nameofsqlfile.sql contents:
perform proc_my_sql_funtion();
Execute this as a cron job.
However, I don't know the commands that I'd need to write to get this cron job executed as a postgres function for a specified host,port,database, user & his password...?
You just need to think of cronjob as running a shell command at a specified time or day.
So your first job is to work out how to run your shell command.
psql --host host.example.com --port 12345 --dbname nameofdatabase --username postgres < my.sql
You can then just add this to your crontab (I recommend you use crontab -e to avoid breaking things)
# runs your command at 00:00 every day
#
# min hour wday month mday command-to-run
0 0 * * * psql --host host.example.com --port 12345 --dbname nameofdatabase < my.sql
In most cases you can put all of the sql source in a shell 'here document'. The nice thing about here documents is that the shell's ${MY_VAR} are expanded even within single quotes, e.g:
#!/bin/sh
THE_DATABASE=personnel
MY_TABLE=employee
THE_DATE_VARIABLE_NAME=hire_date
THE_MONTH=10
THE_DAY=01
psql ${THE_DATABASE} <<THE_END
SELECT COUNT(*) FROM ${MY_TABLE}
WHERE ${THE_DATE_VARIABLE_NAME} >= '2011-${THE_MONTH}-${THE_DAY}'::DATE
THE_END
YMMV
Check this
http://archives.postgresql.org/pgsql-admin/2000-10/msg00026.php
and
http://www.dbforums.com/postgresql/340741-cron-jobs-postgresql.html
or you can just create a bash script to include your coding and call it from crontab
For Postgresql 10 and above you can use pg_cron. As stated in its README.md,
pg_cron is a simple cron-based job scheduler for PostgreSQL (10 or higher) that runs inside the database as an extension. It uses the same syntax as regular cron, but it allows you to schedule PostgreSQL commands directly from the database:
I need to loop a oracle sqlplus query using bash.
my scenario is like this. I have a set of names in a text file and i need to find out details of that names using a sqlplus query.
textfile.txt content:
john
robert
samuel
chris
bash script
#!/bin/bash
while read line
do
/opt/oracle/bin/sqlplus -s user#db/password #query.sql $line
done < /tmp/textfile.txt
sql query: query.sql
set verify off
set heading off
select customerid from customers where customername like '%&1%';
exit
problem is when I run the script I get errors like
SP2-0734: unknown command beginning
"robert..." - rest of line ignored.
can someone tell me how to solve this?
The way I do this all the time is as follows:
#!/bin/bash
cat textfile.txt |while read Name
do
sqlplus -s userid/password#db_name > output.log <<EOF
set verify off
set heading off
select customerid from customers where customername like '%${Name}%'
/
exit
EOF
Bash will auto magically expand ${Name} for each line and place it into the sql command before sending it into sqlplus
Do you have set define on ? Is your wildcard & ? You could check glogin.sql to know.
And yes, establishing n connections to pass n queries is probably not a good solution. Maybe it's faster for you to develop and you will do that one time, but if not, you should maybe think of crafting a procedure.