Postgres rotate cron backups - sql

I`m trying to make a script which will make following:
Connect to postgres database, make dump, gzip dump and store to dir.
All this must be rotated - 24 backups for day, 7 for week
File name must contain date, hour and minutes in file name.
Old ones must be deleted (I dont want more backus as described)
This script will run every hour trough cron.
I wrote this script (change some found at web) but there are some bugs:
1) It says: "awk: line 1: syntax error at or near end of line"
2) when I run script it on overwrites dayly backup and don`t make new one
3) hour backup make only folder with name "hourly" and didn`t make backup
4) all backups will be backed up to synology NAS via rsync
Can anyone help me please?
Script backup.sh:
# !/bin/bash
# for use with cron, eg:
# 0 3 * * * postgres /var/db/db_backup.sh example_db
if [[ -z "$1" ]]; then
echo "Usage: $0 <example_db> [pg_dump example_db]"
exit 1
fi
DB="$1"; shift
DUMP_EXAMPLE_DB=$#
DIR="/var/db/backups/$DB"
KEEP_HOURLY=24
KEEP_DAILY=7
KEEP_WEEKLY=5
KEEP_MONTHLY=12
function rotate {
rotation=$1
fdate=`date +%Y-%m-%d-$H -d $date`
file=$DIR/daily/*$fdate*.gz
mkdir -p $DIR/$rotation/ || abort
if [ -f $file ]; then
cp $file $DIR/$rotation/ || abort
else
echo
fi
}
function prune {
dir=$DIR/$1
keep=$2
ls $dir | sort -rn | awk " NR > $keep" | while read f; do rm $dir/$f; done
}
function abort {
echo "aborting..."
exit 1
}
mkdir -p $DIR/hourly || abort
mkdir -p $DIR/daily || abort
mkdir -p $DIR/weekly || abort
mkdir -p $DIR/monthly || abort
mkdir -p $DIR/yearly || abort
date=`date +%Y-%m-%d` || abort
hour=`date -d $date +%H` || abort
minute=`date -d $date +M` || abort
day=`date -d $date +%d` || abort
weekday=`date -d $date +%w` || abort
month=`date -d $date +%m` || abort
# Do the daily backup
/usr/bin/pg_dump $DB $DUMP_EXAMPLE_DB | gzip > $DIR/daily/${DB}_$date.gz
test ${PIPESTATUS[0]} -eq 0 || abort
# Perform rotations
if [[ "$weekday" == "0" ]]; then
rotate weekly
fi
if [[ "$hour/$minute" == "0/0" ]]; then
rotate hourly
fi
if [[ "$day" == "01" ]]; then
rotate monthly
fi
if [[ "$month/$day" == "01/01" ]]; then
rotate yearly
fi
prune hourly $KEEP_HOURLY
prune daily $KEEP_DAILY
prune weekly $KEEP_WEEKLY
prune monthly $KEEP_MONTHLYOD
Many Thanks

Related

Why does the PID change when the find command is issued?

I am studying and making a timeout command that exists in CentOS.
I run the sleep and find commands in the "timeout_bash.sh" file.
However, when the find command is executed, the "timeout_bash.sh" file appears to have been executed by the find command in the process list.
Can you tell me why?
Images URL : https://i.stack.imgur.com/t1U9l.jpg
#script File Name : timeout_bash.sh
#!/bin/bash
timeout_bash() {
scriptName="${0##*/}"
declare -i DEFAULT_TIMEOUT=5
declare -i DEFAULT_INTERVAL=1
declare -i DEFAULT_DELAY=1
declare -i timeout=DEFAULT_TIMEOUT
declare -i interval=DEFAULT_INTERVAL
declare -i delay=DEFAULT_DELAY
while getopts ":t:i:d:" option; do
case "$option" in
t) timeout=$OPTARG ;;
i) interval=$OPTARG ;;
d) delay=$OPTARG ;;
esac
done
shift $((OPTIND - 1))
if (($# == 0 || interval <= 0)); then
echo "Not Enter the Time. So not to execute timeout command"
exit 1
fi
# kill -0 pid Exit code indicates if a signal may be sent to $pid process.
(
((t = timeout))
while ((t > 0)); do
sleep $interval
kill -0 $$
((t -= interval))
done
# Be nice, post SIGTERM first.
# The 'exit 0' below will be executed if any preceeding command fails.
kill -s SIGTERM $$ && kill -0 $$
sleep $delay
kill -s SIGKILL $$
) 2> /dev/null &
exec "$#"
}
sleep 5
timeout_bash -t 60 find / -nouser -print 2>/dev/null >> a.txt
timeout_bash -t 60 find / -nogroup -print 2>/dev/null >> b.txt
timeout_bash -t 60 find / -xdev -perm -2 -ls | >> c.txt
timeout_bash -t 60 find / -xdev -name '.*' -print >> d.txt &

Running a script when connecting to server with ssh

I use the kitty terminal emulator, so when I connect to a new server, I (usually) need to ad the terminfo (at least, this way it seems to work). To do this I wrote a script. While I was at it, I added a bit of code to add a public key if the user wants it to.
Not really relevant for the question, but here is the code:
#!/bin/bash
host=$1
ip=$(echo $host | cut -d# -f2 | cut -d: -f1)
# Check if it is a unknown host
if [[ -z $(ssh-keygen -F $ip) ]]; then
# Check if there are any ssh-keys
if [ $(ls $HOME/.ssh/*.pub > /dev/null | wc -l) -ne 0 ]; then
keys=$(echo $( (cd $HOME/.ssh/ && ls *.pub) | sed "s/.pub//g" ))
ssh -q -o PubkeyAuthentication=yes -o PasswordAuthentication=no $host "ls > /dev/null 2>&1"
# Check if the server has one of the public keys
if [ $? -ne 0 ]; then
echo "Do you want to add a SSh key to the server?"
while true; do
read -p " Choose [$keys] or leave empty to skip: " key
if [[ -z $key ]]; then
break
elif [[ -e $HOME/.ssh/$key ]]; then
# Give the server a public key
ssh $host "mkdir -p ~/.ssh && chmod 700 ~/.ssh && echo \"$(cat $HOME/.ssh/$key.pub)\" >> ~/.ssh/authorized_keys"
break
else
echo "No key with the name \"$key\" found."
fi
done
fi
fi
# Copy terminfo to server
ssh -t $host "echo \"$(infocmp -x)\" > \"\$TERM.info\" && tic -x \"\$TERM.info\" && rm \$TERM.info"
fi
It is not the best code, but it seems to work. Tips are ofcourse welcome.
The problem is that I need to run this script every time I connect te a new remote server (or I need to keep track of which server is new, but that is even worse). Is there a way to run this script every time I connect to a server (the script checks if the ip is a known host).
Or is there an other way to do this? Adding the public keys is nice to have, but not very important.
I hope somone can help,
Thanks!
There is a trick to identify that you are using ssh to login on the target machine:
pgrep -af "sshd.*"$USER |wc -l
The above command will count the user's processes using sshd
You can add the above command in the target machine, to test if you are connected via ssh. Add the above command to your .profile or .bash_profile script in the target machine.
So that only if you login via ssh your script will run initiation script on the target machine when you login/connect.
Sample .bash_profile on target machine
#!/bin/bash
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
if [[ $(pgrep -af "sshd.*"$USER |wc -l) -gt 0 ]]; then
your_init_script
fi

AccuRev: how do you list the managed files?

I need to see which files have been added or removed between two streams. The most obvious way would be "git lsfiles" in each stream. Except this is not GIT and I do not see an analogous command. So for today:
for f in $(find * -type f);do
accurev stat "$f"
done | \
fgrep -v '(external)' | \
awk '{print $1}' > .list
If there is a better way, it should be clear and easy to find here:
http://www.accurev.com/download/docs/5.7.0_books/AccuRev_5_7_User_CLI.pdf
but it is not. Help? Thank you.
If you want to see the difference between two streams, run the following command: accurev diff -a -v "Stream1" -V "Stream2"
As the command line question has been answered, here's how to do the same via the AccuRev GUI.
Select one dynamic stream, workspace or snapshot.
Right click and select "Show Diff By Files"
Select a different dynamic stream, workspace or snapshot.
You'll be presented with a list of files different between the two choices, and yes you can mix-and-match between dynamic streams, workspaces and snapshots.
You can then select any file and select "Show Difference" to see differences between the two files.
Since neither of the two answers addressed the question, I eventually worked out a script to do what is really needed. "accurev lsfiles" is sorely needed.
#! /bin/bash
declare -r progpid=$$
declare -r progdir=$(cd $(dirname $0) >/dev/null && pwd)
declare -r prog=$(basename $0)
declare -r program="$progdir/$prog"
declare -r usage_text=' [ <directory> ... ]
If no directory is specified, "." is assumed'
die() {
echo "$prog error: $*"
exec 1>/dev/null 2>&1
kill -9 $progpid
exit 1
} 1>&2
usage() {
test $# -gt 0 && {
exec 1>&2
echo "$prog usage error: $*"
}
printf "USAGE: $prog %s\n" "$usage_text"
exit $#
}
init() {
shift_ct=$#
tmpd=$(mktemp -d ${TMPDIR:-/tmp}/ls-XXXXXX)
test -d "$tmpd" || die "mktemp -d does not work"
exec 4> ${tmpd}/files
trap "rm -rf '$tmpd'" EXIT
prune_pat=
while :
do
test $# -eq 0 && break
test -f "$1" && break
[[ "$1" =~ -.* ]] || break
case "X$1" in
X-p )
prune_pat+="${2}|"
shift 2 || usage "missing arg for '-p' option"
;;
X-p* )
prune_pat+="${1#-p}"
shift
;;
X-x* )
set -x
tput reset 1>&2
PS4='>${FUNCNAME:-lsf}> '
shift
;;
X-* )
usage "unknown option: $1"
;;
* )
break
;;
esac
done
(( shift_ct -= $# ))
test ${#prune_pat} -gt 0 && \
prune_pat='(^|/)('${prune_pat%|}')$'
}
chkdir() {
declare list=$(exec 2> /dev/null
for f in "$#"
do ls -d ${f}/* ${f}/.*
done | \
grep -v -E '.*/\.\.*$' )
for f in $(accurev stat ${list} | \
grep -v -F '(external)' | \
awk '{print $1}' | \
sed 's#^/*\./##')
do
test ${#prune_pat} -gt 0 && [[ $f =~ ${prune_pat} ]] && continue
if test -d "$f"
then chkdir "$f"
elif test -f "$f" -o -L "$f"
then echo "$f" 1>&4
fi
done
}
init ${1+"$#"}
(( shift_ct > 0 )) && shift ${shift_ct}
test $# -eq 0 && set -- .
chkdir "$#"
exec 4>&-
sort -u ${tmpd}/files
It is a bit over-the-top, but I have a boilerplate I always use for my scripts.

shell script not working

I have a shell script to do the following things
sudo as a user (johnsmith) and perform few things
Exit from that user and check url status
If status is not equal to 1 , ssh to one more server and execute a
script.
But when I am running it, the lines inside 'ENDBASH' are not getting executed at all.
#!/bin/ksh
echo "Outside ENDBASH ${###*/}"
sudo -u johnssmith bash <<'ENDBASH'
echo "Inside ENDBASH ${###*/}"
#Obtaining the new version file
for file in "${###*/}"
do
if echo "$file" | grep -E "abc_cde_efg"; then
echo "Version found: $file"
else
echo "Version not found"
fi
done
exit
ENDBASH
urlArray=('http://server:port/servicename1/services/servicename1?wsdl' 'http://server:port/servicename2/services/servicename2?wsdl')
status=0
for url in "${urlArray[#]}"
do
result=`curl -s $url`
if (echo $result | grep '<?xml' >/dev/null 2>&1); then
service=$(echo $url | cut -d"/" -f4)
echo "$service is Running"
else
service=$(echo $url | cut -d"/" -f4)
echo "$service is not Running"
status=1
fi
done
if [ $status != 1 ] ; then
ssh -t username#hostname /home/dev_was/test1.sh
fi
You need to explicitly pass the arguments received by your script to the internal script:
sudo -u johnssmith bash -s "$#" <<'ENDBASH'

Host key verification failed [rsync: connection unexpectedly closed]

I'm stuck with a peculiar problem, where rsync command is not running when it is executed through crontab.
Below is the code :
#!/bin/sh -x
PATH=/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/mysql/bin:/opt/android-sdk-linux/tools:/opt/android-sdk-linux/platform-tools:~/usr/lib/jvm/jdk-6/bin
/bin/sh /etc/profile
MyUSER="root" # USERNAME
MyPASS="password" # PASSWORD
MyHOST="localhost" # Hostname
Password="" #Linux Password
MYSQL="$(which mysql)"
if [ -z "$MYSQL" ]; then
echo "Error: MYSQL not found"
exit 1
fi
MYSQLADMIN="$(which mysqladmin)"
if [ -z "$MYSQLADMIN" ]; then
echo "Error: MYSQLADMIN not found"
exit 1
fi
CHOWN="$(which chown)"
if [ -z "$CHOWN" ]; then
echo "Error: CHOWN not found"
exit 1
fi
CHMOD="$(which chmod)"
if [ -z "$CHMOD" ]; then
echo "Error: CHMOD not found"
exit 1
fi
GZIP="$(which gzip)"
if [ -z "$GZIP" ]; then
echo "Error: GZIP not found"
exit 1
fi
CP="$(which cp)"
if [ -z "$CP" ]; then
echo "Error: CP not found"
exit 1
fi
MV="$(which mv)"
if [ -z "$MV" ]; then
echo "Error: MV not found"
exit 1
fi
RM="$(which rm)"
if [ -z "$RM" ]; then
echo "Error: RM not found"
exit 1
fi
RSYNC="$(which rsync)"
if [ -z "$RSYNC" ]; then
echo "Error: RSYNC not found"
exit 1
fi
MYSQLBINLOG="$(which mysqlbinlog)"
if [ -z "$MYSQLBINLOG" ]; then
echo "Error: MYSQLBINLOG not found"
exit 1
fi
# Get data in dd-mm-yyyy format
NOW="$(date +"%d-%m-%Y-%T")"
DEST="/home/db-backup"
mkdir $DEST/Increment_backup.$NOW
LATEST=$DEST/Increment_backup.$NOW
$MYSQLADMIN -u$MyUSER -p$MyPASS flush-logs
newestlog=`ls -d /usr/local/mysql/data/mysql-bin.?????? | sed 's/^.*\.//' | sort -g | tail -n 1`
echo $newestlog
for file in `ls /usr/local/mysql/data/mysql-bin.??????`
do
if [ "/usr/local/mysql/data/mysql-bin.$newestlog" != "$file" ]; then
echo $file
echo $Password | sudo -S $CHMOD 0777 $file
#sudo $MYSQLBINLOG $file>$file.$NOW.sql
$CP "$file" $LATEST
#$RM "$file.$NOW.sql"
#$MV $file.sql.gz /$LATEST
fi
done
for file1 in `ls $LATEST/mysql-bin.??????`
do
$MYSQLBINLOG $file1>$file1.$NOW.sql
$GZIP -9 "$file1.$NOW.sql"
$RM "$file1"
done
$RSYNC -v -e ssh $LATEST abc#192.168.1.9:/home/rsync-backup/
#FILE=$LATEST/"mysql-bin.??????"
#$MYSQLBINLOG $FILE>$FILE.$NOW.sql
#$GZIP -f "$FILE.$NOW.sql"
pwd
Rsync happens when the code is run manually, but fails when it is run through crontab. Rest of the commands are working fine. From the logs I got this information:
Host key verification failed.^M
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(600) [sender=3.0.6]
This is basically due to the first time authentication issue for ssh. If you were to ensure that the host is added to known_hosts manually or have an expect for the prompt in your script, it should work.
The authenticity of host '[IP]:20022 ([IP]:22)' can't be established.
RSA key fingerprint is bc:87:52:cf:ac:3e:67:74:1b:e1:0b:e3:e2:06:d8:21.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[IP]:22' (RSA) to the list of known hosts
This kind of error is usually caused by differences in the environment. A good troubleshooting step is to run "env" at the start of the program and compare the cron and non-cron version.
You should also run the command as "sh -x" which will show you all the expansions which will help identify which variable is not being set properly.
This could be your HOME directory is encrypted.
If your user is logged, it works, but when it's a cron, even it's executing the same user it does not have access to your $HOME/.ssh directory