I'm using s3cmd to backup my databases to Amazon S3, but I'd also like to backup a certain folder and archive it.
I have this part from this script that successfully backups the databases to S3:
# Loop the databases
for db in $databases; do
# Define our filenames
filename="$stamp - $db.sql.gz"
tmpfile="/tmp/$filename"
object="$bucket/$stamp/$filename"
# Feedback
echo -e "\e[1;34m$db\e[00m"
# Dump and zip
echo -e " creating \e[0;35m$tmpfile\e[00m"
mysqldump -u root -p$mysqlpass --force --opt --databases "$db" | gzip -c > "$tmpfile"
# Upload
echo -e " uploading..."
s3cmd put "$tmpfile" "$object"
# Delete
rm -f "$tmpfile"
done;
How can I add another section to archive a certain folder, upload to S3 and then delete the local archive?
Untested and basic but this should get the job done with some minor tweaks
# change to tmp dir - creating archives with absolute paths can be dangerous
cd /tmp
# create archive with timestamp of dir /path/to/directory/to/archive
tar -czf "$stamp-archivename.tar.gz" /path/to/directory/to/archive
# upload archive to s3 bucket 'BucketName'
s3cmd put "/tmp/$stamp-archivename.tar.gz" s3://BucketName/
# remove local archive
rm -f "/tmp/$stamp-archivename.tar.gz"
Related
When s3 integration is enabled on Ant media server, recorded VODs are uploading to s3 bucket but when we are uploading any VOD file on server it is not uploading on s3 bucket. Is there any way to do that on the server side?
You can automatically transfer the VoD files with this script to the S3 bucket by following the steps below.
1. Install FFmpeg
apt-get update && apt-get install ffmpeg -y
2. Save the script under /usr/local/antmedia/ and change permission by using chmod command it. (Don't forget to add AWS Access/Secret keys)
You can download the script from the following link or you can find it at the bottom of the page.
https://github.com/muratugureminoglu/Scripts/blob/master/vod-upload-s3.sh
chmod +x /usr/local/antmedia/vod-upload-s3.sh
3. Modify the red5-web.properties file in your webapps as follows.
vim [AMS-DIR]/webapps/your_application/WEB-INF/red5-web.properties
Add or change the following line.
settings.vodUploadFinishScript=/usr/local/antmedia/vod-upload-s3.sh
4. Restart Ant Media Server service.
systemctl restart antmedia
5. Follow the link below to play the uploaded VoD files.
VOD not playing after s3 recording enabled in Ant Media server
That's it.
Script
#!/bin/bash
#
# Installation Instructions
#
# apt-get update && apt-get install ffmpeg -y
# vim [AMS-DIR]/webapps/applications(LiveApp or etc.)/WEB-INF/red5-web.properties
# settings.vodUploadFinishScript=/Script-DIR/vod-upload-s3.sh
# sudo service antmedia restart
#
# Check if AWS CLI is installed
if [ -z `which aws` ]; then
rm -r aws* > /dev/null 2>&1
echo "Please wait. AWS Client is installing..."
curl "https://d1vvhvl2y92vvt.cloudfront.net/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" > /dev/null 2>&1
unzip awscliv2.zip > /dev/null 2>&1
sudo ./aws/install &
wait $!
echo "AWS Client installed."
rm -r aws*
fi
# Delete the uploaded VoD file from local disk
DELETE_LOCAL_FILE="Y"
AWS_ACCESS_KEY=""
AWS_SECRET_KEY=""
AWS_REGION=""
AWS_BUCKET_NAME=""
#AWS Configuration
aws configure set aws_access_key_id $AWS_ACCESS_KEY
aws configure set aws_secret_access_key $AWS_SECRET_KEY
aws configure set region $AWS_REGION
aws configure set output json
tmpfile=$1
mv $tmpfile ${tmpfile%.*}.mp4"_tmp"
ffmpeg -i ${tmpfile%.*}.mp4"_tmp" -c copy -map 0 -movflags +faststart $tmpfile
rm ${tmpfile%.*}.mp4"_tmp"
aws s3 cp $tmpfile s3://$AWS_BUCKET_NAME/streams/ --acl public-read
if [ $? != 0 ]; then
logger "$tmpfile failed to copy file to S3. "
else
# Delete the uploaded file
if [ "$DELETE_LOCAL_FILE" == "Y" ]; then
aws s3api head-object --bucket $AWS_BUCKET_NAME --key streams/$(basename $tmpfile)
if [ $? == 0 ];then
rm -rf $tmpfile
logger "$tmpfile is deleted."
fi
fi
fi
I would like to rename all files in my Amazon S3 bucket with extension.PDF to .pdf (lowercase).
Did someone already have to do this? There are a lot of files (around 1500). Is S3cmd the best way to do this? How would you do?
s3cmd --recursive ls s3://bucketname |
awk '{ print $4 }' | grep "*.pdf" | while read -r line ; do
s3cmd --recursive mv s3://<s3_bucketname>/$line/ s3://<s3_bucketname>/${line%.*}".PDF"
done
A local linux/unix example for renaming all files with .pdf extension to .PDF extension.
mkdir pdf-test
cd pdf-test
touch a{1..10}.pdf
Before
ls
a1.pdf a2.pdf a4.pdf a6.pdf a8.pdf grep.sh
a10.pdf a3.pdf a5.pdf a7.pdf a9.pdf
The script file grep.sh
#/bin/bash
ls |grep .pdf | while read -r line ; do # here use ls from s3
echo "Processing $line"
# your s3 code goes here
mv $line ${line%.*}".PDF"
done
Add permissions and try
chmod u+x grep.sh
./grep.sh
After
ls
a1.PDF a2.PDF a4.PDF a6.PDF a8.PDF grep.sh
a10.PDF a3.PDF a5.PDF a7.PDF a9.PDF
You can apply the same logic. instead of mv use s3 mv.
I need to compare the contents of a local folder with a AWS S3 bucket so that where there are differences a script is executed on the local files.
The idea is that local files (pictures) get encrypted and uploaded to S3. Once the upload has occurred I delete the encrypted copy of the pictures to save space. The next day new files get added to the local folder. I need to check between the local folder and the S3 bucket which pictures have already been encrypted and uploaded so that I only encrypt the newly added pictures rather than all of them all over again. I have a script that does exactly this between two local folders but I'm struggling to adapt it so that the comparison is performed between a local folder and a S3 bucket.
Thank you to anyone who can help.
Here is the actual script I am currently using for my picture sorting, encryption and back up to S3:
!/bin/bash
perl /volume1/Synology/scripts/Exiftool/exiftool '-createdate
perl /volume1/Synology/scripts/Exiftool/exiftool '-model=camera model missing' -r -if '(not $model)' -overwrite_original -r /volume1/photo/"input"/ --ext .DS_Store -i "#eaDir"
perl /volume1/Synology/scripts/Exiftool/exiftool '-Directory
cd /volume1/Synology/Pictures/"Pictures Glacier back up"/"Compressed encrypted pics for Glacier"/post_2016/ && (cd /volume1/Synology/Pictures/Pictures/post_2016/; find . -type d ! -name .) | xargs -i mkdir -p "{}"
while IFS= read -r file; do /usr/bin/gpg --encrypt -r xxx#yyy.com /volume1/Synology/Pictures/Pictures/post_2016/**///$(basename "$file" .gpg); done < <(comm -23 <(find /volume1/Synology/Pictures/Pictures/post_2016 -type f -printf '%f.gpg\n'|sort) <(find /volume1/Synology/Pictures/"Pictures Glacier back up"/"Compressed encrypted pics for Glacier"/post_2016 -type f -printf '%f\n'|sort))
rsync -zarv --exclude=#eaDir --include="/" --include=".gpg" --exclude="" /volume1/Synology/Pictures/Pictures/post_2016/ /volume1/Synology/Pictures/"Pictures Glacier back up"/"Compressed encrypted pics for Glacier"/post_2016/
find /volume1/Synology/Pictures/Pictures/post_2016/ -name ".gpg" -type f -delete
/usr/bin/aws s3 sync /volume1/Synology/Pictures/"Pictures Glacier back up"/"Compressed encrypted pics for Glacier"/post_2016/ s3://xyz/Pictures/post_2016/ --exclude "" --include ".gpg" --sse
It would be inefficient to continually compare the local and remote folders, especially as the quantity of objects increases.
A better flow would be:
Unencrypted files are added to a local folder
Each file is:
Copied to another folder in an encrypted state
Once that action is confirmed, the original file is then deleted
Files in the encrypted local folder are copied to S3
Once that action is confirmed, the source file is then deleted
The AWS Command-Line Interface (CLI) has an aws s3 sync command that makes it easy to copy new/modified files to an Amazon S3 bucket, but this could be slow if you have thousands of files.
I need to transfer all *.png from different directories from a remote server BUT preserving the full path of each .png file because all .png files have the same name.
scp -r -e server:coverages/K4me3/*/pos/output/*/*.png Desktop/
While coping it rewrites already existing .png files because the names od them are the same in different directories. I want to preserve the full directory path,s o that when copying, the .png files are copied within their own directories.
SCP doesn't preserve the paths of files on its own, as you have discovered.
You'll probably want to use rsync to do this, since rsync does preserve paths
I think the command would be:
rsync -a -r -v -z server_config:/path/to/root/directory/on/server [destination_folder]
This is the reverse of this question: scp a folder to a remote system keeping the directory layout
Alternatively, and as the comments suggest, you can write a script to get all of the files or lower level directories (with absolute path) and call an scp transfer on each of them. Here is a script that I at one point used to copy files in this way:
#!/usr/bin/env python
from multiprocessing.dummy import Pool
from subprocess import call
from functools import partial
root = # Root Directory
files = [
root + # Sub 1,
root + # Sub 2,
root + # Sub 3,
root + # Sub etc,
]
command_s = "scp -r -v -c arcfour -F /path/to/.ssh/config Server:"
command_e = " Output_Dir/"
max_processes = 4
# Transfer the files 4 at a time because my computer is busy with other stuff
cmds = []
for filename in files:
cmds.append(command_s + filename + command_e)
pool = Pool(max_processes)
for i, returncode in enumerate(pool.imap(partial(call, shell=True), cmds)):
if returncode != 0:
print ("%d command failed: %d" % (i, returncode))
Here is an answer to preserve directory structure and copy just the png files from a server to a local system based on ssh.
ssh user#server 'find /server/path -name "*.png" -print0 | xargs -0 tar -cO' | tar -xfv - -C .
Source: Link
I have a large number of files which I need to backup, problem is there isn't enough disk space to create a tar file of them and then upload it offsite. Is there a way of using python, php or perl to tar up a set of files and upload them on-the-fly without making a tar file on disk? They are also way too large to store in memory.
I always do this just via ssh:
tar czf - FILES/* | ssh me#someplace "tar xzf -"
This way, the files end up all unpacked on the other machine. Alternatively
tar czf - FILES/* | ssh me#someplace "cat > foo.tgz"
Puts them in an archive on the other machine, which is what you actually wanted.
You can pipe the output of tar over ssh:
tar zcvf - testdir/ | ssh user#domain.com "cat > testdir.tar.gz"