There are many tools available to help backup Linux systems to Amazon S3 but finding the right one to use can be difficult. Jeremy Zawodny made a good list of various S3 backup tools which is very helpful, if a little outdated. I experimented with a few tools, including some of the standard scripts published by Amazon but I found each had their own shortcomings.
One tool that is very simple to use is s3cmd which is a linux command line tool to upload, retrieve, and manage data in Amazon S3. The tool is written in python so should install and run on pretty much any modern linux distro and I have found it works very nicely and seems to be an ideal tool to use if you want to write a basic backup script.
Once installed you just run “s3cmd –configure” to configure the script with your Amazon S3 access keys. You can then use s3cmd from the command line to manage your S3 buckets and their contents. Full details of the various options can be obtained through running “s3cmd –help”.
I have used s3cmd to create automated daily rotated backups of a single directory (e.g. /home) using a simple bash script which I adapted from the automysqlbackup script. The script I have used is shown below, and provides rotated backups of the previous 7 days, each day individually archived in a single tar.gz file. This is a fairly limited solution as it only provides a backup of a single directory, does not do incremental backups, and it will break if your tar.gz file exceeds the S3 individual file storage limit of ~6GB. However, if you just need a simple solution to maintain daily backups of a single folder then its a nice tidy option. It will also email you the results, and can optionally include jobs to run pre- and post- backup.
#!/bin/bash
#
# Backup Script
#
#=====================================================================
#=====================================================================
# Set the following variables to your system needs
#=====================================================================
# Directory to save daily tar.gz backup files to e.g /backups
BACKUPDIR="/backups"
# Directory to backup
BACKUPSRCDIR="/home"
# Maximum size of backup files in MB (larger files will be split into separate archives)
# Note: not implemented yet
MAXTARSIZE="1000"
# S3 Settings
# The name of the S3 bucket to upload to e.g. "my_s3_bucket"
S3BUCKET="my_s3_bucket"
# Mail setup
# What would you like to be mailed to you?
# - log : send only log file
# - files : send log file and tar.gz files as attachments
# - stdout : will simply output the log to the screen if run manually.
# - quiet : Only send logs if an error occurs to the MAILADDR.
MAILCONTENT="log"
# Set the maximum allowed email size in k. (4000 = approx 5MB email [see docs])
MAXATTSIZE="4000"
# Email Address to send mail to? (user@domain.com)
MAILADDR="user@domain.com"
# Command to run before backups (uncomment to use)
#PREBACKUP="/etc/backup-pre"
# Command run after backups (uncomment to use)
#POSTBACKUP="/etc/backup-post"
#=====================================================================
#=====================================================================
#=====================================================================
#
# Should not need to be modified from here down!!
#
#=====================================================================
#=====================================================================
#=====================================================================
PATH=/usr/local/bin:/usr/bin:/bin:
DATE=`date +%Y-%m-%d_%Hh%Mm` # Datestamp e.g 2002-09-21
DOW=`date +%A` # Day of the week e.g. Monday
DNOW=`date +%u` # Day number of the week 1 to 7 where 1 represents Monday
DOM=`date +%d` # Date of the Month e.g. 27
M=`date +%B` # Month e.g January
W=`date +%V` # Week Number e.g 37
VER=0.1 # Version Number
HOST=`hostname` # Hostname for LOG information
LOGFILE=$BACKUPDIR/$HOST-`date +%N`.log # Logfile Name
LOGERR=$BACKUPDIR/ERRORS_$HOST-`date +%N`.log # Error log Name
BACKUPFILES=""
# Create required directories
if [ ! -e "$BACKUPDIR" ] # Check Backup Directory exists.
then
mkdir -p "$BACKUPDIR"
fi
# IO redirection for logging.
touch $LOGFILE
exec 6>&1 # Link file descriptor #6 with stdout.
# Saves stdout.
exec > $LOGFILE # stdout replaced with file $LOGFILE.
touch $LOGERR
exec 7>&2 # Link file descriptor #7 with stderr.
# Saves stderr.
exec 2> $LOGERR # stderr replaced with file $LOGERR.
# Functions
# Backup function: removes last weeks archive from S3, creates new tar.gz and sends to S3
SUFFIX=""
dobackup () {
s3cmd ls s3://"$S3BUCKET" | grep s3 | sed "s/.*s3:\/\/$S3BUCKET\//s3:\/\/$S3BUCKET\//" | grep "$DOW" | xargs s3cmd del
tar cfz "$1" "$2"
echo
echo Backup Information for "$1"
gzip -l "$1"
echo
s3cmd put "$1" s3://"$S3BUCKET"
return 0
}
# Run command before we begin
if [ "$PREBACKUP" ]
then
echo ======================================================================
echo "Prebackup command output."
echo
eval $PREBACKUP
echo
echo ======================================================================
echo
fi
echo ======================================================================
echo BackupScript VER $VER
echo http://www.cenolan.com/
echo
echo Backup of Server - $HOST
echo ======================================================================
echo Backup Start Time: `date`
echo ======================================================================
# Daily Backup
echo Daily Backup of Directory \( $BACKUPSRCDIR \)
echo
echo Rotating last weeks Backup...
eval rm -fv "$BACKUPDIR/*.$DOW.tar.gz"
echo
dobackup "$BACKUPDIR/$DATE.$DOW.tar.gz" "$BACKUPSRCDIR"
BACKUPFILES="$BACKUPFILES $BACKUPDIR/$DATE.$DOW.tar.gz"
echo
echo ----------------------------------------------------------------------
echo Backup End Time: `date`
echo ======================================================================
echo Total disk space used for backup storage..
echo Size - Location
echo `du -hs "$BACKUPDIR"`
echo
echo ======================================================================
echo ======================================================================
# Run command when we're done
if [ "$POSTBACKUP" ]
then
echo ======================================================================
echo "Postbackup command output."
echo
eval $POSTBACKUP
echo
echo ======================================================================
fi
#Clean up IO redirection
exec 1>&6 6>&- # Restore stdout and close file descriptor #6.
exec 1>&7 7>&- # Restore stdout and close file descriptor #7.
if [ "$MAILCONTENT" = "files" ]
then
if [ -s "$LOGERR" ]
then
# Include error log if is larger than zero.
BACKUPFILES="$BACKUPFILES $LOGERR"
ERRORNOTE="WARNING: Error Reported - "
fi
#Get backup size
ATTSIZE=`du -c $BACKUPFILES | grep "[[:digit:][:space:]]total$" |sed s/\s*total//`
if [ $MAXATTSIZE -ge $ATTSIZE ]
then
BACKUPFILES=`echo "$BACKUPFILES" | sed -e "s# # -a #g"` #enable multiple attachments
mutt -s "$ERRORNOTE MySQL Backup Log and SQL Files for $HOST - $DATE" $BACKUPFILES $MAILADDR < $LOGFILE #send via mutt
else
cat "$LOGFILE" | mail -s "WARNING! - Backup exceeds set maximum attachment size on $HOST - $DATE" $MAILADDR
fi
elif [ "$MAILCONTENT" = "log" ]
then
cat "$LOGFILE" | mail -s "Backup Log for $HOST - $DATE" $MAILADDR
if [ -s "$LOGERR" ]
then
cat "$LOGERR" | mail -s "ERRORS REPORTED: MySQL Backup error Log for $HOST - $DATE" $MAILADDR
fi
elif [ "$MAILCONTENT" = "quiet" ]
then
if [ -s "$LOGERR" ]
then
cat "$LOGERR" | mail -s "ERRORS REPORTED: Backup error Log for $HOST - $DATE" $MAILADDR
cat "$LOGFILE" | mail -s "Backup Log for $HOST - $DATE" $MAILADDR
fi
else
if [ -s "$LOGERR" ]
then
cat "$LOGFILE"
echo
echo "###### WARNING ######"
echo "Errors reported during Backup execution.. Backup failed"
echo "Error log below.."
cat "$LOGERR"
else
cat "$LOGFILE"
fi
fi
if [ -s "$LOGERR" ]
then
STATUS=1
else
STATUS=0
fi
# Clean up Logfile
eval rm -f "$LOGFILE"
eval rm -f "$LOGERR"
exit $STATUS
I have posted this largely for my own information but maybe somebody else will find this useful.
Txs, man i found your post usefull! Just wanted to let you know.
Nice script. I love to see people using S3. It’s extremely helpful to have people like yourself share their experience.
I’m not much of a Linux expert, but I had recently tried to write my own backup script to S3. It seems my script was much shorter than yours, so now I’m wondering if I might have missed some important steps? Would any Linux experts care to review my thoughts and let me know if I’m missing anything?
http://blog.datajelly.com/2008/07/ubuntu-script-to-backup-data-to-amazon.html
Thanks for posting on this topic (Linux backups to S3). I’m looking for a solution that does incremental backups to S3. Have you found a solution for that, by chance? Any advice will be appreciated. Marc
Hi Marc,
I use an incremental backup solution myself and have been meaning to do a write-up of it for *ages* but never seem to get around to it. Basically I use duplicity and I’ve written a little wrapper script that handles all the encryption, rotation, logging and other bits and bobs. I will try to put together a guide ASAP. Meanwhile check out the duplicity package.
Nice work. The script works like a charm. thanks very much.
If you find yourself needing a more flexible solution, my S3 backup script may interest you: http://dev.davidsoergel.com/trac/s3napback/. It’s very easy to use and handles backup rotation, incremental backups, compression, encryption, and MySQL and Subversion dumps. Enjoy!
This is really clear and very functional. Fantastic.
The best crafted tool and tutorial to be found – Thanks Chris!
thanks for this chris. i was going to ask you to your face, but you’d have just told me to google it. so i googled it, and you still ended up telling me.
ho ho
Great share. I just finished my first remote backup on AWS S3.
Awesome! Happy holidays ;)
Hi,
Simple and effective script- Thank you. But i need to encrypt before it uploads to the s3. Is there any options. s3cmd as a option of -e, but all are in vain – no use
Please help me on this….
There is also a GUI client for linux http://www.dragondisk.com/
Chechout this url,
http://jackal777.wordpress.com/2011/03/22/cpanel-backup-to-amazon-s3/
Right now this script removes the previous backup form s3, but what if I want it to leave it there? …
Hi
I am new to s3…
How can i use this script and what are all the per-requesites is required to execute is script.
Thanks in advance.
Guatt
When I try to run this from terminal I get this error:
any ideas why?
Hey Steven,
Old post, I know, but the reason you’re getting that error is line wrapping. The error says that line 17 in your file is “separate archives)”… but that’s actually supposed to be the end of line 16.
Some text editors are better than others at preserving full lines. I’m not sure what you’re using. I just copied it into Notepad++ on Windows and it’s correct. On linux, vim/vi should work, but I know some editors like pico and nano will wrap by default.
Thanks for that. Yeah, I used nano.