I have used this method on Fedora, YDL, and CentOS but the instructions should equally apply to other Linux distributions including Debian and Ubuntu. It will even work on OS X using the MacPorts version of duplicity.

Aims of this guide

This guide explains how to create a simple wrapper script for duplicity that allows you to automatically create GPG encrypted incremental backups that are saved to an Amazon S3 bucket. The script is designed to be executed as a daily cron job so that incremental snapshot backups are created each day. The script creates a full backup set on the 1st day of each month (or when an appropriate full backup cannot be found) and then creates incremental backups on subsequent days.

This guide provides a walk-through of how to create the GPG encryption key, and provides full scripts and example usage for both backup and restore. You could easily adapt the backup script so that it makes full backups each week, or otherwise adjust it to suit your individual needs.

This guide is written with the general Linux user in mind: you do need some understanding of basic linux concepts such as cron, permissions, and directory structures.

What is duplicity?

From the duplicity home page:

Duplicity backs [up] directories by producing encrypted tar-format volumes and uploading them to a remote or local file server. Because duplicity uses librsync, the incremental archives are space efficient and only record the parts of files that have changed since the last backup. Because duplicity uses GnuPG to encrypt and/or sign these archives, they will be safe from spying and/or modification by the server.

I think that says it all much more concisely than I could manage.

One thing to note is that in my experience, and on certain machines, duplicity can cause a lot of overhead and take a long time to complete. Thus duplicity is not always a viable option when backing up huge amounts of data. That said, for backing up the critical data from a standard web server it can be a great solution. Remember, that if you're backing up databases then you need to dump them into SQL files first. For MySQL databases I recommend automysqlbackup for this. As always, YMMV.

Before we start

You need to install duplicity (version >= 0.4.3 for S3 support). This how-to doesn't cover that aspect, but suffice to say that duplicity is available as a package for most major distros so crack open your package manager (be it yum, apt, synaptics or whatever) and install duplicity along with all it's dependencies.

You also need GnuGP and librsync but they should both be automatically installed as dependencies of duplicity.

Step 1 - Generate a new GPG key

If you already have a GPG key that you want to use then skip this bit - you'll just need to know what your key is which you can get through "gpg --list-keys" - it is the bit after the / in the "pub" line. Otherwise, read on...

I am going to presume that you'll be running your backup jobs as root, so open a terminal and become root. If you're going to run them as a different user then become that user instead but ensure that the user you have chosen has sufficient permissions to backup the data you require.

Now run "gpg --gen-key" to generate your key and follow the prompts:

# gpg --gen-key
gpg (GnuPG) 1.4.9; Copyright (C) 2008 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please select what kind of key you want:
   (1) DSA and Elgamal (default)
   (2) DSA (sign only)
   (5) RSA (sign only)
Your selection?

Accept the default (Enter) or press 1 for DSA and Elgamal.

DSA keypair will have 1024 bits.
ELG-E keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 

Again, the default (2048) is fine. Just hit Enter.

Requested keysize is 2048 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 

I don't want my key to expire, so I just hit Enter again to accept the default. Do whatever you want.

Key does not expire at all
Is this correct? (y/N) 

Sure is. Hit y and then Enter.

You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
    "Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"

Real name: Duplicity Backup
Email address: duplicity@mydomain.com
Comment: Key for duplicity
You selected this USER-ID:
    "Duplicity Backup (Key for duplicity) <duplicity@mydomain.com>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit?

Enter the requested details and then press O for Okay.

You need a Passphrase to protect your secret key.

Enter Passphrase:

Enter a passphrase here. It should be something long and complex. Anything will do, but make sure you remember it because you'll need it later. When finished press Enter and then re-enter your passphrase when prompted and then press Enter again.

At this stage you may have to help generate some entropy by doing some other task - I find that running "updatedb" in another shell is pretty good, or just randomly tapping the keyboard can do the trick too.

Once it has finished you should get a message like this:

gpg: key BE9274BD marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
pub   1024D/BE9274BD 2008-11-30
      Key fingerprint = 2FB4 A20E 57BA 80BA 9576  3ABD F79F D430 BE92 74BD
uid                  Duplicity Backup (Key for duplicity) <duplicity@mydomain.com>
sub   2048g/F8F35AD8 2008-11-30

Make a note of the key (BE9274BD in this case) as you'll need that later too.

Important: Remember to backup your GPG key pair somewhere safe and off the current machine. Without this key pair your backups are totally useless to you, so if you lose it and need to restore a backup then you're up a creak without a paddle. This article shows the proper way to export (and import) your GPG key pair.

Step 2 - The backup wrapper script

This bash wrapper script does a full backup on the 1st day of each month followed by incremental backups on subsequent days. It will also delete old backup sets after X months have passed and it also emails a log report each day giving some valuable statistics about your backup and reporting any errors.

You will need to have the following information handy to edit this backup script for your needs:

• Your Amazon S3 Access Key ID
• Your Amazon S3 Secret Access Key
• Your GPG key
• Your GPG key passphrase
• A list of directories you want to back up
• An email address to send the logs to
• A unique name for an Amazon S3 bucket (the bucket will be created if it doesn't yet exist)

The script is as follows, you need to change the bits in bold at least but pay attention to all the variables as you may want to tweak them to suit your needs.

Note that includes/excludes work on a 'fist match' basis. So if you want to exclude something in a directory, you need to exclude the file/subdirectory before including the directory. For more info see the duplicity man pages.

#!/bin/bash

# Set up some variables for logging
LOGFILE="/var/log/backup.log"
DAILYLOGFILE="/var/log/backup.daily.log"
HOST=`hostname`
DATE=`date +%Y-%m-%d`
MAILADDR="sysadmin@mydomain.com"

# Clear the old daily log file
cat /dev/null > ${DAILYLOGFILE}

# Trace function for logging, don't change this
trace () {
        stamp=`date +%Y-%m-%d_%H:%M:%S`
        echo "$stamp: $*" >> ${DAILYLOGFILE}
}

# Export some ENV variables so you don't have to type anything
export AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="YOUR_SECRET_ACCESS_KEY"
export PASSPHRASE="YOUR_GPG_PASSPHRASE"

# Your GPG key
GPG_KEY=YOUR_GPG_KEY

# How long to keep backups for
OLDER_THAN="3M"

# The source of your backup
SOURCE=/

# The destination
# Note that the bucket need not exist
# but does need to be unique amongst all
# Amazon S3 users. So, choose wisely.
DEST="s3+http://your_s3_bucket_name"

FULL=
if [ $(date +%d) -eq 1 ]; then
        FULL=full
fi;

trace "Backup for local filesystem started"

trace "... removing old backups"

duplicity remove-older-than ${OLDER_THAN} ${DEST} >> ${DAILYLOGFILE} 2>&1

trace "... backing up filesystem"

duplicity \
    ${FULL} \
    --encrypt-key=${GPG_KEY} \
    --sign-key=${GPG_KEY} \
    --volsize=250 \
    --include=/vhosts \
    --include=/etc \
    --include=/home \
    --include=/root \
    --exclude=/** \
    ${SOURCE} ${DEST} >> ${DAILYLOGFILE} 2>&1

trace "Backup for local filesystem complete"
trace "------------------------------------"

# Send the daily log file by email
cat "$DAILYLOGFILE" | mail -s "Duplicity Backup Log for $HOST - $DATE" $MAILADDR

# Append the daily log file to the main log file
cat "$DAILYLOGFILE" >> $LOGFILE

# Reset the ENV variables. Don't need them sitting around
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export PASSPHRASE=

Save the script somewhere and give it an appropriate name. I saved it a /usr/bin/duplicity-backup and make sure to chmod the script to 700 - it contains some sensitive information so we don't want none privileged users to have read access to it. Run the script as a test then set it up as a daily cron job to run at an appropriate time of night when the server isn't doing much else.

Step 3 - The restore wrapper script

Clearly we need a way to restore from a backup, so use the following script to do just that:

#!/bin/bash
# Export some ENV variables so you don't have to type anything
export AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="YOUR_SECRET_ACCESS_KEY"
export PASSPHRASE="YOUR_GPG_PASSPHRASE"

# Your GPG key
GPG_KEY=YOUR_GPG_KEY

# The destination
DEST="s3+http://your_s3_bucket_name"

if [ $# -lt 3 ]; then echo "Usage $0 <date> <file> <restore-to>"; exit; fi

duplicity \
    --encrypt-key=${GPG_KEY} \
    --sign-key=${GPG_KEY} \
    --file-to-restore $2 \
    --restore-time $1 \
    ${DEST} $3

# Reset the ENV variables. Don't need them sitting around
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export PASSPHRASE= 

Again, save this file as something sensible and chmod it to 700 to prevent prying eyes. I saved it as /usr/bin/duplicity-restore but feel free to put it wherever you like.

To do a restore simply invoke the script as follows:

duplicity-restore <date> <file> <restore-to>

Some notes on usage: Paths are relative not absolute. So /home/username would be backed up as home/username. You can restore whole directories but the destination needs to exist first. For example, to restore /home/username from November 20 2008 to a local directory 'restore', doing the following would not work because ./home does not exist:

cd ~
mkdir restore
cd restore
duplicity-restore "2008-11-20" home/username home/username

However, the following would work and would restore the directory to ./username:

duplicity-restore "2008-11-20" home/username username

That's all there is to it. As mentioned I've been using this method for several months to back up a variety of servers and it works very nicely. I hope it works just as well for you too!

Credits

This solution is the combination of a couple of tips and tricks I found while trawling the web, notably from this howto at randys.org and this post over at the linode.com forums. Credit and thanks goes to the original authors - I have merely hacked their ideas together and added a few touches of my own.

If you find this useful or have any comments or questions then please respond below!

This is an official-release of Broadcom's IEEE 802.11a/b/g/n hybrid Linux device driver for use with Broadcom's BCM4311-, BCM4312-, BCM4321-, and BCM4322-based hardware. This driver also supports the incorrectly identified BCM4328 chipset which is actually a BCM4321 or BCM4322 chipset.

Previously I explained how to build the Broadcom STA driver from source but now the installation and updates can all be taken care of using yum and the rpmfusion non-free repository. Just follow these two simple steps:

1) Enable the rpmfusion non-free repository.

The broadcom-wl and wl-kmod RPMs that we need are in the rpmfusion non-free repository which also requires the rpmfusion free repository. To enable these repos in Fedora simply do this:

su -c 'rpm -Uvh http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-stable.noarch.rpm http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-stable.noarch.rpm'

2) Update and install the driver package

Now that you have the appropriate repository enabled, to install the driver package we first ensure that we have the latest updates and then install the "broadcom-wl" package which will bring with it the required "kmod-wl" package:

su -
yum update
yum install broadcom-wl

3) Make a couple of adjustments

At this stage you should reboot and provided you have Network Manager running (default setting for F10) it should detect your Broadcom wireless device and you should be able to connect to your wireless network.

If you're having problems then it is likely some conflict between Network Manager and the network service. To ensure that Network Manager can use the wireless go to System > Administration > Network and select the wireless device (if it's not there then this doesn't apply to you). Edit the device and check "Controlled by Network Manager" and "Activate device when computer starts". Close Network Configuration, save changes and then reboot.

You should also read the license and readme which are located in /usr/share/doc/broadcom-wl-5.10.27.6/

4) Enjoy your wireless!

That's all there is to it. At this stage you may need to reboot (if you didn't already) in order to enable the new driver and any new kernel that was installed during the update.

When future kernels are released a simple "yum update" command should install the new kernel and also pull in the updated Broadcom driver for that new kernel.

Update 15 November 2008: Just a note to mention that I've packaged this up into an RPM and so this driver is now available as an RPM in the rpmfusion repos for Fedora 8, 9 and 10.

See this post for instructions of how to install using the RPM version (much easier!).

Update 26 January 2009: These instructions are now fairly outdated. The latest releases of the broadcom driver don't require the same patches as mentioned here to make them build correctly against recent kernels. I highly recommend using the RPM installation instructions linked above, or if you require help with building the latest drivers please drop me a message or leave a comment below.

Happy, happy days! At long last, a Linux Broadcom driver for the BCM4328 chipset that doesn't require ndiswrapper and Windows drivers. For me, this is really, really huge: ndiswrapper has never worked properly with NetworkManager using WPA security but this new Broadcom driver seems bullet-proof. It is even supposed to support 802.11n standard but I can't verify that just yet.

The source packages currently available from Broadcom (version 5.10.27.6) don't build on the current Fedora 9 kernel (2.6.26.5-45) and probably won't compile on any newer kernel either. Digging around a bit I found a patch that makes the driver build successfully.

Great, but that's not the whole story: I then found that with the new driver I was unable to SSH or telnet into any remote servers - bummer. However, some more digging turned up another patch that fixes this problem. With these two patches in place the new driver really rocks. For the first time in 10 months (since I bought my MacBook) I can actually connect to WPA secured networks using NetworkManager - no more fiddling around with wpa_supplicant scripts for me!

Anyhow, here's a little how-to guide to install the new Broadcom driver in Fedora 9. Note: I'm a little unsure of which Broadcom chipsets this driver actually supports but I can confirm that it works beautifully with the BCM4328 which is standard on MacBook 3,1 and 4,1 versions.

Important note: Since writing this guide Broadcom have released an updated driver (v 5.10.27.11). The updated driver and updated patches can be downloaded here along with the original driver/patches mentioned in this guide. Adjust the instructions below according to the version you are using.

1) Preparation

Ensure you have the kernel-headers package installed for your current kernel (I presume you already have make and gcc etc installed):

su -
yum install kernel-headers

2) Patching the source yourself

If you want to patch the driver yourself, download the original driver file from the broadcom website here. Make sure to get the 32 or 64 bit version depending on your installed kernel. I've made the patches available at my Fedora 9 page so go there and grab the patches then patch the source code.

3) Using my pre-patched source (easier)

If you're not sure how to patch the source code or just can't be bothered, then go grab my ready-patched tarballs from my Fedora 9 page and untar them. Here's how:

32bit users:

wget http://www.cenolan.com/fedora9/broadcom-patched-x86_32-5.10.27.6.tar.gz
tar zxvf broadcom-patched-x86_32-5.10.27.6.tar.gz
cd broadcom-patched-x86_32-5.10.27.6/

64bit users:

wget http://www.cenolan.com/fedora9/broadcom-patched-x86_64-5.10.27.6.tar.gz
tar zxvf broadcom-patched-x86_64-5.10.27.6.tar.gz
cd broadcom-patched-x86_64-5.10.27.6/

4) Building the driver

Now we've got our patched source building the driver kernel module is easy:

make -C /lib/modules/`uname -r`/build M=`pwd`

This should create a file called wl.ko - this is the magic driver file.

5) Install the new driver

We need to copy the new kernel module to the correct location, add wlan0 as an alias for this driver so that it loads on boot and resolve the module dependencies:

su -c "cp wl.ko /lib/modules/`uname -r`/kernel/net/wireless/
su -
echo "alias wlan0 wl" >> /etc/modprobe.conf
depmod -a

6) Blacklist conflicting drivers

This new driver suffers the same conflicts as the ndiswrapper driver so we need to blacklist the b43, ssb, and bcm43xx drivers, as well as the ndiswrapper driver:

su -
echo "blacklist bcm43xx" >> /etc/modprobe.d/blacklist
echo "blacklist ssb" >> /etc/modprobe.d/blacklist
echo "blacklist b43" >> /etc/modprobe.d/blacklist
echo "blacklist ndiswrapper" >> /etc/modprobe.d/blacklist

7) Remove the old module and activate the new one

su -
rmmod bcm43xx; rmmod b43; rmmod b43legacy; rmmod ndiswrapper
modprobe ieee80211_crypt_tkip; modprobe wl

8) All done!

Reboot, or restart NetworkManager and you've hopefully got the new driver installed.

Say hello to faster and more robust connections, and say goodbye to ndiswrapper and crappy badly supported Windows drivers!

References

Broadcom 802.11 Linux STA driver at FedoraForum.org

Broadcom 802.11 Linux STA driver at broadcom.com

*OFFICIAL* Broadcom Linux driver BCM4312 at jomcode.com

Broadcom driver for 4328 b/g/n wireless device at ubuntuforums.com

Bug #259816 wl: telnet/ssh connections blocked when going through NAT to external sites at launchpad.net

Fedora 9 fixes and files for MacBook 3,1 and 4,1 (Santa Rosa)