Duplicity to dreamhost and Amazon S3

January 28th, 2007

Howto use duplicity to back to accounts and Amazon S3:


Dreamhost has some amazing offerings, one of which is a huge amount of storage, that increases over time! Using duplicity there is easy because you can simply ssh, or even better: scp, into it.

From the manual page of duplicity (i.e. "man duplicity") :

duplicity --full /home/me scp://

Put your username in where it has "uid", making sure you have shell access at dreamhost for that user. Put your domain name which you host at dreamhost, such as Make sure whatever directory you choose exists in your home directory at dreamhost. I chose duplicity_backups so it would be easy to recognize.

Duplicity will then ask you for a gpg passphrase. Make it a good one. Since the first full backup will take awhile, I started the session by typing "screen", that way I could exit the shell with CNTL-a-d, and go on to other things. To return to that terminal session, just type "screen -r".

Strange thing though is the low throughput rates. On a tier-1 backbone connection, I'm only getting top speeds of 50KB/s, which is only about 400kbps. What gives? I should be at least getting two to three times that speed. Oh well, perhaps does some ingress throttling to dampen any adverse network latency effects.

Word to the wise: Start small, if during your initial "first" backup, the transfer fails, you will have to start all over. To me that it is rediculous, but I guess its the cost of encryption.

Amazon S3

This isn't so easy. I'm trying duplicity with BitBucket, with directions from here:

But I'm getting these errors when I try to run duplicity-bin.

ImportError: No module named _librsync

Ugh that's just not going to work.

OK, found this in CVS-Readme:

CVS README - Notes for people checking out of CVS-------------------------------------------------

Getting duplicity to run:-------------------------

If you want to run a version of duplicity checked out of CVS into your$DUP_ROOT directory, change to $DUP_ROOT/duplicity and run the./ file.  With any luck, a library will appear

in that directory.  Then run duplicity-bin, making sure that all the

files are in your PYTHONPATH:

PYTHONPATH=$DUP_ROOT duplicity-bin



Running the unit tests:-----------------------

If you want to try some of tests, you first have to untar the

testfiles.tar.gz as root (the tarball contains device files, files

with various uid/gid, etc):

cd testing; tar -xvzf testfiles.tar.gz

Then run the various * files, for instance:

cd testing; python

Except now I'm getting: error: command 'gcc' failed with exit status 1

Argh. OK so I couldn't get to compile, but I installed rdiff_backup with:

apt-get install rdiff_backup

and then copied the to the duplicity src directory.

cp /var/lib/python-support/python2.4/rdiff_backup/ duplicity

Then got the bitbucket:


tar -xzvf BitBucket-0.4a.tar.gz

cd BitBucket-0.4a

python install

Holy cow - it works! Gosh I do wish I could get it to show me how quickly the files are loading.


Yearly Indexes: 2003 2004 2006 2007 2008 2009 2010 2011 2012 2013 2015 2019 2020 2022