Using rsync for synch and backup

I’m using rsync to keep my documents synchronized between the three computers I use often, and a remote host:

The remote host serves as an intermediary, since firewalls prevent me from connecting directly from my home workstations to my office computer. Rsync is an effective choice because not only does it only send files which have changed, but it optimizes transfers by sending only the differences between files. Since I work with quite a few .text and .xhtml documents, this is very efficient.

Rsync doesn’t do two way synchronization, so I have to keep track of which device has the most recent content manually. In practice, this is not a big deal; I just start and end each work session by synching from and to wrecking.org. I just have to make sure I pick the right command when I do so.

Method

I keep all my stuff in a folder called Documents my home directory on all three workstations (~/Documents). On wrecking I have a folder named ’cbdox-200704’ which is my backup target, and a symlink pointing to it called ’cbdox’. (I’ll explain why below.)

On Linux, I run this command to test a sync from any of the three workstations to wrecking.org:

rsync -n -av -e ssh --progress --delete-after ~/Documents/ wrecking@wrecking.org:cbdox

Exploded, here’s what that means:

-n
run rsync in test mode---don’t actually do the transfer
-a
run in "archive" mode, preserving permissions & timestamps
-v
be verbose
-e ssh
use ssh to do the transfer
--progress
show a nice display of progress
--delete-after
delete any files which are no longer needed AFTER doing the transfer. I pick this option because it allows me to see quickly if I’m going the wrong way---if I’ve created new stuff and rsync wants to delete them...
~/Documents/
source directory
wrecking@wrecking.org:cbdox
target (in this case, a user@host with directory)

That spits out something like

building file list ... 
1943 files to consider
./
rsync-howto.html
rsync-howto~.text
s07/.DS_Store
s07/180/
s07/180/.DS_Store
s07/180/arg/arg-comments-20070419.odt
s07/180/doorsign-20070424.odt
s07/195/
s07/195/doorsign-20070424b.odt
s07/do-20070423.odt
deleting rsync-howto.text

sent 47818 bytes  received 52 bytes  3545.93 bytes/sec
total size is 561898810  speedup is 11738.02

Once I’m sure I’m going the right way, I drop off the "-n" to do the actual sync, which will show the transfer itself:

building file list ... 
1943 files to consider
./
rsync-howto.html
        4749 100%    0.00kB/s    0:00:00  (1, 63.6% of 1943)
rsync-howto~.text
        3276 100%  290.84kB/s    0:00:00  (2, 63.7% of 1943)
s07/.DS_Store
       15364 100%  577.07kB/s    0:00:00  (3, 65.2% of 1943)
s07/180/
s07/180/.DS_Store
       15364 100%  107.94kB/s    0:00:00  (4, 65.3% of 1943)
s07/180/arg/arg-comments-20070419.odt
       10951 100%   67.69kB/s    0:00:00  (5, 65.6% of 1943)
s07/180/doorsign-20070424.odt
        6515 100%   40.01kB/s    0:00:00  (6, 66.6% of 1943)
s07/195/
s07/195/doorsign-20070424b.odt
        6498 100%   35.25kB/s    0:00:00  (7, 70.2% of 1943)
s07/do-20070423.odt
        6560 100%   35.39kB/s    0:00:00  (8, 81.5% of 1943)
deleting rsync-howto.text

sent 86163 bytes  received 696 bytes  8272.29 bytes/sec
total size is 561899181  speedup is 6469.10

To go the other way, from the remote server to my home computer, works like this:

rsync -av -e ssh --progress --delete-after wrecking@wrecking.org:cbdox/ ~/Documents/

Note that the remote directory has a trailing slash when it’s the source.

When I do Erin's backup, I skip the folder which contains our PVR files by adding an exclude switch:

--exclude="EyeTV*"

It's my understanding you can add multiple excludes to quickly skip files.

Versions

Most data loss comes from human error, not hardware failure---saving over the wrong file. To reduce the likelihood of this problem, I create new versions of my stuff on wrecking.org monthly (cbdox-200703, cbdox-200704, etc) and use a symlink to point to the right version. When it’s time to roll over a month, I just ssh to wrecking and do something like this:

rm cbdox
cp -a cbdox-200704/ cbdox-200705/
ln -s cbdox-200705/ cbdox

This leaves a copy of my files as of April 2007, and changes the active synch directory to "cbdox-200705". The symlink means I don’t have to change my command or remember what month it is to synch.

Future

If I was smart, I’d make aliases or shell scripts which were more human-readable: 

sync-to-wrecking --test
sync-to-wrecking
sync-from-wrecking

etc. But I haven’t done that yet. Heck, I could probably make (or find) a smart script which compared the local and remote files and suggested the right way to sync. Someday...

Also, I could use rsync once a week on wrecking.org to save files which had change in the last week, adding a second layer of protection against accidental deletion or modification.

I welcome your comments or suggestions. Back to my home page