[K12OSN] Re: automating network-based backups (Matt Oquist)

[Matt Oquist]

> I wrote most of this up for Steve, Art, and Josh after I did it on
> their servers, but after talking with Dave Trask today, I thought this
> might be useful for a wider audience.
> 
> I'm including the document here, but it is also available at
> http://majen.net/docs/network-backups.txt
> 
> I hope it's useful for someone.

Or, perhaps I *wasn't* including the document there... *blush*

--matt

---------------------------------------------------
How-To automate network based backups between two Linux systems
http://majen.net/docs/network-backups.txt
<moquist-howto at majen.net>

This document explains how to set up automated backups of any amount
of data (for example, only /home, or even the whole system if you
want) from one Linux system to another one.

This document will refer to "system1" and "system2".  You should
substitute names/IP addresses of systems on your own network
accordingly.

1. Create a passwordless public/private ssh key pair:
on system1:
$ cd .ssh
$ ssh-keygen -t dsa -f to_system2
  ==> Just hit return when it asks for a password/passphrase - you
      don't want one, or you want a blank one.  Whatever.
  ==> The key name I used, "to_system2", is only a convention I like.
      You can use any naming scheme that suits you.
$ scp to_system2.pub root at system2:.ssh/

on system2:
$ cd .ssh
$ echo \# to_system2.pub, for copying files to here from system1 >> authorized_keys
$ cat to_system2.pub >> authorized_keys
 ==> 'man sshd' and search for "AUTHORIZED_KEYS" for details

on system1:
$ ssh -i ~/.ssh/to_system2 system2
  ==> If this doesn't log you in without a password, you've got
      a problem and you need to figure it out before proceeding.
      ==> 'man sshd' (and search for "AUTHORIZED_KEYS")
      ==> 'man sshd_config'
      ==> 'man ssh'

on system1:
$ crontab -e
  ==> Enter the date/time when you want your backup to start.
  ==> Enter the command you want to run to do your backup.
      ==> Either 'rsync -a -e ssh' or 'tar ... | ssh' can work.
  ==> See 'man 5 crontab' for the file format.
  ==> Here is an example from ltsp.gbecs.org:
####################################################
0 0 * * * /usr/bin/rsync -avz -e "ssh -i /root/.ssh/gbecs_backup" --delete --exclude=/proc --exclude=/export --exclude=/backup --exclude=/sys / apps:/backup/ltsp/ 2>&1 >> /backup/ltsp.log
####################################################
  Explanation:
  - 0 0 * * *: This tells the cron daemon to start this job at midnight every day.
  - We're running the rsync command with -a (archive), -v (verbose), -z (zip),
    and -e (which we tell to connect via ssh, using the private key
    /root/.ssh/gbecs_backup).
  - We tell rsync to --delete any files on the destination that don't exist on the source.
  - We tell rsync NOT to copy anything from /proc, /export, /backup, or /sys on the source.
    (/proc and /sys are runtime filesystems that you don't need to back up, and
     they'll complain if you try to copy them, anyway.
     On our example system, /export and /backup have things being
     automatically copied into them from cron-jobs just like this one,
     and you don't want stuff copied by cron to be copied by cron to
     be copied by cron to be copied by cron...)
  - We tell rsync to copy everything from / on the local system (the
    SOURCE) to /backup/ltsp/ on 'apps' (the DESTINATION).
  - We tell the shell (bash, which executes our cron commands) to
    redirect the stderr (Standard Error) output into stdout (Standard
    Output), and we >> (append) that output to /backup/ltsp.log on the
    local system.
  - The End.

  ==> Here is almost the same thing (but without an equivalent to
      "--delete"), using 'tar' instead of 'rsync':
####################################################
0 0 * * * /bin/tar -czf - --exclude=/proc --exclude=/export --exclude=/backup --exclude=/sys / 2>> /backup/ltsp.log | ssh -i /root/.ssh/gbecs_backup apps "cd /backup/ltsp/; tar -xzvf - >> /backup/ltsp.log 2>&1"
####################################################
  - We're telling the 'tar' command to create an archive (-c), zip
    that archive (-z), and put the archive in the file (-f) "-".  The
    single hyphen is a special "filename" that means "Standard
    Output", i.e., print the archive to the screen.  
    (Try running "tar cf - /tmp/" in a fresh terminal and see what
    happens.  When it's done, type 'reset' and hit return.  :)
  - BUT, instead of printing that output to the screen, we're piping
    that output into an ssh process which we're telling to connect to
    the machine named 'apps', using the private key gbecs_backup.
  - Any errors printed by 'tar' on the SOURCE system will be appended
    (">>") to the file /backup/ltsp.log in the SOURCE system.
  - After the ssh process connects, it will run the command that
    starts with 'cd'.
    - It will Change Directory into /backup/ltsp/ on the remote
      system.
    - It will start the 'tar' command.
    - 'tar' will eXtract (-x) the input, unZip (-z) the input,
      Verbosely print information about the input (-v), and listen for
      input from Standard Input.  (In the eXtraction context, the
      special "filename" denoted by the single hyphen tells tar to
      listent to Standard Input, just as in the archive Creation
      context the same single hyphen told tar to output to Standard
      Output.)
    - Anything printed by the DESTINATION 'tar' command to Standard
      Error will be appended (">>") to the file /backup/ltsp.log on
      the DESTINATION system ("apps").

- Note that if you've never logged into system2 from system1 before,
  ssh will want you to type in that "yes", the remote system's key is
  ok.  If you don't do that before the cron job tries to run, the
  login will fail and no backup will happen.

Misc. Sorta-Related Goodies:
- The 'dd' command reads and write binary data.  So imagine that you
  have a CD (Fedora Core 3 disc 1) in your CDROM drive, and you would
  like to copy its ISO to your Linux fileserver, which is named
  "moochie".  You could do this slick little command:
  $ dd if=/dev/cdrom bs=1M | ssh root at moochie "cd /export/isos/; dd of=fedora-core-3-cd1.iso"
- Or maybe you'd like to copy all the files from a CD onto your
  fileserver, maybe so you can do NFS installs.  But you don't have
  the disc burned, you only have the ISO...
  $ mkdir /mnt/tmp
  $ mkdir /export/fc3/
  $ mount -t iso9660 -oloop /export/isos/fc3/cd1.iso /mnt/tmp
  $ rsync -avu /mnt/tmp/ /export/fc3

$Id: network-backups.txt,v 1.2 2005/02/05 20:21:10 moquist Exp moquist $