A hard link problem

[Cameron Simpson]

On 14Apr2009 21:30, mark <m.roth2006 at rcn.com> wrote:
| Cameron Simpson wrote:
| > On 14Apr2009 10:34, m.roth2006 at rcn.com <m.roth2006 at rcn.com> wrote:
| > | We're backing up using rsync and hard links. The problem is that the
| > | fs is filling up *fast*.
| > 
| > Is this the usual scheme of:
| >   - hard link previous backup tree to new backup tree
| >   - rsync to new backup tree
| > ?
| > (I gather you can nearly do that in one rsync step these days.)
| > Anyway, is that what you mean?
| 
| You can do it in one step with rsync these days. However, there's still too
| much.

Have you checked if there are files whose content is the same but have
changes metadata (mtime, ownership, etc)? You could run an md5sum over
the system looking for unlinked duplicates. Hardlinking such things can
regain a lot of space sometimes, though it damages the integrity of the
archive (two identical files with differering metadata now have the same
metadata, you'll get files hard linked in restores if you ever restore
preserving hard links, etc). For some data this tradeoff is acceptable.
If it is I have an efficient hardlinking program I can send you (Linux
ships with an awful script with horrible n^2 performance - avoid it!)

| I'm in the process, probably successfully, of convincing my boss to let
| me do compressed tar backups of the rsync'd directories, with
| beginning-of-month and weekly fulls, and the rest incrementals.

I presume keeping a bunch of hardlinked backups and tarring the older
stuff? That way you have easy restore, partial restore and browsing.

I also have a script for pruning forests of this kind of backup; it loops
(optional) removing the oldest backup until sufficent free space is
available, keeping a specifiable minimum number of backups around. You
may find it useful, or adapt it.

Cheers,
-- 
Cameron Simpson <cs at zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/