compression tools

[Jon Shorie]

> On Wednesday, Jun 22nd 2005 at 19:42 +0200, quoth Gérard Milmeister:
>
> =>On Wed, 2005-06-22 at 13:25 -0400, Scot L. Harris wrote:
> =>> On Wed, 2005-06-22 at 12:54, sharif islam wrote:
> =>> > Is there any compression mechanism that's rated for tar files as big
> =>> > as, say, 100GB?
> =>>
> =>> I don't believe anyone rates compression programs based the size of the
> =>> file to be compressed.
Not actually true.  Older versions of gzip (such as the one that came with 
Solaris 8) could only work with tar archives (or any file) that was <2GB.  

With new versions this is not the case.

> =>
> =>For files of that size, it is probably useful for compress program to be
> =>fast. Bzip2 for example would take ages to compress 100GB. Gzip with a
> =>low compress degree (< 5) would be more useful.
>
> I think you'll find that gzip and bzip2 will give very similiar results
> both in time and space by appropriately lowering the compression factor.
>
> e.g., bzip2 -1 and gzip -1 will be very comparable.
>
> The point is that the larger the chunk that any compression program has to
> operate upon the more compression you're going to end up with and the
> greater the time requirement will be for the chunk.

I have found that when compressing large files Gzip seems to be faster with 
default options especially if the original is on an NFS mount.

We have a Linux backup server that backs up all of our other servers nightly.

We use gzip in a command such as the following:

tar cvf /backup/alpha/payroll/monday.tar.gz /export/alpha/payroll/* --gzip

To back up our payroll program that is running on an Alpha with Redhat 7.1.

On some of our larger archives we build the tar.gz on the server to be backed 
up and then just copy over the file to the backup server.

cp /export/penuing/backup/websites.tar.gz /backup/penguin/monday.tar.gz

We found that when backing up our Solaris 8 box we had to use

tar cvf /backups/homedirectories.tar /export/home/*
compress /backups/homedirectories.tar

and then on our backup server

cp /export/sunbox/backup/homedirectories.tar.Z /backup/sunbox/monday.tar.Z

because the solaris 8 version of gzip would only work with archives < 2GB 
although it is usually more efficient than compress for archives that it will 
work on.

Jon Shorie
Medina County Sanitary Engineers