[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: rsyncing 79gb of data to 250gb drive


> > At 15:20 5/28/2004, Michael Mansour wrote:
> > >Not only do I wonder if tar will respond the same
> way
> > >as rsync without the -S option, but I'm also
> wondering
> > >if backup software which use rsync to perform
> backups
> > >would also take this into account...
> > 
> > I also wonder if there is any downside to using -S
> or whether it might 
> > produce an undesirable result in some cases. If
> not, perhaps one should 
> > always use it just in case?
> Sparse files are a boon or a bust.  
> If you make a sparse file 
>   open(); write(byte); lseek(way-out-there);
> write(byte); close()
> you have not reserved or allocated the disk blocks
> for all the
> possible stuff in the middle.  If you expect those
> blocks to be real
> and there is not enough disk space in the future
> things will bust.
> In some cases if you compress the file then expand
> it you can run
> out of disk space on the uncompress pass (compress,
> gzip, bzip2, zip).
> Watch out for uncompress bombs.
> Some backup and restore tools will run you out of
> tape or out of disk
> or both with sparse files. Gnu tar has a --sparse
> flag.  You might not
> be able to restore the contents of a file system if
> you are not
> cautious/ aware.
> Some data base files might be sparse but you could
> run out of space
> a year later when you do not expect and need the
> blocks to be real.
> mkswap (to files) might generate files that look
> like they should be
> sparse to backup and restore tools but should not
> be. i.e. You should
> not use swapon on a file with holes.
> Holes on a file system with one size of allocation
> block may not be
> holes on another file system with a smaller
> allocation block size.
> Since allocation block size is  heuristically
> determined one could
> be confused when moving files from one FS to another
> of a different 
> size or type.  mv does not have a --sparse flag that
> I can see.
> The big issue is backup and restore processes
> (including cp, mv).
> I guess a utility (holeify and holecount) might be
> written.
> -- 
> 	T o m  M i t c h e l l 
> 	/dev/null the ultimate in secure storage.

Thanks Tom, excellent explanation.

In regards to where these sparse files came from, I
set about to determine this and found out.

As mentioned in earlier posts, I use that data volume
under Samba, which is shared out to apps/clients to
store their files.

Azureus is a Java BitTorrent client, and when starting
torrent downloads on that data drive, the entire size
of the torrent is reserved on the data drive (so a
600Mb ISO shows up on the filesystem even though it is
empty). A "du -ks" shows the actual size (a few bytes
at the most), although an "ls -l" shows the 600Mb file
which the filesystem recognises as space taken.

So when I rsynced that data volume to another drive
(without the -S option), the rsync naturally copied
exactly what it saw, spacewise, but when using -S it
recognised the sparse files and therefore didn't take
up the space for them.

We learn something everyday :) ... I'm wondering now
if those sparse files that got "un-sparsed" on the
rsync will still be valid if, for example, Azureus was
to continue downloading to them.


Find local movie times and trailers on Yahoo! Movies.

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]