spd_readdir.c and readdir_r [real new version]

[Ross Boylan]

My previous attachment had only a link for the main file; the current
one should have the real thing.

For the full backup, using the preload library changed the backup time
from  over 35 hours to 22 hours for a full backup.  The full backup got
much slower as it progressed; my guess is something other than the
preload library (perhaps the snapshotting itself, bacula, or postgresql)
accounts for that.

The percentage change for incremental backups, which involve relatively
more time scanning, is larger: from 3 hours to under .5 hours.

There's no obvious speedup for the jobs involving Reiser filesystems.

All in all, a big win.  Thanks to everyone for your help, and especially
to Ted for the original code.

Ross Boylan

On Sun, 2008-06-08 at 21:26 -0700, Ross Boylan wrote:
> I've attached a modified version of Ted's spd_readdir.c that adds
> support for readdir_r and readdir64_r.  It appears to be working
> (readdir64_r is the only new routine getting exercised), but should be
> taken as a rough cut.  I also added a Makefile and a test program.
> 
> It also looks as if this is giving me a huge speed improvement (at least
> x4) of my backups of my ext3 partitions.  I'll try to report after a
> full and incremental backup complete, which will be a couple of days.
> 
> Originally I tried taking the threading code from the system
> implementations of the original readdir_r.  When that didn't work (since
> it was designed to be part of a libc build) I switched to pthreads.  I
> don't know if recursive locking is essential; I activated it at one
> point while trying to get things to work.
> 
> For big directories this code could use quite a lot of memory.  It
> allows an optional max size, beyond which it reverts to the original
> system calls.  I wonder if instead taking large directories in chunks
> would preserve much of the speedup while putting a bound on memory use.
> 
> Ross Boylan
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: RBspd_dir.tgz
Type: application/x-compressed-tar
Size: 3147 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/ext3-users/attachments/20080615/d814b0fa/attachment.bin>