The lost art of benchmarking was Re: Reiser4

[Jonathan Corbet]

Leonard den Ottolander <leonard at den.ottolander.nl> wrote:

> > Because I benchmarked an actual task that my readers are likely to
> > perform?  
> 
> No, because if you benchmark a file system you shouldn't benchmark a CPU
> intensive task like compiling a kernel as file system performance will
> only have a relatively small impact on the compilation time
> (signal/noise). 

A change from 1400 seconds (ext3) to 1583 (reiser4) strikes me as more
than noise.  If I run reiser4 on that particular system, I have to wait
three more minutes to get my kernel built.  (Yes, it drops to 1445
seconds if you happen to have a kernel tarball built in the
reiser4-optimum order, but that is worth noting too).

Reiser4 is a CPU-heavy filesystem - by admission of its own developers.
So its performance will certainly suffer, relative to lighter
filesystems, when executing a CPU-intensive, filesystem-intensive task.
Running benchmarks which do not load the processor can be useful for
generating those "guaranteed not to exceed" numbers, but they mask the
extra cost involved in using a filesystem which requires a lot of
processor time to operate.

If I wanted to print best-case-scenario benchmarks, I could have simply
copied them from Namesys.com.  I think there is also a place for "real
world" tests.

jon