The lost art of benchmarking was Re: Reiser4

[Jean Francois Martinez]

I have taken a quick look at the first article and I have a BIG
problem with the benchmark part.

First of all it looks like the guy divided his 4GB disk in two 2G
partitions and installed Reiser on one partition and EXT3 in the
other.  WRONG!  Modern disks have more sectors in the outer, 
lower numbered tracks. Meaning that you will get higher throughput 
from hda1 than from hda2.

Second.  I hope he stopped all daemons and performed his tests
in single mode.

He also compiled the kernel as a benchmark.  99% of times this is the
touchstone for lack of proper thinking in benchmarking.  Compiling a
kernel is a terrible benchmark for a filesystem: first, because even on
a machine with a  2000+ Athlon over 50% of the time will be spent
crunching gcc code, the rest is kernel time (a good part of it
not being FS related), and waiting for I/O (that is FS-related)
so we can guess only about 25 or 30% of total time is influenced by
FS, and you get "noise" from the other 70%.  But on slower boxes
like the one he used the relation from CPU (ie mostly gcc) to total
time will be still higher

But that is the not the sole reason why compiling a kernel is a poor
benchmark: kernel source is a bunch of small files packed into
moderately sized directories and its compiling it is only about
_sequential_ access.   What about the performance on huge files?  What
about direct access like needed for databases?  What about huge
directories?  Compiling the kernel will teach you nothing about these.
Same thing for grepping through  the source code (but at least there
is lesser user-mode cpu time)


In addition a filesystem could be better at reading than at writing,
when partition is nearly empty or when nearly full, when partition has
been recently formatted or when it has had lots of insertions and
deletions.  Where are the tests for measuring those aspects of
performance?

Making a real benchmark involves ensuring that no external factors come
to distort the resullts (eg using the same partition for both
filesystems), that the tests are significative (ie they spent 90% of 
the time exerting what we are trying to measure) and that all important
factors of performance are measured and withted for final evaluation
 
 


Le dim 19/09/2004 à 00:19, Matias Feliciano a écrit :
> Le sam 18/09/2004 à 18:03, Manu Abraham a écrit :
> > On Sat September 18 2004 2:07 am, Steve Bergman wrote:
> > > On Thu, 2004-09-16 at 21:33 +0400, Manu Abraham wrote:
> > > > Hi,
> > > > 		Reiser4 has been officially released and considered stable by Mr. Hans
> > > > Reiser.
> > >
> > > Just out of curiosity, why would you want reiser4 in FC3?
> > 
> > 		Reiser4 sounds cool, that's why.
> > >
> > > It's not ready.  It's unclear as to whether its concepts are even a good
> > > idea, despite the Namesys marketing propaganda.
> > 		what makes one think so ?
> > 
> 
> http://lwn.net/Articles/99232/
> http://lwn.net/Articles/100148/
> 
> > Manu 
> > >
> > > -Steve Bergman
> > 
> > 
> 
> ______________________________________________________________________
> -- 
> fedora-test-list mailing list
> fedora-test-list at redhat.com
> To unsubscribe: 
> http://www.redhat.com/mailman/listinfo/fedora-test-list