The lost art of benchmarking was Re: Reiser4

[Jean Francois Martinez]

Le lun 20/09/2004 à 16:31, Jonathan Corbet a écrit :
> Jean Francois Martinez <jfm512 at free.fr> wrote:
> 
> > First of all it looks like the guy divided his 4GB disk in two 2G
> > partitions and installed Reiser on one partition and EXT3 in the
> > other.  WRONG!  Modern disks have more sectors in the outer, 
> > lower numbered tracks. Meaning that you will get higher throughput 
> > from hda1 than from hda2.
> 
> Wrong indeed.  The same partition was used for all tests.  One can give
> me a little credit, even if it hurts.
> 

Sorry, when I saw you used a 2G partition on a 4G disk I thought you
had cut in two halves, one for Reiser, the other one for EXT3. 

> > He also compiled the kernel as a benchmark.  99% of times this is the
> > touchstone for lack of proper thinking in benchmarking.  
> 
> Because I benchmarked an actual task that my readers are likely to
> perform?  
> 


The other day I had a discussion with the sysadmin of
another department and our common position was:  1)  The state of
the art in kernel building (ie use of modules and cpu detection by
the installer) allows vendors to ship kernels whose
performance is close enough to a custom built one the user should
not need to bother with recompiling.  2) If the kernel you get out
of the box is an underperformer then it is a _bug_.  And we both
refuse to fix the bad work of a vendor: if he doesn't fix the bug
then he is out.  So here you have two persons who administer several
Linux and Unix boxes who never compile kernels but who run databases
(both free and proprietary).



> Look, the benchmarks in the LWN articles were a low-value exercise, just
> to get a rough feel for the performance benefits claimed by reiser4.  I
> believe I succeeded in that goal.  Among other things, I turned up the
> fact that, if you fail to create your files in the right order,
> performance suffers significantly - something which the Namesys
> marketing literature somehow failed to mention.
> 

That is good.

Now forgive me a former statistician who tends to be pedantic (and
harsh) about benchmarks.


 

> jon
> 
> Jonathan Corbet
> Executive editor, LWN.net
> corbet at lwn.net
>