Re: ext3 performance issue with a Berkeley db application

[Andrew Morton <akpm digeo com>]

Greg Louis <glouis dynamicro on ca> wrote:
>
> Here's the ext3 part.  On another machine I got the following times to
> build that list of 530,000 tokens, starting by creating empty db files
> on a partition mounted as:

How large are the generated output files?

> ...
> Some further findings:
> 
> o  It doesn't matter if the source (mbox) file is on ext3/ordered or
>    ext2; the difference in time is insignificant.
> 
> o  Just processing a message to classify it takes about four times as
>    long if the .db files are on ext3/ordered as it does if the .db
>    files are on ext2.

I would be suspecting that the database is opening the files with O_SYNC or
is running fsync or such.  Maybe.

> o  Dumping the tokens and counts from the database in text form and
>    reloading them into a new database file is not subject to serious
>    performance problems; on the machine that needs 24 minutes to
>    build from a 200-Mb mbox, rebuilding the database from a list of
>    tokens took eight seconds -- this was on ext3 in ordered mode.

How does this operation differ from the operation which is "slow"?


> These data were obtained on machines running linux kernels
> 2.4.21-pre3-ac4 and -ac5 and 2.4.21-pre4-ac1; kernel 2.4.20-ac2 appears
> to give similar results though this has not been thorougly tested. 
> Results like those reported were initially obtained with db-3.1.17; the
> tests shown here used db-4.1.25.
> 
> More info available on request; tuning hints most gratefully received
> and tested.

If you can suggest an easy way in which I can reproduce this, that would be
efficient.