Poor Performance WhenNumber of Files > 1M
smccauliff at mail.arc.nasa.gov
Thu Aug 9 19:02:31 UTC 2007
dumpe2fs reports that dir_index option is enabled. But thank you for
Bruno Wolff III wrote:
> On Wed, Aug 01, 2007 at 18:55:53 -0700,
> Sean McCauliff <smccauliff at mail.arc.nasa.gov> wrote:
>> Hi all,
>> I plan on having about 100M files totaling about 8.5TiBytes. To see
>> how ext3 would perform with large numbers of files I've written a test
>> program which creates a configurable number of files into a configurable
>> number of directories, reads from those files, lists them and then
>> deletes them. Even up to 1M files ext3 seems to perform well and scale
>> linearly; the time to execute the program on 1M files is about double
>> the time it takes it to execute on .5M files. But past 1M files it
>> seems to have n^2 scalability. Test details appear below.
>> Looking at the various options for ext3 nothing jumps out as the obvious
>> one to use to improve performance.
>> Any recommendations?
> Did you make sure directory indexing is available? I think that is the
> default now for ext3, but maybe it wasn't turned on for your test.
More information about the Ext3-users