Poor Performance WhenNumber of Files > 1M

Sean McCauliff smccauliff at mail.arc.nasa.gov
Thu Aug 9 19:02:31 UTC 2007


dumpe2fs reports that dir_index option is enabled.  But thank you for 
the suggestion.

Sean

Bruno Wolff III wrote:
> On Wed, Aug 01, 2007 at 18:55:53 -0700,
>   Sean McCauliff <smccauliff at mail.arc.nasa.gov> wrote:
>> Hi all,
>>
>> I plan on having about 100M files totaling about 8.5TiBytes.   To see 
>> how ext3 would perform with large numbers of files I've written a test 
>> program which creates a configurable number of files into a configurable 
>> number of directories, reads from those files, lists them and then 
>> deletes them.  Even up to 1M files ext3 seems to perform well and scale 
>> linearly; the time to execute the program on 1M files is about double 
>> the time it takes it to execute on .5M files.  But past 1M files it 
>> seems to have n^2 scalability.  Test details appear below.
>>
>> Looking at the various options for ext3 nothing jumps out as the obvious 
>> one to use to improve performance.
>>
>> Any recommendations?
> 
> Did you make sure directory indexing is available? I think that is the
> default now for ext3, but maybe it wasn't turned on for your test.
> 




More information about the Ext3-users mailing list