Poor Performance WhenNumber of Files > 1M

Sean McCauliff smccauliff at mail.arc.nasa.gov
Thu Aug 2 01:55:53 UTC 2007

Hi all,

I plan on having about 100M files totaling about 8.5TiBytes.   To see 
how ext3 would perform with large numbers of files I've written a test 
program which creates a configurable number of files into a configurable 
number of directories, reads from those files, lists them and then 
deletes them.  Even up to 1M files ext3 seems to perform well and scale 
linearly; the time to execute the program on 1M files is about double 
the time it takes it to execute on .5M files.  But past 1M files it 
seems to have n^2 scalability.  Test details appear below.

Looking at the various options for ext3 nothing jumps out as the obvious 
one to use to improve performance.

Any recommendations?


Parameter one is number of files, parameter two is number of directories 
to write into.

Dell MD3000 + LVM2. 2x7 10k rpm SAS disks 128k stripe RAID-0 for 3.8 
TiBytes of total storage.  Fedora Core 6 x86_64.  2xQuad Core Xeon. 
Default mount and ext3 options used.

[root at galaxy filestore]# time /soc/abyss/test/fileSystemTest.pl 10000 1000

real    0m1.054s
user    0m0.128s
sys     0m0.382s

[root at galaxy filestore]# time /soc/abyss/test/fileSystemTest.pl 1000000 1000

real    1m0.938s
user    0m12.203s
sys     0m40.358s
[root at galaxy filestore]# time /soc/abyss/test/fileSystemTest.pl 10000000 

real    13m39.881s
user    2m6.645s
sys     7m26.665s
[root at galaxy filestore]# time /soc/abyss/test/fileSystemTest.pl 20000000 

real    44m46.359s
user    4m22.911s
sys     17m2.792s

More information about the Ext3-users mailing list