Max Files Per Directory
Tony Nelson
tonynelson at georgeanelson.com
Sat Jul 16 18:22:51 UTC 2005
At 8:53 AM -0400 7/16/05, Matthew Miller wrote:
>On Sat, Jul 16, 2005 at 08:04:22AM -0400, fredex wrote:
>> Now, for performance reasons, it is often not a good idea ot have
>> many thousands of files in a single directory. As the number of files
>> grows large the time it takes to access a file grows larger. I haven't
>> looked into this on any linux file system, but on other unixes I've
>> observed delays reaching up into the whole-second region when many thousands
>> of files are in a single directory.
>
>Shouldn't be a problem on a new install of ext3 on modern Linux. See
><http://lwn.net/Articles/11481/>.
I suppose the simple test is to try it. Time (microseconds) per file to run
a script that creates files by touching them, to ls a few near the end, and
to remove all the files; see script at end of transcript.
#files touch ls rm
------ ----- ----- -----
1000 309 016 054
10000 331 008 051
100000 326 007 357
200000 330 007 1065
I didn't want to try a million files. Creation and listing seem to be
linear with the number of files, but rm seems quadratic. I think this
indicates that in my current 2.6.12 FC3 kernel using Ext3 the directory
data structure is still a list and not a tree.
I expect that some sort of directory hierarchy to limit the number of files
per directory would still be a win.
Transcript follows. Max line width 114 chars.
---------- cut here ----------
[tonyn at localhost ~]$ time ./manyfilestest 1000
real 0m0.309s
user 0m0.066s
sys 0m0.237s
[tonyn at localhost ~]$ time ls manyfiles/99*
manyfiles/99 manyfiles/991 manyfiles/993 manyfiles/995 manyfiles/997 manyfiles/999
manyfiles/990 manyfiles/992 manyfiles/994 manyfiles/996 manyfiles/998
real 0m0.016s
user 0m0.007s
sys 0m0.004s
[tonyn at localhost ~]$ time rm -rf manyfiles
real 0m0.054s
user 0m0.000s
sys 0m0.049s
[tonyn at localhost ~]$ time ./manyfilestest 10000
real 0m3.312s
user 0m0.728s
sys 0m2.518s
[tonyn at localhost ~]$ time ls manyfiles/999*
manyfiles/999 manyfiles/9991 manyfiles/9993 manyfiles/9995 manyfiles/9997 manyfiles/9999
manyfiles/9990 manyfiles/9992 manyfiles/9994 manyfiles/9996 manyfiles/9998
real 0m0.075s
user 0m0.051s
sys 0m0.021s
[tonyn at localhost ~]$ time rm -rf manyfiles
real 0m0.519s
user 0m0.006s
sys 0m0.501s
[tonyn at localhost ~]$ time ./manyfilestest 100000
real 0m32.561s
user 0m7.494s
sys 0m24.285s
[tonyn at localhost ~]$ time ls manyfiles/9999*
manyfiles/9999 manyfiles/99991 manyfiles/99993 manyfiles/99995 manyfiles/99997 manyfiles/99999
manyfiles/99990 manyfiles/99992 manyfiles/99994 manyfiles/99996 manyfiles/99998
real 0m0.686s
user 0m0.513s
sys 0m0.162s
[tonyn at localhost ~]$ time rm -rf manyfiles
real 0m35.653s
user 0m0.082s
sys 0m5.561s
[tonyn at localhost ~]$ time ./manyfilestest 200000
real 1m6.031s
user 0m15.243s
sys 0m47.962s
[tonyn at localhost ~]$ time ls manyfiles/9999*
manyfiles/9999 manyfiles/99991 manyfiles/99993 manyfiles/99995 manyfiles/99997 manyfiles/99999
manyfiles/99990 manyfiles/99992 manyfiles/99994 manyfiles/99996 manyfiles/99998
real 0m1.459s
user 0m1.113s
sys 0m0.269s
[tonyn at localhost ~]$ time rm -rf manyfiles
real 3m32.851s
user 0m0.159s
sys 0m12.205s
[tonyn at localhost ~]$ cat manyfilestest
#!/bin/bash
# Make lots of files in a directory.
TESTDIR="manyfiles"
MAXFILES=${1:-100000}
mkdir $TESTDIR
cd $TESTDIR
for (( i=0; i*10<$MAXFILES; i=i+1 )) ; do touch ${i}0 ${i}1 ${i}2 ${i}3 ${i}4 ${i}5 ${i}6 ${i}7 ${i}8 ${i}9 ; done
[tonyn at localhost ~]$
---------- cut here ----------
____________________________________________________________________
TonyN.:' <mailto:tonynelson at georgeanelson.com>
' <http://www.georgeanelson.com/>
More information about the fedora-list
mailing list