[Linux-cluster] Why GFS is so slow? What it is waiting for?
s.wendy.cheng at gmail.com
Thu May 8 21:51:03 UTC 2008
Ja S wrote:
> Hi, All:
> I used to post this question before, but have not
> received any comments yet. Please allow me post it
> I have a subdirectory containing more than 30,000
> small files on a SAN storage (GFS1+DLM, RAID10). No
> user application knows the existence of the
> subdirectory. In other words, the subdirectory is free
> of accessing.
Short answer is to remember "ls" and "ls -la" are very different
commands. "ls" is a directory read (that reads from one single file) but
"ls -la" needs to get file attributes (file size, modification times,
ownership, etc) from *each* of the files from the subject directory. In
your case, it needs to read more than 30,000 inodes to get them. The "ls
-la" is slower for *any* filesystem but particularly troublesome for a
cluster filesystem such as GFS due to:
1. Cluster locking overheads (it needs readlocks from *each* of the
2. Depending on when and how these files are created. During file
creation time and if there are lock contentions, GFS has a tendency to
spread the file locations all over the disk.
3. You use iscsi such that dlm lock traffic and file block access are on
the same fabric ? If this is true, you will more or less serialize the
Hope above short answer will ease your confusion.
> However, it took ages to list the subdirectory on an
> absolute idle cluster node. See below:
> # time ls -la | wc -l
> real 3m5.249s
> user 0m0.628s
> sys 0m5.137s
> There are about 3 minutes spent on somewhere. Does
> anyone have any clue what the system was waiting for?
> Thanks for your time and wish to see your valuable
> comments soon.
> Be a better friend, newshound, and
> know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ
> Linux-cluster mailing list
> Linux-cluster at redhat.com
More information about the Linux-cluster