[Linux-cluster] slow GFS2 stat() performance

[Steven Whitehouse]

Hi,

On Mon, 2010-03-15 at 00:54 +0100, Sven Karlsson wrote:
> Hello,
> 
> We have a newly setup GFS2 cluster on a shared FC storage with three
> nodes. After setting up the backup software we noticed that it ran
> very slow, and investigated. It turns out that we have about a million
> files in varying sizes, and that stat()'ing these takes a long time.
> 
> For example, a "find" command on the GFS share, limited to 4000 files,
> takes 0.021s. A "find -ls" command in the same circumstances takes 17
> seconds...!
> 
> And that is only to get the permissions/ownership etc, no data reading.
> 
> We've looked for GFS tuning tips and the filesystem is mounted with
> noatime,nodiratime, statfs_slow=0, etc. ping_pong gives about 3200
> locks/s for read and write. Also tried mounting only one node and
> having a local lock manager, with some better performance but not
> satisfactory. Using a non-cluster filesystem on the same SAN manages
> "find -ls" in the order of a fraction of a second, which suggests that
> it's GFS that .
> 
> Should it be this slow? Any hints to improve performance or to do debugging?
> 
> Regards
>  SK
> 
This is probably down to the access pattern. Assuming that your find
test is the only thing running on the cluster, do you find that it goes
a lot faster the second time it is run from the same node?

I'd be surprised if running find from GFS/GFS2 on a single node,
lock_nolock basis would be much slower than any other fs, particularly
once the required disk blocks have been cached after the initial run.

Which version of GFS2 are you using?

Steve.