[Linux-cluster] GFS RG size (and tuning)

[Glen Dosey]

We have several 3.3TB and larger filesystems on GFS1. I found that using
the larger RG helps with performance. Increasing the statfs_slots seemed
to help with df time and similar tasks as well. We have a filesystem
with smaller RG's and I don't think the difference is worth moving the
data around. If there is some other reason to make changes then I will
increase the RG size. The statfs_slots can be increased on the fly and
you should hopefully see an improvement. 

Going from RHEL4 to RHEL5 provided measurable benefits for us. You may
also want to look at the option on your disk to see if you can increase
the read ahead and similar underlying performance tunables.

My understanding, and someone correct me if I am wrong, is that GFS2
should resolve the df type issues as well as provide better performance.
I tested it briefly on a 4TB volume and it clearly performs better.
We're just waiting for it to be 'ready'.



On Sat, 2007-10-27 at 14:57 +0200, Jos Vos wrote:
> On Fri, Oct 26, 2007 at 07:57:18PM -0400, Wendy Cheng wrote:
> 
> > 1. 3TB is not "average size". Smaller RG can help with "df" command - 
> > but if your system is congested, it won't help much.
> 
> The df also takes ages on an almost idle system.  Also, the system often
> needs to do rsyncs on large trees and this takes a very long time too.
> 
> In <http://sourceware.org/cluster/faq.html#gfs_tuning> it is suggested
> that you should then make the RG larger (i.e. less RGs).  As this requires
> shuffling aroung with TB's of data before recreating a GFS fs, I want to
> have some idea of what my chances are that this is usefull.
> 
> > 2. The gfs_scand issue is more to do with the number of glock count. One 
> > way to tune this is via purge_glock tunable. There is an old write-up in:
> > http://people.redhat.com/wcheng/Patches/GFS/readme.gfs_glock_trimming.R4 
> > . It is for RHEL4 but should work the same way for RHEL5.
> 
> I'll try.  I assume I can do this per system (so that I don't have to
> bring the whole cluster down, only stop the cluster services and unmount
> the GFS volumes per node)?
> 
> Any chance this patch will make it into the standard RHEL-package?
> I want to avoid to maintain my own patched packages, although as long
> as gfs.ko is in the separate kmod-gfs package that's doable.
>