[Linux-cluster] GFS2: Huge number of glocks

[Allen Belletti]

Hi All,

I had posted about this once before and didn't get a response.  Was
really hoping that Steven or another person who's involved might be able
to comment at least briefly.  It would really help to know if this is
"normal" or not.

Thanks,
Allen

-------------

I've been running GFS and now GFS2 for several years on a two-node mail
cluster, generally with good results, especially once GFS2 became
production ready and we upgraded.  However from time to time (ranging
from a few days to a month), we'll get a "stuck" lock on one particular
file or another which them blocks a user from their mail.  While looking
into this, I've recently become aware of a VERY large number of glocks
being left behind after our nightly rsync backups.  I'm checking on the
lock situation with "gfs2_tool lockdump /home" and counting locks by
piping through "grep ^G | wc -l".  We have two GFS2 filesystems
mounted.  On one of them, the number of glocks returns to "normal" after
the backup (currently showing about 5400.)  On the other, it stays very
high although it will drop somewhat throughout the day.  Currently I am
seeing over 500,000.  Given the ten minutes or so that it takes to list
them, this seems like it can't be great for performance.

Most of the locks look like this:

G:  s:SH n:5/b25806 f: t:SH d:EX/0 l:0 a:0 r:3
H: s:SH f:EH e:0 p:31042 [(ended)] gfs2_inode_lookup+0x114/0x1f0 [gfs2]

Note that the pid (31042 in this case) corresponds to one of the
completed rsync processes which generated the locks in the first place.

My questions are 1) Is this a bad thing?  My gut feeling is "yes" but
perhaps the system is highly efficient in dealing with these locks, and
2) Can anything be done about it?  The tuning opportunities in GFS2 are
very limited compared to GFS, and the few things I've tried seem to have
no effect.

By the way, I am running with plock_ownership="1" and
plock_rate_limit="0" in cluster.conf.

Thanks in advance,
Allen

-- 
Allen Belletti
allen at isye.gatech.edu                             404-894-6221 Phone
Industrial and Systems Engineering                404-385-2988 Fax
Georgia Institute of Technology