[Linux-cluster] GFS2 lock accumulation

[Allen Belletti]

Hi All,

I've been running GFS and now GFS2 for several years on a two-node mail 
cluster, generally with good results, especially once GFS2 became 
production ready and we upgraded.  However from time to time (ranging 
from a few days to a month), we'll get a "stuck" lock on one particular 
file or another which them blocks a user from their mail.  While looking 
into this, I've recently become aware of a VERY large number of glocks 
being left behind after our nightly rsync backups.  I'm checking on the 
lock situation with "gfs2_tool lockdump /home" and counting locks by 
piping through "grep ^G | wc -l".  We have two GFS2 filesystems 
mounted.  On one of them, the number of glocks returns to "normal" after 
the backup (currently showing about 5400.)  On the other, it stays very 
high although it will drop somewhat throughout the day.  Currently I am 
seeing over 500,000.  Given the ten minutes or so that it takes to list 
them, this seems like it can't be great for performance.

Most of the locks look like this:

G:  s:SH n:5/b25806 f: t:SH d:EX/0 l:0 a:0 r:3
 H: s:SH f:EH e:0 p:31042 [(ended)] gfs2_inode_lookup+0x114/0x1f0 [gfs2]

Note that the pid (31042 in this case) corresponds to one of the 
completed rsync processes which generated the locks in the first place.

My questions are 1) Is this a bad thing?  My gut feeling is "yes" but 
perhaps the system is highly efficient in dealing with these locks, and 
2) Can anything be done about it?  The tuning opportunities in GFS2 are 
very limited compared to GFS, and the few things I've tried seem to have 
no effect.

By the way, I am running with plock_ownership="1" and 
plock_rate_limit="0" in cluster.conf.

Thanks in advance,
Allen
-- 

Allen Belletti
allen at isye.gatech.edu                             404-894-6221 Phone
Industrial and Systems Engineering                404-385-2988 Fax
Georgia Institute of Technology