[Linux-cluster] Can anyone (Ken?) explain why num_glockd mount option is there? TIA. EOM.

[Ken Preslan]

On Thu, Aug 26, 2004 at 10:55:29AM -0700, Cahill, Ben M wrote:
> Can anyone (Ken?) explain why num_glockd mount
>         option is there? TIA. EOM.

One of the things (probably the major thing) that GFS uses memory for
that's different from other filesystems is locks.  That introduces the
interesting property that in order for GFS to free that memory, it has
to do network I/O to unlock locks.  (The same is true for memory that
contains dirty disk blocks, but the VM knows about that.)  This means that
freeing memory can only happen so quickly.

In the past, GFS had one thread (gfs_glockd) that would scan through the
glock hash table looking for cached locks that were no longer needed.
It would then unlock those locks and free the memory associated with them.
But as it turned out, there were memory problems, if you had a few
processes that were scanning through huge directory trees.  You could
get into a situation where you were acquiring new locks much faster than
gfs_glockd could release old ones.  (There were many threads acquiring,
but only one thread releasing.)  If this kept up, you'd soon run out
of memory and bad things would happen.

My first quick-n-dirty solution to this was to add the num_glockd mount
option.  It would create many many threads that would look for unused
locks and unlock them.  You could balance out acquiring and releasing
processes if you knew what workload you were running.  So, that's why the
option was there originally.

Multiple gfs_glockd processes didn't completely solve the problem, though.
You could still get into the situation where GFS wasn't responding
quickly enough to memory pressure.  So I made a bunch of changes that
made things a lot better:

1) I broke gfs_glockd into two threads:
	A) gfs_scand - scans the glock hash table looking for glocks
	   to demote.  When it finds one, it puts it onto a reclaim list
	   of unneeded locks, and wakes up gfs_glockd.  
	B) gfs_glockd - looks at that the glocks on that reclaim list
	   and starts demoting them.
2) When the number of locks on the reclaim list becomes too great,
   threads that want to acquire new locks will pitch in and release a
   couple of locks before acquiring a new one.
3) GFS is more proactive about putting locks that it knows it won't need
   onto the reclaim list.  This reduces the need for gfs_scand and
   actually walking the hash table.  There's more work that could be done
   in this area.

So, I left the num_glockd option there to in case it's still needed for
some reason.  But because of #2 above, I don't think it will be.  The
option may go away in the future.

-- 
Ken Preslan <kpreslan at redhat.com>