[Cluster-devel] [PATCH 00/11] gfs2: iopen glock locking scheme rework

[Andreas Gruenbacher]

This patch queue reworks the iopen glock locking scheme to avoid
resource group glock bouncing and cache thrashing when inodes are
deleted that have been accessed on other cluster nodes.

When the link count of an inode drops to zero, the inode is marked
GFS2_BLKST_UNLINKED in the bitmap.  Before the inode can be deleted, we
need to make sure that it's no longer in use locally or on any other
cluster node.  We use the inode reference count to keep track of local
users, and the iopen glock to keep track of other users across the
cluster: all nodes that have an inode cached or in active use keep a
shared lock on the iopen glock.  When deleting an inode, the iopen glock
is promoted to an exclusive lock first.  Other nodes may keep the inode
open, so there's no guarantee that we'll get that exclusive lock in
definite time.

To work around that, when a node fails to get that exclusive lock, it
will rely on the other nodes to perform the delete.  The delete will
happen eventually, on the first node that obtains the iopen glock
exclusively, leading to resource group glock bouncing and the associated
loss of caching.  Note that the inode doesn't have to be in active use
on multiple nodes at the same time to trigger that behavior; all it
takes is to delete the inode on another node that the one on which it
was created.

This patch queue reworks this locking scheme in the following ways: when
trying to obtain the iopen glock exclusively, nodes will give the other
nodes holding the inode cached a chance to drop the inode and release
their shared lock.  When a node cannot drop the inode immeidately in
response to such a request (because the inode is held open), it will
signal that to the cluster by causing activity on the associated inode
glock.

Because we cannot rely on existing nodes to cooperate in this new
scheme, after giving up a shared iopen glock, we need to verify that the
inode was deleted eventually.  This would normally require accessing the
resource group, negating any benefits we've had so far, but we can
remember that the inode was deleted in the inode glock lock value block
(LVB) to get around that.

Thanks,
Andreas

Andreas Gruenbacher (10):
  gfs2: Don't add glocks to the LRU while still in use
  gfs2: Keep track of deletes in inode LVBs
  gfs2: Turn gl_delete into a delayed work
  gfs2: Give up the iopen glock on contention
  gfs2: Try harder to delete inodes locally
  gfs2: Minor gfs2_lookup_by_inum cleanup
  gfs2: Move inode generation number check into gfs2_inode_lookup
  gfs2: Check inode generation number in delete_work_func
  gfs2: Wake up when setting GLF_DEMOTE
  gfs2: Smarter iopen glock waiting

Bob Peterson (1):
  gfs2: Allow ASPACE glocks to also have an lvb

 fs/gfs2/export.c                 |   4 +-
 fs/gfs2/glock.c                  | 137 ++++++++++++++++++++++++++++---
 fs/gfs2/glock.h                  |   4 +-
 fs/gfs2/glops.c                  |   5 +-
 fs/gfs2/incore.h                 |   8 +-
 fs/gfs2/inode.c                  |  34 ++++++--
 fs/gfs2/inode.h                  |   2 +-
 fs/gfs2/rgrp.c                   |   3 +-
 fs/gfs2/super.c                  |  72 +++++++++++++---
 include/uapi/linux/gfs2_ondisk.h |   6 ++
 10 files changed, 237 insertions(+), 38 deletions(-)

-- 
2.20.1