[Linux-cluster] GS2 try_rgrp_unlink consuming lots of CPU

[Miller, Gordon K]

Hi,
We are still struggling with the problem of try_rgrp_unlink consuming large amounts of CPU time over durations exceeding 15 minutes. We see several threads on the same node repeatedly calling try_rgrp_unlink with the same rgrp and the same group of inodes being retuned over and over until 15 minutes later one of the calls fails to find an inode and returns zero causing GFS2_RDF_CHECK in rd_flags to be cleared and thus stopping the calls to try_rgrp_unlink. We have not determined what triggers this behavior. What causes the GFS2_RDF_CHECK flag to get set? Are there any locking issues with the rgrp?

Our kernel version is 2.6.18-128.7.1

Gordon

-----Original Message-----
From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Steven Whitehouse
Sent: Tuesday, October 27, 2009 2:57 AM
To: linux clustering
Subject: RE: [Linux-cluster] GS2 try_rgrp_unlink consuming lots of CPU

Hi,

On Mon, 2009-10-26 at 15:47 -0700, Miller, Gordon K wrote:
> When making our GFS2 filesystems we are using default values with the exception of the journal size which we have set to 16MB. Our resource groups are 443 MB in size for this filesystem.
>
> I do not believe that we have the case of unlinking inodes from one node while it is still open on another.
>
> Under what conditions would try_rgrp_unlink return the same inode when called repeatedly in a short time frame as seen in the original problem description? I am unable to correlate any call to gfs2_unlink on any node in the cluster with the inodes that try_rgrp_unlink is returning.
>
> Gordon
>
It depends which kernel version you have. In earlier kernels it tried to deallocate inodes in an rgrp only once for each mount of the filesystem.
That proved to cause a problem for some configurations where we were not aggressive enough in reclaiming free space. As a result, the algorithm was updated to scan more often.

However in both cases, it was designed to always make progress and not continue to rescan the same inode, so something very odd is going on.
The only reason that an inode would be repeatedly scanned is that it has been unlinked somewhere (since the scanning is looking only for unlinked
inodes) and cannot be deallocated for some reason (i.e. still in use) and thus is still there when the next scan comes along.

Steve.


--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster