[Linux-cluster] GFS2 - monitoring the rate of Posix lock operations

[Jankowski, Chris]

Steve,

Q2:
>>> Are you sure that the workload isn't causing too many cache invalidations due to sharing files/directories between nodes? This is the most usual cause of poor performance.

The other node is completely idle and kept that way by design. Users are connecting through an IP alias managed by the appplication service. Application administrators also log in through the alias to do their maintenance work. In the case of this particular test I manually listed what is running where. I am very concious of the fact that accesses from multiple nodes invalidate local in-memory caching.

Q3:
>>> Have you used the noatime mount option? If you can use it, its highly recommended. Also turn off selinux if that is running on the GFS2 filesystem.

The filesystem is mounted with noatime and no nodiratime options. SELinux is disabled.

Q4:
>>>Potentially there might be. I don't know enough about the application to say, but it depends on how the workload can be arranged,

The application runs on one node at a time.  It has to, as it uses shared memory. The application uses a database of indexed files. There are thousands of them. Also, it uses standard UNIX flile locking and range locking.

What else can I do to minimise the GFS2 locking overhead in this asymetrical configuration.

Q5:
Is this the case that when gfs_controld gets to 100% of one coe DPU usage then this is a hard limit on the number of Posix locks taken.  Is there only one gfs_lockd daemon servicng all GFS2 filesystems or are they run on a per filesystems basis?  In the latter case I would have thought that breaking the one filesystem that I have into several may help. Would it not?

Thanks and regards,

Chris



-----Original Message-----
From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Steven Whitehouse
Sent: Saturday, 27 March 2010 00:26
To: linux clustering
Subject: Re: [Linux-cluster] GFS2 - monitoring the rate of Posix lock operations

Hi,

On Fri, 2010-03-26 at 02:31 +0000, Jankowski, Chris wrote:
> Hi,
> 
> I understand that GFS2 by default has a limit on the rate of POSIX locks to 100 per second.
> This limit can be removed by the following entry in /etc/cluster/cluster.conf:
> 
> <gfs_controld plock_rate_limit="0" plock_ownership="1"/>
> 
> 
> Question 1:
> ------------
> How can I monitor the rate of POSIX lock operations?
> 
> The reason I am asking this question is that I am trying to maximise application throughput in a cluster. This is a database type application running on one node with the other node being an idle standby. Under large generated workload I see the system in a state in which it still has available unused CPU, memory, IO rate and bandwidth and network bandwidth capacity, but will not go any faster. I am suspecting that GFS POSIX lock processing is the bottleneck, but at the moment have no data to prove it, and no information on how to tune it to remove this bottleneck.
> 
There is a ping_pong lock test program which can be used to get an idea of the locking rate, but so far as I know there is no way to monitor this in a running system.

> 
> Question 2:
> -----------
> What may be limiting the throughput of GFS2 with plock_rate_limit set to 0 and in the absence of global physical shortage of resouces?  Could this be the gfs_controld process saturating one CPU core?  I indeed see gfs_lockd using 90%+ of one CPU core in top(1).
> 
Are you sure that the workload isn't causing too many cache invalidations due to sharing files/directories between nodes? This is the most usual cause of poor performance.

> 
> Question 3:
> -----------
> What else can I tune to get higher maximum throughput from GFS2 used in this asymmetrical configuration?  Potentially, I need much more throughput, as my real production cluster is to support 1,000+ transactional users.
> 
Have you used the noatime mount option? If you can use it, its highly recommended. Also turn off selinux if that is running on the GFS2 filesystem.

> 
> Question 4:
> -----------
> Is there a leaner, more efficient way of using GFS2 for such asymmetrical operation when all accesses are from only one node and the other acts as a hot standby with no fsck needed on failover of the service to the other node?  I am in a position to move the mounting of the GFS2 fiesystem into the service script if that would be of help.
> 
Potentially there might be. I don't know enough about the application to say, but it depends on how the workload can be arranged,

Steve.

> Your comments and ideas will be much appreciated.
> 
> Regards,
> 
> Chris
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster


--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster