[Linux-cluster] GFS tuning for combined batch / interactive use

[Steven Whitehouse]

Hi,

On Thu, 2010-12-16 at 00:47 +0100, Kevin Maguire wrote:
> Hi
> 
> We are running a 20 node cluster, using Scientific Linux 5.3, with a GFS 
> shared filesystem hosted on our SAN. Cluster nodes are dual core units 
> with 4 GB of RAM, and a standard Qlogic FC HBA.
> 
> Most of the 20 nodes form a batch-processing cluster, and our users are 
> happy enough with the performance they get, but some nodes are used 
> interactively. When the filesystem is under stress due to large batch 
> processing jobs running on other nodes, interactive use becomes very slow 
> and painful.
> 
> Is there any tuning I (the sysadmin) can do that might help in this 
> situation?  Would a migration to gfs2 make a difference? Are all nodes 
> treated identically, or can hosts mounting the filesystem have any kind of 
> priority/QoS? Which tools could I use to track down any bottlenecks?
> 
There are no priority/QoS controls currently available to the users, I'm
afraid. All nodes are treated equally as you say. I suspect that the
reason that interactive use becomes slow is just down to locality of
accesses. The GFS locking is done on a per-inode basis, so where writes
are going on to an inode, ensuring that reads to that same inode are
also done on the same node as much as possible should improve
performance.

In other words, it would be better to divide up jobs in the cluster
according to the data which they access rather than according to whether
they are interactive or not.

Are you using mmap() at all? If so then GFS2 should be significantly
more scalable than GFS,

Steve.


> In theory we could update kernel+gfs bits to a later release, though we 
> saw the same issues when using the same cluster with a SL4.x stack, but 
> for now it's
> 
> kernel-2.6.18-128.1.1.el5.i686
> kmod-gfs-0.1.31-3.el5.i686
> gfs-utils-0.1.20-7.el5.i386
> gfs2-utils-0.1.53-1.el5_3.1.i386
> 
> Thanks for any help/suggestions,
> Kevin
> 
> 
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster