[Linux-cluster] GFS Performance Problem / coherency between GFS access speed and network bandwidth

[Josef Bacik]

On Thu, Sep 13, 2007 at 06:10:35PM +0200, Christian Brandes wrote:
> I found out, that there must be a coherency between GFS access speed and
> network bandwidth allthough i/o is going through FC to the RAID in my 
> 4-node cluster.
> 
> It is 4 nodes each redundantly connected by 4GB-FC to the two
> controllers of a FC-to-FC RAID System.
> Devices are accessed through multipath-tools' /dev/mapper devices on the
> primary path.
> Each node has 1 100BaseT and 2 1000BaseTX network interfaces.
> 
> Before installing the cluster I did some performance tests such as
> writing from /dev/zero to the RAID containing an XFS file system and now
> with a GFS on it.
> 
> Here are some numbers:
> 
> FS      heartbeat-network      access-speed     latency
> ----------------------------------------------------------
> GFS     10MBit/s               50MB/s           very high
> GFS     100MBit/s              66MB/s           high
> GFS     1000MBit/s             100MB/s          low
> XFS                            260MB/s          none
> 
> What does GFS do over network when it is accessed by a node.
> Does dlm use such a high network bandwith before the first bit is
> written through FC to the RAID?
> 
> I had a VM on VMWare Server running on GFS that was interrupted every 10
> seconds for one ore two seconds.
> Now with 1000MBit/s it is not interrupted any more. But do not want to
> waste more than 60% of our RAID's performance.
> 
> What can I do to speed up GFS?
> 

Are you mounting with noatime?  Everything GFS does requires a lock, so yeah
your network speed is going to directly affect how fast GFS is.  Using noatime
will cut down on the amount of dlm traffic you are doing so that should help
some.

Josef