[Linux-cluster] Slowness above 500 RRDs

[David Teigland]

On Thu, Apr 26, 2007 at 06:36:02PM +0200, Ferenc Wagner wrote:
> David Teigland <teigland at redhat.com> writes:
> 
> > Thanks, some of these numbers look odd to me; I'll need run the test on my
> > own rhel4 cluster to understand them better.
> 
> I'm working with three nodes: 1, 2 and 3.  Looks like the mount by 3
> makes a big difference.  When the filesystem is mounted by 1 and 2
> only, my test runs much faster.  Filesystem mounted by 3 alone is also
> fast.  But 3 doesn't seem to cooperate with anyone else with
> reasonable performance.
> 
> If I mount the filesystem on all three nodes and run the test on 1,
> the network traffic of 2 and 3 is rather unbalanced: tcpdump receives
> 19566 packets on 2 and 29181 on 3.  It's all 21064/tcp traffic, I can
> provide detailed data if that seems useful.

It sounds like your tests are mixing the effects of the flocks/plocks with
the effects of gfs's own internal file locking.  If you want to test and
compare flock/plock performance you need to make sure that gfs's internal
dlm locks are always mastered on the same node (either locally, in which
case it'll be fast, or remotely in which case it'll be slow).  The first
node to use a lock will be the master of it.

> > In the end, though, I think you'll get the best performance by using
> > flocks under the new cluster infrastructure and kernel.
> 
> Can you point me to the relevant documentation and version numbers for
> upgrading to the new cluster infrastructure?  Unfortunately it's not
> prepackaged, which means much more maintenance work, but I'm willing
> to test if it helps enough to justify that.

It depends on what kind of packaging you want... there's RHEL5 and its
clones, of course.  There's the cluster-2.00.00.tar.gz release.  You can
look at this for usage:

http://sources.redhat.com/cgi-bin/cvsweb.cgi/cluster/doc/usage.txt?cvsroot=cluster

Dave