[Linux-cluster] GFS cluster / DLM locking - Mostly idle but high load

[Marc Grimme]

On Wednesday 17 October 2007 09:52:20 Gordan Bobic wrote:
> On Wed, 17 Oct 2007, Marc Grimme wrote:
> >>>> I have a cluster (3 nodes at the moment, may grow up to 16) for
> >>>> handling a lot of small files (Maildir). When I test the system by
> >>>> sending around 3-5 messages/second I see the load on the cluster nodes
> >>>> go up to about 20-30, even though the CPUs on the cluster are about
> >>>> 90% idle at all times.
> >>>>
> >>>> I am guessing that this is due to the clustered machines waiting for
> >>>> DLM locks to be established, which causes a lot of processes to be
> >>>> fighting to run, but since they don't get to run very soon, they back
> >>>> up and cause the load averages to go up.
> >>>>
> >>>> Assuming the DLM runs over the interface specified by IP and MAC in
> >>>> cluster.conf, it is running over gigabit ethernet.
> >>>>
> >>>> Are there any configuration changes or tuning parameters I can apply
> >>>> to DLM to alleviate this condition? The machine I'm running the test
> >>>> from (the one sending messages) is about 1/4 of the spec of each of
> >>>> the cluster nodes, and it's running a load average of about 0.4. It
> >>>> seems crazy that a single low-spec node should be able to completely
> >>>> overwhelm a cluster 12x it's spec several times over.
> >>>
> >>> I don't know alot about GFS but since no one else has replied yet, my
> >>> understanding is that it's not suitable for an applications like what
> >>> you describe (many small files being opened frequently). I think GFS2,
> >>> which is still a tech preview, has been redesigned to improve this
> >>> situation.
> >>
> >> Indeed, I am aware that GFS2 is still broken, but I seem to be getting
> >> no worse a performance out of GFS than I get out of NFS. The only
> >> penalty is the high load, but the throughput is actually similar. The
> >> advantage that makes GFS win is that I don't need an arbitrating server
> >> to handle the NFS exports, which makes the clustering and redundancy a
> >> bit tidier.
> >
> > with your testing did you also try to adapt the size of the
> > rsbtbl_size/lkbtbl_size? I would be quite interested if this increases
> > your performance or not.
>
> I cannot find these files in /proc (that's where they are implied to be in
> the docs). Can you please point me in the right direction?
Sorry I new I forgot something ;-)
http://www.opensharedroot.org/Members/marc/blog/blog-on-dlm/red-hat-dlm-__find_lock_by_id/influence-of-locktable-sizes-rsbtbl_size-lkbtbl_size
>
> > Do you have lot of small files?
>
> Yes. The problem doesn't seem to be so bad when files are in different
> directories, but when lots of files are being written to the same
> directory, the load goes up quite badly.
Then this should help. Also enable lock_purging if not already done.
>
> Gordan
Marc.
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster



-- 
Gruss / Regards,

Marc Grimme
http://www.atix.de/               http://www.open-sharedroot.org/