[Linux-cluster] Groupd semaphore limits

[David Teigland]

On Tue, Sep 29, 2009 at 10:43:33AM +0200, Juan Ramon Martin Blanco wrote:
> Hi folks!
> 
> Is it anywhere documented how many semaphores groupd uses for each gfs
> filesystem that is mounted? Is it anywhere documented what is the
> maximum number of gfs filesystems that can be mounted on a cluster?
> We have had an issue in our production server with gfs filesystems not
> being mounted and groupd consuming 100% cpu. Stracing it we saw it
> could not create any new semaphore (with semget). After increasing
> limits (modifying sysctl kernel.sem variable), everything is working
> fine.
> The default limit on rhel 5.3 is 128 semaphore matrix (each matrix has
> 3 semaphores). Mounting a gfs filesystem seems to consume 6
> semaphores, so a maximum of 64 can be mounted.
> 
> I hope it can help anybody facing this problem in the future. I would
> be fine if you document it somewhere.

This is a regression caused by openais ipc being rewritten to use semaphores.
Anything that uses openais (through various libraries) indirectly does this
ipc and consumes semaphores; it's hard to say exactly when you'll run out.  As
you've found, the only thing to do is to increase limit to something that
works for you.

(The ipc has since changed again to *not* use semaphores, I don't know what
release that will appear in.)

Dave