[Linux-cluster] New virtual synchrony API for the kernel: was Re: [Openais] New API in openais

Steven Dake sdake at mvista.com
Fri Sep 10 20:53:15 UTC 2004

On Fri, 2004-09-10 at 10:51, Lars Marowsky-Bree wrote:
> On 2004-09-01T23:03:12,
>    Steven Dake <sdake at mvista.com> said:
> I've been pretty busy the last couple of days, so please bear with me
> for my late reply.
> A virtual synchrony group messaging component would certainly be
> immensely helpful. As it pretty strongly ties to membership events (as
> you very correctly point out), I do think we need to review the APIs
> here.
> Could you post some sample code and how / where you'd propose to merge
> it in?

The virtual synchrony APIs I propose we start with can be reviewed at:


There is a sample program using these interfaces at:


> Also, again, I'm not sure this needs to be in the kernel. Do you have
> upper bounds of the memory consumption? Would the speed really benefit
> from being in the kernel?
Right now, the entire openais project with all the services it provides
consumes 2MB of ram at idle.  I'd expect under load its about 3MB.  The
group messaging protocol portion of that uses perhaps 1 MB of ram.  It
will reject new messages if its buffers are full, so it cannot grow

Your definately correct; a virtual syncrhony protocol doesn't absolutely
have to be in the kernel.  In fact, it is implemented today completely
in userland with 8.5MB/sec throughput to large groups with encryption
and authentication.

You could gain some network performance by ridding UDP from the IP
header, but this is only 8 bytes.

There is little performance gain in using the kernel (as basically, a
kernel thread/process would have to be created to operate the protocol).

The only point of a kernel virtual syncrhony API is to standardize on
one set of messaging APIs for the kernel projects that require messaging
(and at a higher level, distributed locks, fencing, etc).

We want to avoid is two seperate messaging protocols operating at the
same time (performance drain).

We also want to choose wisely the messaging model and protocol we use. 
If we don't, we could have problems later.

> OTOH, all other networking protocols such as TCP, SCTP or even IP/Sec
> live in kernel space, so clearly there's prior evidence of this being a
> reasonable idea.
> Sincerely,
>     Lars Marowsky-Brée <lmb at suse.de>

More information about the Linux-cluster mailing list