[Libguestfs] Libnbd asynchronous API with epoll

[Richard W.M. Jones]

On Sun, Jul 09, 2023 at 11:01:47AM +0100, Richard W.M. Jones wrote:
> (Sorry for the late reply, was a bit involved in a qemu bug last week ...)
> 
> On Fri, Jul 07, 2023 at 08:58:50AM +0000, Tage Johansson wrote:
> > On 7/6/2023 7:06 PM, Nir Soffer wrote:
> > 
> > 
> >     - After calling for example aio_notify_read(3), can I know that the next
> >     reading from the file descriptor would block?
> > 
> >     No, you have to call again aio_get_direction() and poll again until the
> >     event happens.
> > 
> > 
> > Well, what I mean is:
> >
> > After calling aio_notify_read, if aio_get_direction returns
> > AIO_DIRECTION_READ or AIO_DIRECTION_BOTH, can I know that the
> > reading on the file descriptor actually blocked? Or might there be
> > cases when aio_notify_read returns and the next direction includes a
> > read and there is still more data to read on the file descriptor?
> 
> It's probably best to have a look at the generated code, but it works
> like this:
> 
> - aio_notify_read is the program telling libnbd that the socket is
>   readable.  ie. The socket won't block, one byte at least is
>   available to read.
> 
> - aio_get_direction is libnbd telling the program that we are in a
>   state where we are expecting to read or write (or either) to the
>   socket.
> 
> As an example, let's say we have just connected to the NBD server.  We
> expect the NBD server to send a magic string, here:
> 
>   https://gitlab.com/nbdkit/libnbd/-/blob/20dceca48623835c3e8a195ef4135cce5e33efba/generator/states-magic.c#L28
> 
> For example:
> 
>   $ nbdkit -s null
>   NBDMAGICIHAVEOPT     <- magic string sent by the server
> 
> But in this case let's say the server is slow to send the magic string.
> 
> The call to recv_into_rbuf -> h->sock->ops->recv -> recv returns
> EWOULDBLOCK (recv_into_rbuf returns 1).  Then we stay in the same
> state (MAGIC.RECV_MAGIC), and return to the main program.
> 
> At this point if the main program calls nbd_aio_get_direction it would
> return AIO_DIRECTION_READ.  We are expecting to read the socket next.
> A write on the socket is not possible in this state (so we will not
> return AIO_DIRECTION_BOTH).

... and to continue the thought:

If the program now detects that the socket is readable using
select/poll/epoll/etc, it can call nbd_aio_notify_read.  This
re-enters the state machine, reading from the socket, getting some
data (hopefully), and moving the state machine on until it blocks
again, at which point nbd_aio_notify_read returns.

Then you'd call nbd_aio_get_direction again to find out if we are
blocked expecting a read, write (or either), and the cycle continues.

Rich.

> I hope that's a bit clearer.
> 
> I'm not sure about the epoll question.  I haven't thought about it
> deeply, but it seems as if edge-triggered notification might be a
> problem, but you could keep track of this from the main program with
> an extra variable, turning edge-triggered into level-triggered.
> 
> Rich.
> 
> > 
> > I guess this is the case, but I must know or the client may hang unexpectedly.
> > 
> > 
> > Best regards,
> > 
> > Tage
> > 
> > 
> 
> > _______________________________________________
> > Libguestfs mailing list
> > Libguestfs at redhat.com
> > https://listman.redhat.com/mailman/listinfo/libguestfs
> 
> 
> -- 
> Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-builder quickly builds VMs from scratch
> http://libguestfs.org/virt-builder.1.html
> _______________________________________________
> Libguestfs mailing list
> Libguestfs at redhat.com
> https://listman.redhat.com/mailman/listinfo/libguestfs

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit