[Libguestfs] [nbdkit PATCH v3 0/5] Play with libnbd for nbdkit-nbd

[Eric Blake]

On 6/12/19 4:18 PM, Richard W.M. Jones wrote:
> On Wed, Jun 12, 2019 at 04:00:08PM -0500, Eric Blake wrote:
>> libnbd-0.1.4-1 is now available in Fedora 29/30 updates testing.
>>
>> Diffs since v2 - rebase to master, bump from libnbd 0.1.2 to 0.1.3+,
>> add tests to TLS usage which flushed out the need to turn relative
>> pathnames into absolute, doc tweaks
>>
>> Now that the testsuite covers TLS and libnbd has been fixed to provide
>> the things I found lacking when developing v2, I'm leaning towards
>> pushing this on Monday, or sooner if I get a positive review.
> 
> This looks reasonable.
> 
> There's a slow-down, but we may be able to fix that with a concurrent
> writer thread in future.

I still need to collect updated numbers.

> 
> The pipe-to-self trick is interesting and comes back to the issue of
> what is the thread model of libnbd.  If we think that libnbd shouldn't
> create threads or do tricky Linux-specific stuff internally, then
> asking callers to do this is reasonable.
> 
> If I understand how this works:
> 
> * There is one "reader" thread (per connection) which is doing polls.
>   Essentially this is a background thread as far as nbdkit is
>   concerned, not a thread that nbdkit has created or knows about.
> 
> * There is a pipe-to-self (per connection).  When a command is
>   submitted from an nbdkit-managed thread a byte is sent down this
>   pipe.

Furthermore, an nbdkit-managed thread IS grabbing the libnbd lock, and
either queues the command (behind other pending work) or proceeds to
send() until the state machine blocks (which may be the full command,
but not always for large NBD_CMD_WRITE).  Waiting for the libnbd lock
shouldn't be too long since all calls into libnbd (from any thread) are
aio and shouldn't be blocking.

> 
> * The reader thread can either wake up because the socket is ready for
>   read or write, or because of an indication on the pipe-to-self.
> 
> * If the socket is ready for read or write then the normal
>   nbd_aio_notify_read|write function is called which will move the
>   state machine along, and also check for command completion.
> 
> * If it's pipe-to-self indication then we will (after checking for
>   command completion) check the direction flag again and reenter the
>   poll.  The reason for this is because when the other thread started
>   an AIO command it might have changed the handle state.

Indeed - if the reader thread was blocked while the handle was READY,
then the nbdkit-managed thread probably got off a send(), and either the
entire message was sent or it blocked and now the reader thread needs to
know that it should check for POLLOUT to finish sending the message.

> 
> * The pipe ensures there is no race (I think).

The libnbd lock ensures no two nbdkit-managed threads are moving the
state machine behind the back of any other thread (whether a different
nbdkit thread's command, or the reader thread), and the pipe ensures
that the reader thread does not deadlock (it doesn't matter which thread
advances the state machine, but the reader thread is the only one that
checks direction and advances the state machine due to nbd_aio_notify).

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/libguestfs/attachments/20190612/19090c7b/attachment.sig>