[Libguestfs] [nbdkit PATCH v3 3/5] nbd: Use libnbd 0.1.3+

[Eric Blake]

On 6/12/19 4:00 PM, Eric Blake wrote:
> This conversion should be feature compatible with the standalone nbd
> code. Note that the use of libnbd makes the binary for this particular
> plugin fall under an LGPLv2+ license rather than BSD; but the source
> code in nbd.c remains BSD.
> 
> A lot of code simply disappears, now that I'm no longer directly
> utilizing the NBD protocol files but relying on libnbd.  Coordination
> between threads from nbdkit and a central state machine thread is
> gated by a lock around the pending transition list, using the same
> trick as before of using a semaphore to wake up the right thread after
> the server's reply, regardless of out-of-order handling from the
> server.
> 
> Benchmark-wise, using the same setup as in commit e897ed70, I see
> a slight slowdown:
> 
> Pre-patch, the runs averaged  0.754s, 2.17E+08 bits/s
> Post-patch, the runs averaged 0.849s, 1.93E+07 bits/s

With the latest libnbd, the numbers are now:

    Pre-patch, the runs averaged  16.259s, 1.03E+10 bits/s
    Post-patch, the runs averaged 17.585s, 9.54E+09 bits/s

for about 8% degradation.

> 
> The explanation is that pre-patch, one nbdkit thread can be write()ing
> the next request to the server at the same time the reader thread is
> read()ing the previous reply; post-patch, libnbd's current structure
> handles all I/O under a lock such that there is no simultaneous
> bidirectional traffic. There are discussions about adding future
> libnbd APIs to address this limitation.

My other insight is that since all our calls are aio, they really
shouldn't be blocking. The biggest delay is thus not waiting for a lock,
but the fact that as long as the state machine is blocked on POLLIN, we
are delaying our next request to the server, whether or not the server
would actually be able to read the request at the moment.  Parallel
threads for read() and write() are essential for max performance under
blocking I/O, but shouldn't really matter for non-blocking I/O, so our
real problem is that libnbd's state machine doesn't have truly
independent h->rstate and h->wstate.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/libguestfs/attachments/20190613/6b4f10a2/attachment.sig>