[Libguestfs] [nbdkit PATCH 3/4] nbd: Use libnbd 0.1

[Eric Blake]

On 5/29/19 10:13 PM, Eric Blake wrote:
> Benchmark-wise, using the same setup as in commit e897ed70, I see
> an order-of-magnitude slowdown:
> 
> Pre-patch, the runs averaged   1.266s, 1.30E+08 bits/s
> Post-patch, the runs averaged 11.663s, 1.41E+07 bits/s
> 
> This will need further profiling to determine how much is nbdkit's
> fault, and how much is libnbd's.  I think that we are probably holding
> some locks for too long, resulting in some serialized performance.
> Also, the standalone code was able to run read of command 1 in
> parallel with write of command 2 via separate threads, whereas
> libnbd's state machine is serializing everything (whether or not the
> state machine spreads out the I/O to do writes from the thread calling
> nbd_aio_FOO and reads from the reader thread, the two are never run at
> once).

Rich identified and fixed the culprit - libnbd was not setting
TCP_NODELAY (disabling Nagle's algorithm) the way nbd-standalone.c did,
which meant that any request that gets split over TCP windowing sizes
waits to send the second packet until the ACK for the first has been
received by the server. While disabling Nagle's increases network
overhead (you are sending more short packets with their overhead rather
than bundling things into larger packets by virtue of the downtime
waiting for the ACK), it is an artificial delay (the server can't
process anything until the packets arrive). libnbd 0.1.2 now disables
Nagle's algorithm, and my repeat of the tests with that change showed an
improvement to 2.180s, which is more eassily explained by the
serialization nature that libnbd is never read()ing from one thread at
the same time another thread is write()ing.

I'd have to intentionally cripple nbd-standalone.c to do that same
serialization, to see if libnbd actually offers an overall performance
gain when I/O is serialized (may not be as easy as it sounds; I was
relying on blocking I/O from separate threads, but merely serializing
that so that no thread is doing I/O concurrent with another risks
deadlock in the case of a client sending a large NBD_CMD_READ followed
by NBD_CMD_WRITE to a server that responds strictly serially, where the
server is waiting for the client to finish the read's response but the
client is waiting for the server to parse the write's request; libnbd
works around that by using non-blocking sockets and prioritizing reads
over writes rather than insisting on complete transactions).


> -/* Connect to a TCP socket, returning the fd on success */
> -static int
> -nbdplug_connect_tcp (void)
> -{
> -  struct addrinfo hints = { .ai_family = AF_UNSPEC,
> -                            .ai_socktype = SOCK_STREAM, };
> -  struct addrinfo *result, *rp;
> -  int r;
> -  const int optval = 1;
> -  int fd;
> -

> -
> -  if (setsockopt (fd, IPPROTO_TCP, TCP_NODELAY, &optval,
> -                  sizeof (int)) == -1) {
> -    nbdkit_error ("cannot set TCP_NODELAY option: %m");
> -    close (fd);
> -    return -1;
> -  }


-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/libguestfs/attachments/20190530/a74d8f15/attachment.sig>