[Libguestfs] More parallelism in VDDK driver

[Eric Blake]

On 8/5/20 9:38 AM, Eric Blake wrote:

> Correct.  Using -W doesn't make sense without -m (if you only have one 
> worker, you might as well proceed linearly than trying to randomize 
> access, but even when you have multiple threads, there are cases where 
> linear operations are still useful, such as 'nbdkit streaming'.  But -m 
> is definitely the knob that controls how many outstanding I/O requests 
> qemu-img is willing to use; and once you are using -m, using -W makes 
> life easier for those coroutines to stay active.  The default -m1 says 
> that at most one request is outstanding, so parallelism in the server is 
> not utilized.  With higher -m, qemu-img issues up to that many requests 
> without waiting for server answers, but all on the same NBD connection. 
> Ideally, you'll get the maximum behavior as 'qemu-img -m' and 'nbdkit 
> --threads' choose the same values; if either side has fewer in-flight 
> operations permitted than the other, then that side has the potential to 
> become a bottleneck.  Right now, nbdkit defaults to 16 threads (that is, 
> up to 16 in-flight operations) for any PARALLEL plugin.

This reminds me of a previous thread: we probably want to add a new mode 
to nbdkit parallelism:
https://www.redhat.com/archives/libguestfs/2019-May/msg00089.html

Use of qemu-nbd -m without -W is like my proposed 
NBDKIT_THREAD_MODEL_SERIALIZE_RETIREMENT (okay, I just invented that 
name; I did not name it in that thread).  Use of qemu-nbd -m with -W is 
then NBDKIT_THREAD_MODEL_PARALLEL.  Use of qemu-nbd without -m (at which 
point -W makes no difference) is like 
NBDKIT_THREAD_MODEL_SERIALIZE_REQUESTS.  The middle mode is where you 
allow multiple in-flight requests, but serialize their retirement 
(different than serializing their issuance), for a client that can't 
deal with out-of-order replies.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org