[Libguestfs] Two multi-conn questions

[Eric Blake]

On Wed, Feb 01, 2023 at 12:40:45PM +0000, Richard W.M. Jones wrote:
> Hi Eric,
> 
> A couple of semi-related multi-conn questions.
> 
> (1) nbdkit-curl-plugin does not currently advertise multi-conn.  However
> forcing multi-conn makes it faster, eg:
> 
> $ time nbdkit -r curl https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img timeout=2000 --run ' nbdcopy --no-extents -p $uri jammy-server-cloudimg-amd64.img '
> █ 100% [****************************************]
> 
> real   1m15.438s
> user   0m2.513s
> sys    0m4.063s

I'm jealous of your connection.  Even with bare wget (without nbdkit
in the middle), for some reason my computer is failing to connect when
trying IPv6.  I then tried 'wget -4 ...', and didn't get the full file
until 5m37s.  And when I tried to reproduce your nbdkit line, I got:

nbdkit: curl[1]: error: pread: curl_easy_perform: Failed sending data to the peer: Connection died, tried 5 times before giving up
nbdcopy: read at offset 245104640 failed: Input/output error

at 6 minutes in.  So slow speeds and some connection flakiness is not
helping me.

> 
> $ time nbdkit -r curl https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img --filter=multi-conn multi-conn-mode=unsafe timeout=2000 --run ' nbdcopy --no-extents -p $uri jammy-server-cloudimg-amd64.img '

Yay - I'm glad my multi-conn filter makes it easier to test things
like this!

Should we tweak the docs in nbdkit-multi-conn-filter(1) to mention
that despite multi-conn-mode=unsafe being unsafe for a plugin that
does not have consistency, it is useful for timing tests on a plugin
where we suspect consistency is available in order to test timing to
see if it actually makes a difference?

> █ 100% [****************************************]
> 
> real   0m59.635s
> user   0m3.010s
> sys    0m4.538s
> 
> (Note that this is line speed - wget also takes 59 seconds)
> 
> I believe that multi-conn should be safe to enable in the curl plugin,
> certainly if we're in read-only mode (which is the most common case
> for this plugin).  It's probably not safe when operating in read-write
> mode, because HTTP doesn't have any concept of flushing, but almost no
> web server supports writing.

Agreed on both fronts; I see you have proposed this patch (and I acked
it) in the meantime.

> 
> (2) qemu's NBD client still doesn't seem to support multi-conn.  This
> makes qemu-img convert very slow when reading from nbdkit, as in the
> example below.  (Note the '.img' file on that website is actually a
> qcow2 file.)

Correct - qemu does NOT do multi-queue (or multipath) client
connections.  It is blindly using exactly one NBD client connection
per block device, and ignoring the server's advertisement of
multi-conn support.

Teaching qemu to support a multipath paradigm is not a trivial task.
I'm wondering if the effort might be easier by teaching libblkio how
to do arbitrary multipath for any underlying block device, coupled
with current efforts to teach qemu how to target any arbitrary
libblkio target, so that qemu proper doesn't have to worry about the
intricacies of multipath; from there, we'd need a way for libblkio to
target NBD devices (probably by libnbd).  In other words, instead of
trying to directly code client MULTI_CONN efforts into qemu proper, it
may be better to segregate things into more modular building blocks.

> 
> $ time nbdkit -r curl https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img --filter=multi-conn multi-conn-mode=unsafe timeout=2000 --run ' qemu-img convert -W -m 16 -p -f qcow2 $uri -O raw jammy-server-cloudimg-amd64.img.raw '
>     (100.00/100%)
> 
> real	3m29.412s
> user	0m17.856s
> sys	0m8.218s
> 
> Since we know from (1) above that copying could go at line speed when
> multi-conn is enabled, and also that qemu-img convert from the same
> local file takes only about 2 seconds, it seems like qemu-img convert
> above could go a lot faster if we allowed multi-conn.
> 
> Is there any work on adding multi-conn support to qemu's NBD client?

Not that I'm aware of at the moment, but we have proof that it may
prove fruitful to have someone spend time on.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org