[Libguestfs] More parallelism in VDDK driver (was: Re: CFME-5.11.7.3 Perf. Tests)

[Richard W.M. Jones]

On Wed, Aug 05, 2020 at 03:39:15PM +0300, Nir Soffer wrote:
> On Wed, Aug 5, 2020 at 3:28 PM Richard W.M. Jones <rjones at redhat.com> wrote:
> >
> >
> > Nir, BTW what are you using for performance testing?
> 
> virt-v2v with local image, or imageio client with local image.
> 
> > As far as I can tell it's not possible to make qemu-img convert use
> > multi-conn when connecting to the source (which is going to be a
> > problem if we want to use this stuff in virt-v2v).
> 
> But do we need multiple connections? qemu can send multiple requests
> on one connection.

As implemented now there is only one VDDK handle per connection and
VDDK doesn't allow multiple requests at the same time on one handle,
so with a single NBD connection everything will be serialized.

Now if we were to implement a thread pool inside nbdkit-vddk-plugin we
could get around that restriction (with a lot of complexity).  But the
results I posted a moment ago show that we wouldn't get anything like
a linear speed up.  It hardly seems worth it to me.

> Did you try to copy an image from nbdkit file plugin to another nbdkit
> file plugin using qemu-img convert?
> 
>     nbdkit file pluing -> qemu-img convert -W nbd:///?socket=src.sock
> nbd:///?socket=dst.sock-> nbdkit file plugin
> 
> I did not try it but I will be surprised if we don't get all 8 threads
> busy in both sides

Likely, but that's because the file plugin uses PARALLEL as thread
model, so even on a single NBD connection it can keep all threads
inside nbdkit busy.

> The reason we use multiple connection in imageio is that we don't
> support async I/O in http client, http server, and nbd client, and
> it is much easier to open new connection with the entire stuck
> compared to rewriting the http server and nbd client.
>
> It also much hard to provide easy to use interface for users
> supporting async I/O.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW