[Libguestfs] More parallelism in VDDK driver (was: Re: CFME-5.11.7.3 Perf. Tests)

[Nir Soffer]

On Wed, Aug 5, 2020 at 5:18 PM Nir Soffer <nsoffer at redhat.com> wrote:
>
> On Wed, Aug 5, 2020 at 5:10 PM Richard W.M. Jones <rjones at redhat.com> wrote:
> >
> > On Wed, Aug 05, 2020 at 04:49:04PM +0300, Nir Soffer wrote:
> > > I see, can change the python plugin to support multiple connections to imageio
> > > using SERIALIZE_REQUESTS?
> > >
> > > The GiL should not limit us since the GIL is released when you write to
> > > imageio socket, and this is likely where the plugin spends most of the time.
> >
> > It's an interesting question and one I'd not really considered at all.
> > Does the Python GIL actively mutex different threads if they call into
> > Python code at the same time?
>
> Yes, only one thread can run python code at the same time, there is no
> way around this. But before calling most syscalls, python releases the GIL.
> When the syscall returns, python acquires the GIL again.
>
> But thread can be switched at any point in time, so must use locking
> in the python if you modify or access shared state.
>
> >  If it's truly a lock, then it should,
> > in which case it should be safe to change the Python plugin to
> > PARALLEL ...
>
> Using multiple threads on the same http socket will not work, you must have
> multiple sockets to imageio.

And we also have to change the rhv-upload-pluign, since we create the
disk and validate
the current hosts in open(). These operation should be done once.

So the flow will be:

1. pre check
2. prepare overlay
3. create disk
4. check if the current host can do the upload
5. run qemu-img convert
6. post (create vm or delete disk on failure)

> > I'll try it out and get back to you.
> >
> > > I'm not sure what triggers using multiple connections in qemu-img and
> > > how it decide how many connections should be used, but we report
> > > the number of writers in OPTIONS:
> > > http://ovirt.github.io/ovirt-imageio/images.html#max_writers
> > >
> > > There is a hard limit in vdsm, because it runs qemu-nbd with
> > > --shared=8, so you should
> > > not use more than 8 connections, they will just block on qemu-nbd forever.
> >
> > It's different for qemu NBD client and server.  Eric told me on IRC a
> > few minutes ago that qemu NBD client does not "yet" support
> > multi-conn.  However it is implemented by qemu-nbd (the server) for
> > readonly connections.
> >
> > Note that multi-conn and --shared are (a bit) different.  Multi-conn
> > is a flag which the server can send to clients to indicate not only
> > that multiple connections are possible, but also that they are come
> > with certain guarantees:
> >
> >   bit 8, NBD_FLAG_CAN_MULTI_CONN: Indicates that the server operates
> >   entirely without cache, or that the cache it uses is shared among
> >   all connections to the given device. In particular, if this flag is
> >   present, then the effects of NBD_CMD_FLUSH and NBD_CMD_FLAG_FUA MUST
> >   be visible across all connections when the server sends its reply to
> >   that command to the client. In the absence of this flag, clients
> >   SHOULD NOT multiplex their commands over more than one connection to
> >   the export.
> >
> > “--shared” limits the number of connections that the server will
> > accept at the same time (like the nbdkit limit filter).  But it would
> > still be possible for an NBD server to accept multiple connections
> > from a single client but not be multi-conn safe.
> >
> > Also NBD lets you multiplex commands on a single connection (which
> > does not require multi-conn or --shared).
> >
> > BTW I found that multi-conn is a big win with the Linux kernel NBD
> > client.
> >
> > > We use 4 connections by default, giving about 100% speed up compared
> > > with one connection. 2 connections give about 80% speed up.  If the
> > > number of connections is related to the number of coroutines, you
> > > can use -m 4 to use 4 coroutines.
> > >
> > > Using -W will improve performance. In this mode every coroutine will
> > > do the I/O when it is ready, instead of waiting for other coroutines
> > > and submit the I/O in the right order.
> >
> > I think Eric might have a better idea about what -m and -W really do
> > for qemu NBD client.  Maybe improve multiplexing?  They don't enable
> > multi-conn :-(
> >
> > Rich.
> >
> > --
> > Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
> > Read my programming and virtualization blog: http://rwmj.wordpress.com
> > virt-df lists disk usage of guests without needing to install any
> > software inside the virtual machine.  Supports Linux and Windows.
> > http://people.redhat.com/~rjones/virt-df/
> >