[Libguestfs] More parallelism in VDDK driver (was: Re: CFME-5.11.7.3 Perf. Tests)

[Nir Soffer]

On Wed, Aug 5, 2020 at 5:10 PM Richard W.M. Jones <rjones at redhat.com> wrote:
>
> On Wed, Aug 05, 2020 at 04:49:04PM +0300, Nir Soffer wrote:
> > I see, can change the python plugin to support multiple connections to imageio
> > using SERIALIZE_REQUESTS?
> >
> > The GiL should not limit us since the GIL is released when you write to
> > imageio socket, and this is likely where the plugin spends most of the time.
>
> It's an interesting question and one I'd not really considered at all.
> Does the Python GIL actively mutex different threads if they call into
> Python code at the same time?

Yes, only one thread can run python code at the same time, there is no
way around this. But before calling most syscalls, python releases the GIL.
When the syscall returns, python acquires the GIL again.

But thread can be switched at any point in time, so must use locking
in the python if you modify or access shared state.

>  If it's truly a lock, then it should,
> in which case it should be safe to change the Python plugin to
> PARALLEL ...

Using multiple threads on the same http socket will not work, you must have
multiple sockets to imageio.

>
> I'll try it out and get back to you.
>
> > I'm not sure what triggers using multiple connections in qemu-img and
> > how it decide how many connections should be used, but we report
> > the number of writers in OPTIONS:
> > http://ovirt.github.io/ovirt-imageio/images.html#max_writers
> >
> > There is a hard limit in vdsm, because it runs qemu-nbd with
> > --shared=8, so you should
> > not use more than 8 connections, they will just block on qemu-nbd forever.
>
> It's different for qemu NBD client and server.  Eric told me on IRC a
> few minutes ago that qemu NBD client does not "yet" support
> multi-conn.  However it is implemented by qemu-nbd (the server) for
> readonly connections.
>
> Note that multi-conn and --shared are (a bit) different.  Multi-conn
> is a flag which the server can send to clients to indicate not only
> that multiple connections are possible, but also that they are come
> with certain guarantees:
>
>   bit 8, NBD_FLAG_CAN_MULTI_CONN: Indicates that the server operates
>   entirely without cache, or that the cache it uses is shared among
>   all connections to the given device. In particular, if this flag is
>   present, then the effects of NBD_CMD_FLUSH and NBD_CMD_FLAG_FUA MUST
>   be visible across all connections when the server sends its reply to
>   that command to the client. In the absence of this flag, clients
>   SHOULD NOT multiplex their commands over more than one connection to
>   the export.
>
> “--shared” limits the number of connections that the server will
> accept at the same time (like the nbdkit limit filter).  But it would
> still be possible for an NBD server to accept multiple connections
> from a single client but not be multi-conn safe.
>
> Also NBD lets you multiplex commands on a single connection (which
> does not require multi-conn or --shared).
>
> BTW I found that multi-conn is a big win with the Linux kernel NBD
> client.
>
> > We use 4 connections by default, giving about 100% speed up compared
> > with one connection. 2 connections give about 80% speed up.  If the
> > number of connections is related to the number of coroutines, you
> > can use -m 4 to use 4 coroutines.
> >
> > Using -W will improve performance. In this mode every coroutine will
> > do the I/O when it is ready, instead of waiting for other coroutines
> > and submit the I/O in the right order.
>
> I think Eric might have a better idea about what -m and -W really do
> for qemu NBD client.  Maybe improve multiplexing?  They don't enable
> multi-conn :-(
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-df lists disk usage of guests without needing to install any
> software inside the virtual machine.  Supports Linux and Windows.
> http://people.redhat.com/~rjones/virt-df/
>