[Libguestfs] [PATCH 6/6] v2v: nbdkit: Match qemu-img number of parallel coroutines

[Nir Soffer]

On Sat, Jan 23, 2021, 08:42 Richard W.M. Jones <rjones at redhat.com> wrote:

> On Sat, Jan 23, 2021 at 12:45:24AM +0200, Nir Soffer wrote:
> > qemu-img is using 8 parallel coroutines by default. I tests up to 16
> > parallel coroutines and it seems that 8 gives good results.
> >
> > nbdkit uses 16 threads by default. Testing nbdkit with qemu-img show
> > that 8 threads give good results.
> >
> > I think for rhv upload plugin matching the number of threads to the
> > number of connections would be optimal. We need to improve this later to
> > use the optimal number for the configured input and output plugins.
> >
> > Testing rhv-upload-plugin show small improvement (~6%) in total
> > connection time. Compared with last version using single connection, we
> > are now 50% faster.
> >
> > Results are not stable, we need to test this with bigger images and real
> > environment.
> >
> > [connection 1 ops, 3.561693 s]
> > [dispatch 550 ops, 2.808350 s]
> > [write 470 ops, 2.482875 s, 316.06 MiB, 127.30 MiB/s]
> > [zero 78 ops, 0.178174 s, 1.26 GiB, 7.05 GiB/s]
> > [flush 2 ops, 0.000211 s]
> >
> > [connection 1 ops, 3.561724 s]
> > [dispatch 543 ops, 2.836738 s]
> > [write 472 ops, 2.503561 s, 341.62 MiB, 136.46 MiB/s]
> > [zero 69 ops, 0.162465 s, 1.12 GiB, 6.89 GiB/s]
> > [flush 2 ops, 0.000181 s]
> >
> > [connection 1 ops, 3.566931 s]
> > [dispatch 536 ops, 2.807226 s]
> > [write 462 ops, 2.508345 s, 326.12 MiB, 130.02 MiB/s]
> > [zero 72 ops, 0.141442 s, 1.30 GiB, 9.20 GiB/s]
> > [flush 2 ops, 0.000158 s]
> >
> > [connection 1 ops, 3.564396 s]
> > [dispatch 563 ops, 2.853623 s]
> > [write 503 ops, 2.592482 s, 361.44 MiB, 139.42 MiB/s]
> > [zero 58 ops, 0.113708 s, 1.01 GiB, 8.88 GiB/s]
> > [flush 2 ops, 0.000149 s]
> >
> > Signed-off-by: Nir Soffer <nsoffer at redhat.com>
> > ---
> >  v2v/nbdkit.ml | 3 +++
> >  1 file changed, 3 insertions(+)
> >
> > diff --git a/v2v/nbdkit.ml b/v2v/nbdkit.ml
> > index 46b20c9d..caa76342 100644
> > --- a/v2v/nbdkit.ml
> > +++ b/v2v/nbdkit.ml
> > @@ -137,6 +137,9 @@ let run_unix cmd =
> >    add_arg "--pidfile"; add_arg pidfile;
> >    add_arg "--unix"; add_arg sock;
> >
> > +  (* Match qemu-img default number of parallel coroutines *)
> > +  add_arg "--threads"; add_arg "8";
>
> This will affect all nbdkit instances -- virt-v2v uses nbdkit for both
> the input and output sides, eg. when converting from VMware -- so it
> would be better to make this configurable as a parameter to the Nbdkit
> module (eg. Nbdkit.set_threads, like the way Nbdkit.set_verbose works).
>

Agree



> Rich.
>
> >    (* Reduce verbosity in nbdkit >= 1.17.4. *)
> >    let version = version (config ()) in
> >    if version >= (1, 17, 4) then (
> > --
> > 2.26.2
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> Fedora Windows cross-compiler. Compile Windows programs, test, and
> build Windows installers. Over 100 libraries supported.
> http://fedoraproject.org/wiki/MinGW
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libguestfs/attachments/20210123/13c17d64/attachment.htm>