[Libguestfs] [PATCH 6/6] v2v: nbdkit: Match qemu-img number of parallel coroutines

[Richard W.M. Jones]

On Sat, Jan 23, 2021 at 12:45:24AM +0200, Nir Soffer wrote:
> qemu-img is using 8 parallel coroutines by default. I tests up to 16
> parallel coroutines and it seems that 8 gives good results.
> 
> nbdkit uses 16 threads by default. Testing nbdkit with qemu-img show
> that 8 threads give good results.
> 
> I think for rhv upload plugin matching the number of threads to the
> number of connections would be optimal. We need to improve this later to
> use the optimal number for the configured input and output plugins.
> 
> Testing rhv-upload-plugin show small improvement (~6%) in total
> connection time. Compared with last version using single connection, we
> are now 50% faster.
> 
> Results are not stable, we need to test this with bigger images and real
> environment.
> 
> [connection 1 ops, 3.561693 s]
> [dispatch 550 ops, 2.808350 s]
> [write 470 ops, 2.482875 s, 316.06 MiB, 127.30 MiB/s]
> [zero 78 ops, 0.178174 s, 1.26 GiB, 7.05 GiB/s]
> [flush 2 ops, 0.000211 s]
> 
> [connection 1 ops, 3.561724 s]
> [dispatch 543 ops, 2.836738 s]
> [write 472 ops, 2.503561 s, 341.62 MiB, 136.46 MiB/s]
> [zero 69 ops, 0.162465 s, 1.12 GiB, 6.89 GiB/s]
> [flush 2 ops, 0.000181 s]
> 
> [connection 1 ops, 3.566931 s]
> [dispatch 536 ops, 2.807226 s]
> [write 462 ops, 2.508345 s, 326.12 MiB, 130.02 MiB/s]
> [zero 72 ops, 0.141442 s, 1.30 GiB, 9.20 GiB/s]
> [flush 2 ops, 0.000158 s]
> 
> [connection 1 ops, 3.564396 s]
> [dispatch 563 ops, 2.853623 s]
> [write 503 ops, 2.592482 s, 361.44 MiB, 139.42 MiB/s]
> [zero 58 ops, 0.113708 s, 1.01 GiB, 8.88 GiB/s]
> [flush 2 ops, 0.000149 s]
> 
> Signed-off-by: Nir Soffer <nsoffer at redhat.com>
> ---
>  v2v/nbdkit.ml | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/v2v/nbdkit.ml b/v2v/nbdkit.ml
> index 46b20c9d..caa76342 100644
> --- a/v2v/nbdkit.ml
> +++ b/v2v/nbdkit.ml
> @@ -137,6 +137,9 @@ let run_unix cmd =
>    add_arg "--pidfile"; add_arg pidfile;
>    add_arg "--unix"; add_arg sock;
>  
> +  (* Match qemu-img default number of parallel coroutines *)
> +  add_arg "--threads"; add_arg "8";

This will affect all nbdkit instances -- virt-v2v uses nbdkit for both
the input and output sides, eg. when converting from VMware -- so it
would be better to make this configurable as a parameter to the Nbdkit
module (eg. Nbdkit.set_threads, like the way Nbdkit.set_verbose works).

Rich.

>    (* Reduce verbosity in nbdkit >= 1.17.4. *)
>    let version = version (config ()) in
>    if version >= (1, 17, 4) then (
> -- 
> 2.26.2

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW