[Libguestfs] Virt-v2v performance benchmarking part 1 [long]

[Richard W.M. Jones]

This email is about looking at the historical performance of virt-v2v
through the versions over the past year.  The year has seen some very
significant changes to virt-v2v meant to improve modularity,
flexibility and in some cases performance.  The main change that could
affect performance is exposing the disk image pipelines of the input
and output sides, and using nbdcopy (instead of qemu-img convert) to
copy the data.  So for the sake of this email when I talk about
virt-v2v, I also mean nbdkit and libnbd which we've been
co-developing.

My first challenge is to try to make some kind of test case that we
can use for testing conversion and copying performance.  For this I'm
using Linux.  Because Windows conversions are always simpler than
Linux conversions, and because Windows is proprietary, it's neither
necessary or convenient to create a test case around Windows.  I
created a simple and reproducible pair of test cases by doing:

  $ mkdir /var/tmp/3G
  $ dd if=/dev/urandom of=/var/tmp/3G/1 bs=1M count=1024
  $ cp /var/tmp/3G/1 /var/tmp/3G/2
  $ cp /var/tmp/3G/1 /var/tmp/3G/3
  $ virt-builder fedora-35 --copy-in /var/tmp/3G:/var/tmp
  $ virt-builder fedora-35 --copy-in /var/tmp/3G:/var/tmp --format=qcow2
  $ ls -lsh fedora-35.*
  3.9G -rw-r--r--. 1 rjones rjones 6.0G Jan  7 11:33 fedora-35.img
  3.9G -rw-r--r--. 1 rjones rjones 3.9G Jan  7 11:37 fedora-35.qcow2

The test case should be locally reproducible for anyone wanting to try
to reproduce the results.  Also the amount of used disk space can be
easily adjusted by using more or fewer 1GB random files, in case
copying is too fast or slow on local hardware.

The test case could then be copied to VMware or used locally, with
virt-v2v commands like these:

  $ virt-v2v -i disk fedora-35.img -o null 
  $ virt-v2v -i disk fedora-35.qcow2 -o null 
  $ virt-v2v -i disk fedora-35.img -o local -os /var/tmp ; time sync
  $ virt-v2v -i disk fedora-35.qcow2 -o local -os /var/tmp ; time sync
  $ virt-v2v -ic esxi://... -it vddk [etc] fedora-35 [-o ...]

A few remarks:

 - Old virt-v2v handled -o null very efficiently by throwing away the
   data inside qemu-img convert.  Modular virt-v2v runs an
   nbdkit-null-plugin instance on the output side and always copies
   the data to it.  This is always going to be a bit slower.
   Therefore I'm preferring the tests that write to local disk.

 - Old virt-v2v did not sync the data to disk, but nbdcopy always
   does, so with old virt-v2v you have to add a "time sync" command
   and include that in the copying time for fairness.

 - Raw and qcow2 use different code paths in the new virt-v2v, so
   benchmarking both is useful.

 - Copying from VMware can test raw over HTTP, VDDK, SSH which are all
   different code paths.  I'm not benchmarking this at the moment.

 - Output to something other than null/local would also be
   interesting, but I'm not benchmarking that at the moment.

>From the non-verbose output of virt-v2v we can derive 3 times of
interest for benchmarking performance:

  $ virt-v2v -i disk fedora-35.img -o null
  [   1.1] Opening the source
  [   7.3] Inspecting the source
  [  12.4] Checking for sufficient free disk space in the guest
  [  12.4] Converting Fedora Linux 35 (Thirty Five) to run on KVM
  [  56.6] Mapping filesystem data to avoid copying unused and blank areas
  [  59.8] Closing the overlay
  [  60.3] Assigning disks to buses
  [  60.3] Checking if the guest needs BIOS or UEFI to boot
  [  61.3] Copying disk 1/1
  █ 100% [****************************************]
  [  62.4] Creating output metadata
  [  62.4] Finishing off

=> Conversion time: 56.6 - 12.4 = 44.2
   Mapping time:    59.8 - 56.6 =  3.2
   Copying time:    62.4 - 61.3 =  1.1 (+ sync time for old virt-v2v)

The "mapping time" is the time taken to run fstrim over the source.  I
include this because it regressed quite a lot just after we
modularised virt-v2v and was later fixed, although it's still not as
fast as old virt-v2v.

Now what I am doing is running some tests over every virt-v2v we have
released in CentOS 9 Stream over the last year.  As mentioned above I
also need to do libnbd & nbdkit since we were co-developing those, so
I'm actually using RHEL 9 composes for this.  However I am leaving
libguestfs, libvirt and qemu at the newest versions throughout this
because older versions had various problems running on current
hardware which have since been fixed (my theory is that the versions
of these should not make any significant difference).

Also:

  $ virt-v2v --version
  virt-v2v: symbol lookup error: virt-v2v: undefined symbol: guestfs_list_9p

This was easily fixed by adding a trivial LD_PRELOAD library which
just loaded phony symbols for guestfs_list_9p & guestfs_mount_9p_argv
(both dropped in later libguestfs).

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v