[Libguestfs] Quantifying libvirt errors in launching the libguestfs appliance

[Richard W.M. Jones]

As people may know, we frequently encounter errors caused by libvirt
when running the libguestfs appliance.

I wanted to find out exactly how frequently these happen and classify
the errors, so I ran the 'virt-df' tool overnight 1700 times.  This
tool runs several parallel qemu:///session libvirt connections both
creating a short-lived appliance guest.

Note that I have added Cole's patch to fix https://bugzilla.redhat.com/1271183
"XML-RPC error : Cannot write data: Transport endpoint is not connected"

Results:

The test failed 538 times (32% of the time), which is pretty dismal.
To be fair, virt-df is aggressive about how it launches parallel
libvirt connections.  Most other virt-* tools use only a single
libvirt connection and are consequently more reliable.

Of the failures, 518 (96%) were of the form:

  process exited while connecting to monitor: qemu: could not load kernel '/home/rjones/d/libguestfs/tmp/.guestfs-1000/appliance.d/kernel': Permission denied

which is https://bugzilla.redhat.com/921135 or maybe
https://bugzilla.redhat.com/1269975.  It's not clear to me if these
bugs have different causes, but if they do then potentially we're
seeing a mix of both since my test has no way to distinguish them.

19 of the failures (4%) were of the form:

  process exited while connecting to monitor: fread() failed

which I believe is a previously unknown bug.  I have filed it as
https://bugzilla.redhat.com/1298122

Finally there was 1 failure:

  Unable to read from monitor: Connection reset by peer

which I believe is also a new bug.  I have filed it as
https://bugzilla.redhat.com/1298124

I would be good if libvirt could routinely test the case of multiple
parallel launches of qemu:///session, since it still contains bugs
even after Cole's fixes.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top