[Libguestfs] [libvirt] Quantifying libvirt errors in launching the libguestfs appliance

[Jiri Denemark]

On Wed, Jan 13, 2016 at 16:25:14 +0100, Martin Kletzander wrote:
> On Wed, Jan 13, 2016 at 10:18:42AM +0000, Richard W.M. Jones wrote:
> >As people may know, we frequently encounter errors caused by libvirt
> >when running the libguestfs appliance.
> >
> >I wanted to find out exactly how frequently these happen and classify
> >the errors, so I ran the 'virt-df' tool overnight 1700 times.  This
> >tool runs several parallel qemu:///session libvirt connections both
> >creating a short-lived appliance guest.
> >
> >Note that I have added Cole's patch to fix https://bugzilla.redhat.com/1271183
> >"XML-RPC error : Cannot write data: Transport endpoint is not connected"
> >
> >Results:
> >
> >The test failed 538 times (32% of the time), which is pretty dismal.
> >To be fair, virt-df is aggressive about how it launches parallel
> >libvirt connections.  Most other virt-* tools use only a single
> >libvirt connection and are consequently more reliable.
> >
> >Of the failures, 518 (96%) were of the form:
> >
> >  process exited while connecting to monitor: qemu: could not load kernel '/home/rjones/d/libguestfs/tmp/.guestfs-1000/appliance.d/kernel': Permission denied
> >
> >which is https://bugzilla.redhat.com/921135 or maybe
> >https://bugzilla.redhat.com/1269975.  It's not clear to me if these
> >bugs have different causes, but if they do then potentially we're
> >seeing a mix of both since my test has no way to distinguish them.
> >
> 
> It looks to me as the same problem.  And as the same problem we were
> talking about bunch of time and, apparently, didn't get to a conclusion.
> 
> For each of the kernels, libvirt labels them (with both DAC and selinux
> labels), then proceeds to launching qemu.  If this is done parallel, the
> race is pretty obvious.  Could you remind me why you couldn't use
> <seclabel model='none'/> or <seclabel relabel='no'/> or something that
> would mitigate this?  If we cannot use this, then we need to implement
> the <seclabel/> element for kernel and initrd.

Hmm, can't we just label kernel and initrd files the same way we label
<shareable/> disk images, i.e., non-exclusive label so that all QEMU
process can access them and avoid removing the label once a domain
disappears?

Jirka