[libvirt] 200ms delay waiting for qemu monitor

[Daniel P. Berrange]

On Thu, Apr 07, 2016 at 09:08:38AM +0200, Michal Privoznik wrote:
> On 06.04.2016 21:45, Richard W.M. Jones wrote:
> > [Thanks to Dan Berrangé for doing the analysis of this one]
> > 
> > I was investigating a 200+ millisecond delay when libvirt starts a
> > qemu guest.  You can see the traces here:
> > 
> > http://oirase.annexia.org/tmp/libvirt.log
> > http://oirase.annexia.org/tmp/libvirtd.log
> > 
> > The delay happens at around 16:52:57.327-6:52:57.528 in the libvirtd log.
> > As you can see the delay is almost precisely 200ms.
> > 
> > Dan found the cause which seems to be this code:
> > 
> > https://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_monitor.c;h=10a6713c06ad36e2055f8144fba00b78630971e5;hb=HEAD#l365
> > 
> > (There are other examples of the same anti-pattern in src/fdstream.c
> > and src/qemu/qemu_agent.c, but it's the particular code above which
> > seems to be causing the delay).
> > 
> > To give you some sense why I regard this as a problem, the TOTAL time
> > taken to launch and shutdown the libguestfs appliance (that includes
> > qemu, BIOS, guest kernel, probing and mouting disks, running the
> > guestfs daemon, and the shutdown process in reverse), without libvirt,
> > is now 900ms.  Libvirt adds about 220ms on top of this.
> > 
> > What can we do about this?  Obviously we could simply reduce the
> > delay, but even if it was set to 20ms, that would be too much (aim is
> > to reduce the whole process from 900ms down to 150ms), and it would
> > also mean that libvirt was essentially polling.
> > 
> > Can we use inotify to detect if the socket has been created?  Seems to
> > create portability problems.

We could do both. ie, we should drop the sleep down to 20ms and poll
for correspondingly larger number of iterations. Then also use inotify
where it is available to avoid having todo the sleep at all on Linux.
The combination of the two get reasonable speed in platform agnostic
case, and perfect speed in the linux case. This would address the
immediate need for libguestfs with existing QEMU versions.

We should none the less work on FD passing for the long term solution.

> > 
> > Dan suggested:
> > 
> > Can we create the socket in libvirtd and pass it to qemu?
> > 
> > Can we pass a file descriptor to qemu?
> 
> Right. This is the goal that we try to reach. Well, not that anybody I
> know of is working on it actively. But the idea would be that libvirt
> would open all the files for qemu and then just pass the FDs.
> As for the monitor - I think qemu counterpart should be fairly easy, but
> we rather confirm with qemu guys. I'm definitely in!

QEMU chardevs have the ability to have a FD passed in, but that
only works by accident and only for file based backend. You can't
currently pass in a FD associated with a UNIX socket. It should
be pretty trivial to enhance QEMU to allow this to be done though.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|