[libvirt] udevadm settle can take too long

[Jim Paris]

Guido Günther wrote:
> Hi,
> On Sun, Apr 22, 2012 at 02:41:54PM -0400, Jim Paris wrote:
> > Hi,
> > 
> > http://bugs.debian.org/663931 is a bug I'm hitting, where virt-manager
> > times out on the initial connection to libvirt.
> 
> I reassigned the bug back to libvirt. I still wonder what triggers this
> though for some users but not for others?
> Cheers,
>  -- Guido

On all of my machines, virt-manager hangs if "udevadm settle" hangs.
You can use the program I posted at that bug report to trigger the
udevadm problem (it can be undone by restarting udev).

Libvirtd only triggers the udevadm problem at startup, through its use
of network namespaces while probing lxc.  If anything else generates
uevents after that point, then the udevadm problem usually goes away.
For example, any module loads, hardware events (ejecting a CD, closing
a laptop lid, etc), or bringing up or down network interfaces (which
libvirt would typically do by itself when starting a new domain).
So most users might just avoid it through luck.  But if you manually
restart libvirtd right before trying virt-manager, you'll probably see
it too.

Thanks,
-jim

> > The basic problem is that, while checking storage volumes,
> > virt-manager causes libvirt to call "udevadm settle".  There's an
> > interaction where libvirt's earlier use of network namespaces (to probe
> > LXC features) had caused some uevents to be sent that get filtered out
> > before they reach udev.  This confuses "udevadm settle" a bit, and so
> > it sits there waiting for a 2-3 minute built-in timeout before returning.
> > Eventually libvirtd prints:
> >   2012-04-22 18:22:18.678+0000: 30503: warning : virKeepAliveTimer:182 : No response from client 0x7feec4003630 after 5 keepalive messages in 30 seconds
> > and virt-manager prints:
> >   2012-04-22 18:22:18.931+0000: 30647: warning : virKeepAliveSend:128 : Failed to send keepalive response to client 0x25004e0
> > and the connection gets dropped.
> > 
> > One workaround could be to specify a shorter timeout when doing the
> > settle.  The patch appended below allows virt-manager to work,
> > although the connection still has to wait for the 10 second timeout
> > before it succeeds.  I don't know what a better solution would be,
> > though.  It seems the udevadm behavior might not be considered a bug
> > from the udev/kernel point of view:
> >   https://lkml.org/lkml/2012/4/22/60
> > 
> > I'm using Linux 3.2.14 with libvirt 0.9.11.  You can trigger the
> > udevadm issue using a program I posted at the Debian bug report link
> > above.
> > 
> > -jim
> > 
> > >From 17e5b9ebab76acb0d711e8bc308023372fbc4180 Mon Sep 17 00:00:00 2001
> > From: Jim Paris <jim at jtan.com>
> > Date: Sun, 22 Apr 2012 14:35:47 -0400
> > Subject: [PATCH] shorten udevadmin settle timeout
> > 
> > Otherwise, udevadmin settle can take so long that connections from
> > e.g. virt-manager will get closed.
> > ---
> >  src/util/util.c |    4 ++--
> >  1 files changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/src/util/util.c b/src/util/util.c
> > index 6e041d6..dfe458e 100644
> > --- a/src/util/util.c
> > +++ b/src/util/util.c
> > @@ -2593,9 +2593,9 @@ virFileFindMountPoint(const char *type ATTRIBUTE_UNUSED)
> >  void virFileWaitForDevices(void)
> >  {
> >  # ifdef UDEVADM
> > -    const char *const settleprog[] = { UDEVADM, "settle", NULL };
> > +    const char *const settleprog[] = { UDEVADM, "settle", "--timeout", "10", NULL };
> >  # else
> > -    const char *const settleprog[] = { UDEVSETTLE, NULL };
> > +    const char *const settleprog[] = { UDEVSETTLE, "--timeout", "10", NULL };
> >  # endif
> >      int exitstatus;
> >  
> > -- 
> > 1.7.7
> > 
> > 
> > --
> > libvir-list mailing list
> > libvir-list at redhat.com
> > https://www.redhat.com/mailman/listinfo/libvir-list
> >