[libvirt] Call to virDomainIsActive hangs forever

[Daniel P. Berrangé]

On Fri, Mar 23, 2018 at 01:24:46PM +0100, Erik Skultety wrote:
> On Thu, Mar 22, 2018 at 06:10:49PM +0200, Mathieu Tarral wrote:
> > Hi !
> >
> > I'm submitting my messages on this mailing list to request a bit of
> > help on a case that I have
> > where a Python application makes a call to virDomainIsActive, and the
> > call never returns.
> >
> > I have tried to investigate, but as there are no debug symbols for
> > libvirt on Debian Stretch,
> > i can only have the following GDB backtrace:
> >
> > (gdb) bt
> > #0  pthread_cond_wait@@GLIBC_2.3.2 () at
> > ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
> > #1  0x00007f49026f5b76 in virCondWait () from /usr/lib/libvirt.so.0
> > #2  0x00007f4902808bab in ?? () from /usr/lib/libvirt.so.0
> > #3  0x00007f490280a433 in virNetClientSendWithReply () from
> > /usr/lib/libvirt.so.0
> > #4  0x00007f490280abe2 in virNetClientProgramCall () from /usr/lib/libvirt.so.0
> > #5  0x00007f49027e0ea4 in ?? () from /usr/lib/libvirt.so.0
> > #6  0x00007f49027ea1bb in ?? () from /usr/lib/libvirt.so.0
> > #7  0x00007f49027b0ef3 in virDomainIsActive () from /usr/lib/libvirt.so.0
> > #8  0x00007f4902b7fbd0 in libvirt_virDomainIsActive () from
> > /usr/lib/python3/dist-packages/libvirtmod.cpython-35m-x86_64-linux-gnu.so
> > #9  0x0000558eeec696df in PyCFunction_Call () at ../Objects/methodobject.c:109
> >
> > The libvirt driver used is QEMU, and i have a specific monitoring in
> > place using virtual machine introspection:
> > https://github.com/KVM-VMI/kvm-vmi
> >
> > Now this specific monitoring somehow triggers this bug, and at this
> > point, i don't know if
> > it's a corner case in the libvirt QEMU driver or not.
> > That's why i would like to have your lights on this.
> >
> > libvirt version: 3.0.0-4
> >
> > -> Could you tell me where i should look in the code ?
> 
> You're probably looking at virLogManagerDomain* methods located in
> src/logging/log_manager.c and the wait call is issued from virNetClientIO.
> 
> > -> Do you have more information about this virCondWait ? Which
> > condition is it waiting for ?
> > -> How can i get the symbols without having the recompile libvirt and
> > install it system wide, erasing the binaries installed by the package
> > ?
> 
> To be honest, I think it's always worth debugging a custom built binary from
> sources, since the debug symbols shipped via distro package are most likely
> generated with optimizations which makes any kind of interactive debugging
> painful. The problem is that you're going to built v3.0.0 tag on a new distro,
> since new GCCs will complain about a lot of stuff, I looked at the code, tried
> a few things, but honestly I didn't see a path where you could get to
> virNetClientProgrammCall from virDomainIsActive (since I don't know what the
> original call was), so unless you post a full backtrace, we can't help you much
> here.

That's easy - virDomainIsActive() calls into the driver APIs. This is client
side trace, so will get into the remote driver client, which will then call
virNetClientProgramCall.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|