[libvirt] Call to virDomainIsActive hangs forever

[Erik Skultety]

On Thu, Mar 22, 2018 at 06:10:49PM +0200, Mathieu Tarral wrote:
> Hi !
>
> I'm submitting my messages on this mailing list to request a bit of
> help on a case that I have
> where a Python application makes a call to virDomainIsActive, and the
> call never returns.
>
> I have tried to investigate, but as there are no debug symbols for
> libvirt on Debian Stretch,
> i can only have the following GDB backtrace:
>
> (gdb) bt
> #0  pthread_cond_wait@@GLIBC_2.3.2 () at
> ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
> #1  0x00007f49026f5b76 in virCondWait () from /usr/lib/libvirt.so.0
> #2  0x00007f4902808bab in ?? () from /usr/lib/libvirt.so.0
> #3  0x00007f490280a433 in virNetClientSendWithReply () from
> /usr/lib/libvirt.so.0
> #4  0x00007f490280abe2 in virNetClientProgramCall () from /usr/lib/libvirt.so.0
> #5  0x00007f49027e0ea4 in ?? () from /usr/lib/libvirt.so.0
> #6  0x00007f49027ea1bb in ?? () from /usr/lib/libvirt.so.0
> #7  0x00007f49027b0ef3 in virDomainIsActive () from /usr/lib/libvirt.so.0
> #8  0x00007f4902b7fbd0 in libvirt_virDomainIsActive () from
> /usr/lib/python3/dist-packages/libvirtmod.cpython-35m-x86_64-linux-gnu.so
> #9  0x0000558eeec696df in PyCFunction_Call () at ../Objects/methodobject.c:109
>
> The libvirt driver used is QEMU, and i have a specific monitoring in
> place using virtual machine introspection:
> https://github.com/KVM-VMI/kvm-vmi
>
> Now this specific monitoring somehow triggers this bug, and at this
> point, i don't know if
> it's a corner case in the libvirt QEMU driver or not.
> That's why i would like to have your lights on this.
>
> libvirt version: 3.0.0-4
>
> -> Could you tell me where i should look in the code ?

You're probably looking at virLogManagerDomain* methods located in
src/logging/log_manager.c and the wait call is issued from virNetClientIO.

> -> Do you have more information about this virCondWait ? Which
> condition is it waiting for ?
> -> How can i get the symbols without having the recompile libvirt and
> install it system wide, erasing the binaries installed by the package
> ?

To be honest, I think it's always worth debugging a custom built binary from
sources, since the debug symbols shipped via distro package are most likely
generated with optimizations which makes any kind of interactive debugging
painful. The problem is that you're going to built v3.0.0 tag on a new distro,
since new GCCs will complain about a lot of stuff, I looked at the code, tried
a few things, but honestly I didn't see a path where you could get to
virNetClientProgrammCall from virDomainIsActive (since I don't know what the
original call was), so unless you post a full backtrace, we can't help you much
here.
One more thing, yes, you'd have to compile libvirt from git, but you don't have
to install it, in fact, I'm always running upstream daemon, but haven't
installed it system-wide, on the contrary, I've got a distro-provided version
installed in the system which is disabled most of the time.

Regards,
Erik