[libvirt] Call to virDomainIsActive hangs forever

Tue Mar 27 13:04:33 UTC 2018

> Are you sure this isa  different thread ? It looks identical to the first
> stack trace you give above.

Yes, the first one is calling libvirtmod.virDomainGetState
and the second one libvirtmod.virDomainIsActive.

> Interesting. This is an identical stack trace - so we have 2 python
> threads both calling virDomainIsActive(). Nothing wrong with that
> per-se - we support multithreaded usage like this.

virDomainGetState()
and
virDomainIsActive()

> Can you confirm there are no other threads running libvirt code
> in your python app ?  Did you have any thread running the libvirt
> event loop perhaps ?

Actually i found 2 others threads in Python app calling libvirt.

So, as a recap:

Thread 1 - calling virDomainGetState
(gdb) py-bt
Traceback (most recent call first):
  <built-in method virDomainGetState of module object at remote 0x7f508e6b1278>
  File "/usr/lib/python3/dist-packages/libvirt.py", line 2551, in state
    ret = libvirtmod.virDomainGetState(self._o, flags)

(gdb) bt
pthread_cond_wait@@GLIBC_2.3.2 () at
../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f508dfe2b86 in virCondWait (c=c at entry=0x55a1fe420728,
m=m at entry=0x55a1fde4d2c0) at ../../../src/util/virthread.c:154
#2  0x00007f508e0f5bbb in virNetClientIO (thiscall=0x55a1fe420710,
client=0x55a1fde4d2b0) at ../../../src/rpc/virnetclient.c:1894
#3  virNetClientSendInternal (client=client at entry=0x55a1fde4d2b0,
msg=msg at entry=0x55a1fdd798f0, expectReply=expectReply at entry=true,
nonBlock=nonBlock at entry=false) at ../../../src/rpc/virnetclient.c:2116
#4  0x00007f508e0f7443 in virNetClientSendWithReply
(client=client at entry=0x55a1fde4d2b0, msg=msg at entry=0x55a1fdd798f0) at
../../../src/rpc/virnetclient.c:2144
#5  0x00007f508e0f7bf2 in virNetClientProgramCall
(prog=prog at entry=0x55a1fdff0f90, client=client at entry=0x55a1fde4d2b0,
serial=serial at entry=108, proc=proc at entry=212, noutfds=noutfds at entry=0,
outfds=outfds at entry=0x0, ninfds=0x0,
    infds=0x0, args_filter=0x7f508e0f13e0
<xdr_remote_domain_get_state_args>, args=0x7ffd1518cfd0,
ret_filter=0x7f508e0f1410 <xdr_remote_domain_get_state_ret>,
ret=0x7ffd1518cfc8) at ../../../src/rpc/virnetclientprogram.c:329
#6  0x00007f508e0cdeb4 in callFull (priv=priv at entry=0x55a1fe5ac460,
flags=flags at entry=0, fdin=fdin at entry=0x0, fdinlen=fdinlen at entry=0,
fdout=fdout at entry=0x0, fdoutlen=fdoutlen at entry=0x0, proc_nr=212,
    args_filter=0x7f508e0f13e0 <xdr_remote_domain_get_state_args>,
args=0x7ffd1518cfd0 "`k.\376\241U", ret_filter=0x7f508e0f1410
<xdr_remote_domain_get_state_ret>, ret=0x7ffd1518cfc8 "",
conn=<optimized out>)
    at ../../../src/remote/remote_driver.c:6636
#7  0x00007f508e0d7f90 in call (conn=<optimized out>,
ret=0x7ffd1518cfc8 "", ret_filter=<optimized out>, args=0x7ffd1518cfd0
"`k.\376\241U", args_filter=<optimized out>, proc_nr=212, flags=0,
priv=0x55a1fe5ac460)
    at ../../../src/remote/remote_driver.c:6658
#8  remoteDomainGetState (domain=0x55a1fe212da0, state=0x7ffd1518d0a4,
reason=0x7ffd1518d0a8, flags=0) at
../../../src/remote/remote_driver.c:2458
#9  0x00007f508e08e248 in virDomainGetState
(domain=domain at entry=0x55a1fe212da0, state=state at entry=0x7ffd1518d0a4,
reason=reason at entry=0x7ffd1518d0a8, flags=0) at
../../../src/libvirt-domain.c:2495
#10 0x00007f508e466f28 in libvirt_virDomainGetState (self=<optimized
out>, args=<optimized out>) at libvirt-override.c:2539
#11 0x000055a1fb4cb6df in PyCFunction_Call () at ../Objects/methodobject.c:109

Thread 32 - calling virDomainIsActive
(gdb) py-bt
Traceback (most recent call first):
  <built-in method virDomainIsActive of module object at remote 0x7f508e6b1278>
  File "/usr/lib/python3/dist-packages/libvirt.py", line 1340, in isActive
    ret = libvirtmod.virDomainIsActive(self._o)

(gdb) bt
#0  pthread_cond_wait@@GLIBC_2.3.2 () at
../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f508dfe2b86 in virCondWait (c=c at entry=0x7f4ff051fdc8,
m=m at entry=0x55a1fde4d2c0) at ../../../src/util/virthread.c:154
#2  0x00007f508e0f5bbb in virNetClientIO (thiscall=0x7f4ff051fdb0,
client=0x55a1fde4d2b0) at ../../../src/rpc/virnetclient.c:1894
#3  virNetClientSendInternal (client=client at entry=0x55a1fde4d2b0,
msg=msg at entry=0x7f4ff051fa10, expectReply=expectReply at entry=true,
nonBlock=nonBlock at entry=false) at ../../../src/rpc/virnetclient.c:2116
#4  0x00007f508e0f7443 in virNetClientSendWithReply
(client=client at entry=0x55a1fde4d2b0, msg=msg at entry=0x7f4ff051fa10) at
../../../src/rpc/virnetclient.c:2144
#5  0x00007f508e0f7bf2 in virNetClientProgramCall
(prog=prog at entry=0x55a1fdff0f90, client=client at entry=0x55a1fde4d2b0,
serial=serial at entry=107, proc=proc at entry=150, noutfds=noutfds at entry=0,
outfds=outfds at entry=0x0, ninfds=0x0,
    infds=0x0, args_filter=0x7f508e0efe70
<xdr_remote_domain_is_active_args>, args=0x7f4fff7fd040,
ret_filter=0x7f508e0efe90 <xdr_remote_domain_is_active_ret>,
ret=0x7f4fff7fd03c) at ../../../src/rpc/virnetclientprogram.c:329
#6  0x00007f508e0cdeb4 in callFull (priv=priv at entry=0x55a1fe5ac460,
flags=flags at entry=0, fdin=fdin at entry=0x0, fdinlen=fdinlen at entry=0,
fdout=fdout at entry=0x0, fdoutlen=fdoutlen at entry=0x0, proc_nr=150,
    args_filter=0x7f508e0efe70 <xdr_remote_domain_is_active_args>,
args=0x7f4fff7fd040 "`k.\376\241U", ret_filter=0x7f508e0efe90
<xdr_remote_domain_is_active_ret>, ret=0x7f4fff7fd03c "",
conn=<optimized out>)
    at ../../../src/remote/remote_driver.c:6636
#7  0x00007f508e0d71cb in call (conn=<optimized out>,
ret=0x7f4fff7fd03c "", ret_filter=<optimized out>, args=0x7f4fff7fd040
"`k.\376\241U", args_filter=<optimized out>, proc_nr=150, flags=0,
priv=0x55a1fe5ac460)
    at ../../../src/remote/remote_driver.c:6658
#8  remoteDomainIsActive (dom=0x55a1fe212da0) at
../../../src/remote/remote_client_bodies.h:2842
#9  0x00007f508e09df03 in virDomainIsActive
(dom=dom at entry=0x55a1fe212da0) at ../../../src/libvirt-domain.c:8467
#10 0x00007f508e46cbd0 in libvirt_virDomainIsActive (self=<optimized
out>, args=<optimized out>) at build/libvirt.c:1288
#11 0x000055a1fb4cb6df in PyCFunction_Call () at ../Objects/methodobject.c:109

Thread 35 - calling virDomainGetXMLDesc
(gdb) py-bt
  <built-in method virDomainGetXMLDesc of module object at remote
0x7f508e6b1278>
  File "/usr/lib/python3/dist-packages/libvirt.py", line 491, in XMLDesc
    ret = libvirtmod.virDomainGetXMLDesc(self._o, flags)

(gdb) bt
#0  pthread_sigmask (how=how at entry=0, newmask=<optimized out>,
newmask at entry=0x7f4ffd7f8d10, oldmask=oldmask at entry=0x7f4ffd7f8c90) at
../sysdeps/unix/sysv/linux/pthread_sigmask.c:50
#1  0x00007f508e0f52fa in virNetClientIOEventLoop
(client=client at entry=0x55a1fde4d2b0,
thiscall=thiscall at entry=0x7f4fe005a350) at
../../../src/rpc/virnetclient.c:1659
#2  0x00007f508e0f5a16 in virNetClientIO (thiscall=0x7f4fe005a350,
client=0x55a1fde4d2b0) at ../../../src/rpc/virnetclient.c:1944
#3  virNetClientSendInternal (client=client at entry=0x55a1fde4d2b0,
msg=msg at entry=0x7f4fe0031f50, expectReply=expectReply at entry=true,
nonBlock=nonBlock at entry=false) at ../../../src/rpc/virnetclient.c:2116
#4  0x00007f508e0f7443 in virNetClientSendWithReply
(client=client at entry=0x55a1fde4d2b0, msg=msg at entry=0x7f4fe0031f50) at
../../../src/rpc/virnetclient.c:2144
#5  0x00007f508e0f7bf2 in virNetClientProgramCall
(prog=prog at entry=0x55a1fdff0f90, client=client at entry=0x55a1fde4d2b0,
serial=serial at entry=105, proc=proc at entry=14, noutfds=noutfds at entry=0,
outfds=outfds at entry=0x0, ninfds=0x0,
    infds=0x0, args_filter=0x7f508e0ecba0
<xdr_remote_domain_get_xml_desc_args>, args=0x7f4ffd7f8fe0,
ret_filter=0x7f508e0ecbd0 <xdr_remote_domain_get_xml_desc_ret>,
ret=0x7f4ffd7f8fd8)
    at ../../../src/rpc/virnetclientprogram.c:329
#6  0x00007f508e0cdeb4 in callFull (priv=priv at entry=0x55a1fe5ac460,
flags=flags at entry=0, fdin=fdin at entry=0x0, fdinlen=fdinlen at entry=0,
fdout=fdout at entry=0x0, fdoutlen=fdoutlen at entry=0x0, proc_nr=14,
    args_filter=0x7f508e0ecba0 <xdr_remote_domain_get_xml_desc_args>,
args=0x7f4ffd7f8fe0 "`k.\376\241U", ret_filter=0x7f508e0ecbd0
<xdr_remote_domain_get_xml_desc_ret>, ret=0x7f4ffd7f8fd8 "",
conn=<optimized out>)
    at ../../../src/remote/remote_driver.c:6636
#7  0x00007f508e0d7b58 in call (conn=<optimized out>,
ret=0x7f4ffd7f8fd8 "", ret_filter=<optimized out>, args=0x7f4ffd7f8fe0
"`k.\376\241U", args_filter=<optimized out>, proc_nr=14, flags=0,
priv=0x55a1fe5ac460)
    at ../../../src/remote/remote_driver.c:6658
#8  remoteDomainGetXMLDesc (dom=<optimized out>, flags=0) at
../../../src/remote/remote_client_bodies.h:2698
#9  0x00007f508e08f5c1 in virDomainGetXMLDesc
(domain=domain at entry=0x55a1fe212da0, flags=0) at
../../../src/libvirt-domain.c:2592
#10 0x00007f508e46c8c0 in libvirt_virDomainGetXMLDesc (self=<optimized
out>, args=<optimized out>) at build/libvirt.c:1212
#11 0x000055a1fb4cb6df in PyCFunction_Call () at ../Objects/methodobject.c:109

Thread 36 - calling virDomainQemuMonitorCommand
(gdb) py-bt
  <built-in method virDomainQemuMonitorCommand of module object at
remote 0x7f506e317908>
  File "/usr/lib/python3/dist-packages/libvirt_qemu.py", line 71, in
qemuMonitorCommand
    ret = libvirtmod_qemu.virDomainQemuMonitorCommand(domain._o, cmd, flags)

(gdb)
#0  pthread_cond_wait@@GLIBC_2.3.2 () at
../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f508dfe2b86 in virCondWait (c=c at entry=0x7f4fe80da208,
m=m at entry=0x55a1fde4d2c0) at ../../../src/util/virthread.c:154
#2  0x00007f508e0f5bbb in virNetClientIO (thiscall=0x7f4fe80da1f0,
client=0x55a1fde4d2b0) at ../../../src/rpc/virnetclient.c:1894
#3  virNetClientSendInternal (client=client at entry=0x55a1fde4d2b0,
msg=msg at entry=0x7f4fe80da160, expectReply=expectReply at entry=true,
nonBlock=nonBlock at entry=false) at ../../../src/rpc/virnetclient.c:2116
#4  0x00007f508e0f7443 in virNetClientSendWithReply
(client=client at entry=0x55a1fde4d2b0, msg=msg at entry=0x7f4fe80da160) at
../../../src/rpc/virnetclient.c:2144
#5  0x00007f508e0f7bf2 in virNetClientProgramCall
(prog=prog at entry=0x55a1fe2b5ae0, client=client at entry=0x55a1fde4d2b0,
serial=serial at entry=106, proc=proc at entry=1, noutfds=noutfds at entry=0,
outfds=outfds at entry=0x0, ninfds=0x0,
    infds=0x0, args_filter=0x7f508e0f34f0
<xdr_qemu_domain_monitor_command_args>, args=0x7f4ffe7fb1a0,
ret_filter=0x7f508e0f3540 <xdr_qemu_domain_monitor_command_ret>,
ret=0x7f4ffe7fb198)
    at ../../../src/rpc/virnetclientprogram.c:329
#6  0x00007f508e0cdeb4 in callFull (priv=priv at entry=0x55a1fe5ac460,
flags=flags at entry=1, fdin=fdin at entry=0x0, fdinlen=fdinlen at entry=0,
fdout=fdout at entry=0x0, fdoutlen=fdoutlen at entry=0x0, proc_nr=1,
    args_filter=0x7f508e0f34f0 <xdr_qemu_domain_monitor_command_args>,
args=0x7f4ffe7fb1a0 "`k.\376\241U", ret_filter=0x7f508e0f3540
<xdr_qemu_domain_monitor_command_ret>, ret=0x7f4ffe7fb198 "",
conn=<optimized out>)
    at ../../../src/remote/remote_driver.c:6636
#7  0x00007f508e0dda39 in call (conn=<optimized out>, ret=<optimized
out>, ret_filter=<optimized out>, args=<optimized out>,
args_filter=<optimized out>, proc_nr=1, flags=1, priv=0x55a1fe5ac460)
    at ../../../src/remote/remote_driver.c:6658
#8  remoteDomainQemuMonitorCommand (domain=0x55a1fe212da0,
cmd=<optimized out>, result=0x7f4ffe7fb290, flags=<optimized out>) at
../../../src/remote/remote_driver.c:5993
#9  0x00007f506deaeda5 in virDomainQemuMonitorCommand
(domain=domain at entry=0x55a1fe212da0, cmd=0x7f506c9b3690 "mouse_move
385 212", result=result at entry=0x7f4ffe7fb290, flags=1) at
../../../src/libvirt-qemu.c:86
#10 0x00007f506e0b50cc in libvirt_qemu_virDomainQemuMonitorCommand
(self=<optimized out>, args=<optimized out>) at
libvirt-qemu-override.c:146
#11 0x000055a1fb4cb6df in PyCFunction_Call () at ../Objects/methodobject.c:109

Thank you.

-- 
Mathieu Tarral