<div dir="ltr"><br>Hi, all,<br><div class="gmail_quote"><div dir="ltr"><div><br></div><div>We were running OpenStack with Ubuntu and libvirt 0.9.10. We found that libvirt monitor command not working well. </div><div>There were a lot of error in libvirtd.log like this</div>
<div>2013-02-07 06:07:39.000+0000: 18112: error : qemuDomainObjBeginJobInternal:773 : Timed out during operation: cannot acquire state change lock</div><div><br></div><div>We dig into libvirtd by strace and find one of the thread only have the following command</div>
<div>futex(0x7f69ac0ec0ec, FUTEX_WAIT_PRIVATE, 2717, NULL<br></div><div><br></div><div>It seems this thread waiting for reply but nothing came back thus other threads would wait for it. We also saw there is a function called virCondWaitUntil(). Is it safe for us to modify the code from virCondWait() to virCondWaitUntil() to prevent such deadlock scenario? Thanks.</div>
<div><br></div><div>Following is the gdb -p 'libvirt.pid' and 'thread id' and 'bt full'</div><div><div>#0 0x00007f69c8c1dd84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0</div>
<div>No symbol table info available.</div><div>#1 0x00007f69c9ee884a in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117</div><div> ret = <optimized out></div><div>
#2 0x000000000049c749 in qemuMonitorSend (mon=0x7f69ac0ec0c0, msg=<optimized out>) at qemu/qemu_monitor.c:826</div><div> ret = -1</div><div> __func__ = "qemuMonitorSend"</div><div> __FUNCTION__ = "qemuMonitorSend"</div>
<div>#3 0x00000000004ac8ed in qemuMonitorJSONCommandWithFd (mon=0x7f69ac0ec0c0, cmd=0x7f6998028280, scm_fd=-1, reply=0x7f69c57829f8)</div><div> at qemu/qemu_monitor_json.c:230</div><div> ret = -1</div><div> msg = {txFD = -1, txBuffer = 0x7f69980e9b00 "{\"execute\":\"query-balloon\",\"id\":\"libvirt-1359\"}\r\n", txOffset = 49, txLength = 49,</div>
<div> rxBuffer = 0x0, rxLength = 0, rxObject = 0x0, finished = false, passwordHandler = 0, passwordOpaque = 0x0}</div><div> cmdstr = 0x7f69980ef2f0 "{\"execute\":\"query-balloon\",\"id\":\"libvirt-1359\"}"</div>
<div> id = 0x7f69980b0a20 "libvirt-1359"</div><div> exe = <optimized out></div><div> __FUNCTION__ = "qemuMonitorJSONCommandWithFd"</div><div> __func__ = "qemuMonitorJSONCommandWithFd"</div>
<div>#4 0x00000000004ae794 in qemuMonitorJSONGetBalloonInfo (mon=0x7f69ac0ec0c0, currmem=0x7f69c5782a48) at qemu/qemu_monitor_json.c:1190</div><div> ret = <optimized out></div><div> cmd = 0x7f6998028280</div>
<div> reply = 0x0</div><div> __FUNCTION__ = "qemuMonitorJSONGetBalloonInfo"</div><div>#5 0x0000000000457451 in qemudDomainGetInfo (dom=<optimized out>, info=0x7f69c5782b50) at qemu/qemu_driver.c:2181</div>
<div> priv = 0x7f69a0093b00</div><div> driver = 0x7f69b80ca8e0</div><div> vm = 0x7f69a0093370 </div><div> ret = -1</div><div> err = <optimized out></div><div> balloon = <optimized out></div>
<div> __FUNCTION__ = "qemudDomainGetInfo"</div><div>#6 0x00007f69c9f63eda in virDomainGetInfo (domain=0x7f69980e3650, info=0x7f69c5782b50) at libvirt.c:4230</div><div> ret = <optimized out></div>
<div> conn = <optimized out></div><div> __func__ = "virDomainGetInfo"</div><div> __FUNCTION__ = "virDomainGetInfo"</div><div>#7 0x0000000000439bca in remoteDispatchDomainGetInfo (ret=0x7f6998000c20, args=<optimized out>, rerr=0x7f69c5782c50, client=0x157e730,</div>
<div> server=<optimized out>, msg=<optimized out>) at remote_dispatch.h:1640</div><div> rv = -1</div><div> tmp = {state = 1 '\001', maxMem = 2097152, memory = 0, nrVirtCpu = 0, cpuTime = 5981880000000}</div>
<div> dom = 0x7f69980e3650</div><div> priv = <optimized out></div><div>#8 remoteDispatchDomainGetInfoHelper (server=<optimized out>, client=0x157e730, msg=<optimized out>, rerr=0x7f69c5782c50, args=<optimized out>, </div>
<div> ret=0x7f6998000c20) at remote_dispatch.h:1616</div><div> __func__ = "remoteDispatchDomainGetInfoHelper"</div><div>#9 0x00007f69c9fbb915 in virNetServerProgramDispatchCall (msg=0x1689cc0, client=0x157e730, server=0x1577c90, prog=0x15825d0)</div>
<div> at rpc/virnetserverprogram.c:416</div><div> ret = 0x7f6998000c20 ""</div><div> rv = -1</div><div> i = <optimized out></div><div> arg = 0x7f6998027950 "\360e\n\230i\177"</div>
<div> dispatcher = 0x73de40</div><div> rerr = {code = 0, domain = 0, message = 0x0, level = 0, dom = 0x0, str1 = 0x0, str2 = 0x0, str3 = 0x0, int1 = 0, int2 = 0, net = 0x0}</div><div>#10 virNetServerProgramDispatch (prog=0x15825d0, server=0x1577c90, client=0x157e730, msg=0x1689cc0) at rpc/virnetserverprogram.c:289</div>
<div> ret = -1</div><div> rerr = {code = 0, domain = 0, message = 0x0, level = 0, dom = 0x0, str1 = 0x0, str2 = 0x0, str3 = 0x0, int1 = 0, int2 = 0, net = 0x0}</div><div> __func__ = "virNetServerProgramDispatch"</div>
<div> __FUNCTION__ = "virNetServerProgramDispatch"</div><div>#11 0x00007f69c9fb6461 in virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x1577c90) at rpc/virnetserver.c:164</div><div> srv = 0x1577c90</div>
<div> job = 0x155dfa0</div><div> __func__ = "virNetServerHandleJob"</div><div>#12 0x00007f69c9ee8e3e in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:144</div><div> data = 0x0</div>
<div> pool = 0x1577d80</div><div> cond = 0x1577de0</div><div> priority = false</div><div> job = 0x162dd20</div><div>#13 0x00007f69c9ee84e6 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161</div>
<div> args = 0x0</div><div> local = {func = 0x7f69c9ee8d00 <virThreadPoolWorker>, opaque = 0x1559f90}</div><div>#14 0x00007f69c8c19e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0</div>
<div>No symbol table info available.</div><div>#15 0x00007f69c89474bd in clone () from /lib/x86_64-linux-gnu/libc.so.6</div><div>No symbol table info available.</div><div>#16 0x0000000000000000 in ?? ()</div><div>No symbol table info available.</div>
<div><br></div><div>Regards,</div><div>Chun-Hung</div></div></div>
</div><br></div>