<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div dir="ltr"><br>
<div class="gmail_extra">
<div class="gmail_quote">2018-01-17 18:38 GMT+02:00 Serhii
Kharchenko <span dir="ltr"><<a
href="mailto:harrykas@gmail.com" target="_blank">harrykas@gmail.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote"><span class="gmail-">2018-01-17
17:32 GMT+02:00 Michal Privoznik <span dir="ltr"><<a
href="mailto:mprivozn@redhat.com"
target="_blank">mprivozn@redhat.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div
class="gmail-m_-7424554455873124548gmail-HOEnZb">
<div
class="gmail-m_-7424554455873124548gmail-h5">Sounds
like qemuConnectGetAllDomainStats() forgot to
unset the job on a<br>
</div>
</div>
domain. Can you please attach gdb to libvirtd and
run 't a a bt' and<br>
share the output?<br>
<span
class="gmail-m_-7424554455873124548gmail-HOEnZb"><font
color="#888888"><br>
Michal<br>
</font></span></blockquote>
<div><br>
</div>
</span>
<div>This is output of gdb 't a a bt' ~5 seconds
after that error message appeared in log with <wbr>remoteDispatchConnectGetAllDom<wbr>ainStats()
mentioned.</div>
<div><br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<div class="gmail_extra">And this is output of gdb 't a a bt'
executed after the VPS is stuck in 'in shutdown' state but error
message is not yet logged. It logged after I executed gdb
command.</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">
<div>Thread 18 (Thread 0x7ff36ffff700 (LWP 29836)):</div>
<div>#0 0x00007ff3c0fe4ca6 in
<a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_timedwait@@GLIBC_2.3.2">pthread_cond_timedwait@@GLIBC_2.3.2</a> () from
/lib64/libpthread.so.0</div>
<div>#1 0x00007ff3c1978082 in virCondWaitUntil () from
/usr/lib64/libvirt.so.0</div>
<div>#2 0x00007ff378017a28 in ?? () from
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so</div>
<div>#3 0x00007ff37801b9ab in qemuDomainObjBeginJob () from
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so</div>
<div>#4 0x00007ff378047306 in qemuProcessBeginStopJob () from
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so</div>
<div>#5 0x00007ff3780b1689 in ?? () from
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so</div>
<div>#6 0x00007ff3c1978a2b in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#7 0x00007ff3c1977db8 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#8 0x00007ff3c0fdd6b3 in start_thread () from
/lib64/libpthread.so.0</div>
<div>#9 0x00007ff3c0d1176f in clone () from /lib64/libc.so.6</div>
<div><br>
</div>
<div>Thread 17 (Thread 0x7ff374fa1700 (LWP 28463)):</div>
<div>#0 0x00007ff3c0fe4746 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0</div>
<div>#1 0x00007ff3c1978016 in virCondWait () from
/usr/lib64/libvirt.so.0</div>
<div>#2 0x00007ff378971a20 in ?? () from
/usr/lib64/libvirt/connection-driver/libvirt_driver_nodedev.so</div>
<div>#3 0x00007ff3c1977de2 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#4 0x00007ff3c0fdd6b3 in start_thread () from
/lib64/libpthread.so.0</div>
<div>#5 0x00007ff3c0d1176f in clone () from /lib64/libc.so.6</div>
<div><br>
</div>
<div>Thread 16 (Thread 0x7ff375fa3700 (LWP 28459)):</div>
<div>#0 0x00007ff3c0fe4746 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0</div>
<div>#1 0x00007ff3c1978016 in virCondWait () from
/usr/lib64/libvirt.so.0</div>
<div>#2 0x00007ff3c1978b63 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#3 0x00007ff3c1977db8 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#4 0x00007ff3c0fdd6b3 in start_thread () from
/lib64/libpthread.so.0</div>
<div>#5 0x00007ff3c0d1176f in clone () from /lib64/libc.so.6</div>
<div><br>
</div>
<div>Thread 15 (Thread 0x7ff3767a4700 (LWP 28458)):</div>
<div>#0 0x00007ff3c0fe4746 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0</div>
<div>#1 0x00007ff3c1978016 in virCondWait () from
/usr/lib64/libvirt.so.0</div>
<div>#2 0x00007ff3c1978b63 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#3 0x00007ff3c1977db8 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#4 0x00007ff3c0fdd6b3 in start_thread () from
/lib64/libpthread.so.0</div>
<div>#5 0x00007ff3c0d1176f in clone () from /lib64/libc.so.6</div>
<div><br>
</div>
<div>Thread 14 (Thread 0x7ff376fa5700 (LWP 28457)):</div>
<div>#0 0x00007ff3c0fe4746 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0</div>
<div>#1 0x00007ff3c1978016 in virCondWait () from
/usr/lib64/libvirt.so.0</div>
<div>#2 0x00007ff3c1978b63 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#3 0x00007ff3c1977db8 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#4 0x00007ff3c0fdd6b3 in start_thread () from
/lib64/libpthread.so.0</div>
<div>
<div>#5 0x00007ff3c0d1176f in clone () from /lib64/libc.so.6</div>
<div><br>
</div>
<div>Thread 13 (Thread 0x7ff3777a6700 (LWP 28456)):</div>
<div>#0 0x00007ff3c0fe4746 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a>
() from /lib64/libpthread.so.0</div>
<div>#1 0x00007ff3c1978016 in virCondWait () from
/usr/lib64/libvirt.so.0</div>
<div>#2 0x00007ff3c1978b63 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#3 0x00007ff3c1977db8 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#4 0x00007ff3c0fdd6b3 in start_thread () from
/lib64/libpthread.so.0</div>
<div>#5 0x00007ff3c0d1176f in clone () from /lib64/libc.so.6</div>
<div><br>
</div>
<div>Thread 12 (Thread 0x7ff377fa7700 (LWP 28455)):</div>
<div>#0 0x00007ff3c0fe4746 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a>
() from /lib64/libpthread.so.0</div>
<div>#1 0x00007ff3c1978016 in virCondWait () from
/usr/lib64/libvirt.so.0</div>
<div>#2 0x00007ff3c1978b63 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#3 0x00007ff3c1977db8 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#4 0x00007ff3c0fdd6b3 in start_thread () from
/lib64/libpthread.so.0</div>
<div>#5 0x00007ff3c0d1176f in clone () from /lib64/libc.so.6</div>
<div><br>
</div>
<div>Thread 11 (Thread 0x7ff3b37fe700 (LWP 28454)):</div>
<div>#0 0x00007ff3c0fe4746 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a>
() from /lib64/libpthread.so.0</div>
<div>#1 0x00007ff3c1978016 in virCondWait () from
/usr/lib64/libvirt.so.0</div>
<div>#2 0x00007ff3c1978b24 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#3 0x00007ff3c1977db8 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#4 0x00007ff3c0fdd6b3 in start_thread () from
/lib64/libpthread.so.0</div>
<div>#5 0x00007ff3c0d1176f in clone () from /lib64/libc.so.6</div>
<div><br>
</div>
<div>Thread 10 (Thread 0x7ff3b3fff700 (LWP 28453)):</div>
<div>#0 0x00007ff3c0fe4746 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a>
() from /lib64/libpthread.so.0</div>
<div>#1 0x00007ff3c1978016 in virCondWait () from
/usr/lib64/libvirt.so.0</div>
<div>#2 0x00007ff3c1978b24 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#3 0x00007ff3c1977db8 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#4 0x00007ff3c0fdd6b3 in start_thread () from
/lib64/libpthread.so.0</div>
<div>#5 0x00007ff3c0d1176f in clone () from /lib64/libc.so.6</div>
<div><br>
</div>
<div>Thread 9 (Thread 0x7ff3b8bb1700 (LWP 28452)):</div>
<div>#0 0x00007ff3c0fe4746 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a>
() from /lib64/libpthread.so.0</div>
<div>#1 0x00007ff3c1978016 in virCondWait () from
/usr/lib64/libvirt.so.0</div>
<div>#2 0x00007ff3c1978b24 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#3 0x00007ff3c1977db8 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#4 0x00007ff3c0fdd6b3 in start_thread () from
/lib64/libpthread.so.0</div>
<div>#5 0x00007ff3c0d1176f in clone () from /lib64/libc.so.6</div>
<div><br>
</div>
<div>Thread 8 (Thread 0x7ff3b93b2700 (LWP 28451)):</div>
</div>
<div>
<div>#0 0x00007ff3c0fe4746 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a>
() from /lib64/libpthread.so.0</div>
<div>#1 0x00007ff3c1978016 in virCondWait () from
/usr/lib64/libvirt.so.0</div>
<div>#2 0x00007ff3c1978b24 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#3 0x00007ff3c1977db8 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#4 0x00007ff3c0fdd6b3 in start_thread () from
/lib64/libpthread.so.0</div>
<div>#5 0x00007ff3c0d1176f in clone () from /lib64/libc.so.6</div>
<div><br>
</div>
<div>Thread 7 (Thread 0x7ff3b9bb3700 (LWP 28450)):</div>
<div>#0 0x00007ff3c0fe4746 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a>
() from /lib64/libpthread.so.0</div>
<div>#1 0x00007ff3c1978016 in virCondWait () from
/usr/lib64/libvirt.so.0</div>
<div>#2 0x00007ff3c1978b24 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#3 0x00007ff3c1977db8 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#4 0x00007ff3c0fdd6b3 in start_thread () from
/lib64/libpthread.so.0</div>
<div>#5 0x00007ff3c0d1176f in clone () from /lib64/libc.so.6</div>
<div><br>
</div>
<div>Thread 6 (Thread 0x7ff3ba3b4700 (LWP 28449)):</div>
<div>#0 0x00007ff3c0fe4746 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a>
() from /lib64/libpthread.so.0</div>
<div>#1 0x00007ff3c1978016 in virCondWait () from
/usr/lib64/libvirt.so.0</div>
<div>#2 0x00007ff3c1978b63 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#3 0x00007ff3c1977db8 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#4 0x00007ff3c0fdd6b3 in start_thread () from
/lib64/libpthread.so.0</div>
<div>#5 0x00007ff3c0d1176f in clone () from /lib64/libc.so.6</div>
<div><br>
</div>
<div>Thread 5 (Thread 0x7ff3babb5700 (LWP 28448)):</div>
<div>#0 0x00007ff3c0fe4746 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a>
() from /lib64/libpthread.so.0</div>
<div>#1 0x00007ff3c1978016 in virCondWait () from
/usr/lib64/libvirt.so.0</div>
<div>#2 0x00007ff3c1978b63 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#3 0x00007ff3c1977db8 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#4 0x00007ff3c0fdd6b3 in start_thread () from
/lib64/libpthread.so.0</div>
<div>#5 0x00007ff3c0d1176f in clone () from /lib64/libc.so.6</div>
<div><br>
</div>
<div>Thread 4 (Thread 0x7ff3bb3b6700 (LWP 28447)):</div>
<div>#0 0x00007ff3c0fe4746 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a>
() from /lib64/libpthread.so.0</div>
<div>#1 0x00007ff3c1978016 in virCondWait () from
/usr/lib64/libvirt.so.0</div>
<div>#2 0x00007ff3c1978b63 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#3 0x00007ff3c1977db8 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#4 0x00007ff3c0fdd6b3 in start_thread () from
/lib64/libpthread.so.0</div>
<div>#5 0x00007ff3c0d1176f in clone () from /lib64/libc.so.6</div>
<div><br>
</div>
<div>Thread 3 (Thread 0x7ff3bbbb7700 (LWP 28446)):</div>
<div>#0 0x00007ff3c0fe4746 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a>
() from /lib64/libpthread.so.0</div>
<div>#1 0x00007ff3c1978016 in virCondWait () from
/usr/lib64/libvirt.so.0</div>
<div>#2 0x00007ff37805f76b in qemuMonitorSend () from
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so</div>
</div>
<div>
<div>#3 0x00007ff378073355 in ?? () from
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so</div>
<div>#4 0x00007ff378075237 in qemuMonitorJSONQueryCPUs ()
from
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so</div>
<div>#5 0x00007ff378061dd2 in qemuMonitorGetCpuHalted () from
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so</div>
<div>#6 0x00007ff378023c9e in qemuDomainRefreshVcpuHalted ()
from
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so</div>
<div>#7 0x00007ff3780a3329 in ?? () from
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so</div>
<div>#8 0x00007ff37808777f in ?? () from
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so</div>
<div>#9 0x00007ff3c1a37abe in virDomainListGetStats () from
/usr/lib64/libvirt.so.0</div>
<div>#10 0x0000560fd00243b5 in ?? ()</div>
<div>#11 0x00007ff3c1a92308 in virNetServerProgramDispatch ()
from /usr/lib64/libvirt.so.0</div>
<div>#12 0x0000560fd0053ef8 in ?? ()</div>
<div>#13 0x00007ff3c1978a2b in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#14 0x00007ff3c1977db8 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#15 0x00007ff3c0fdd6b3 in start_thread () from
/lib64/libpthread.so.0</div>
<div>#16 0x00007ff3c0d1176f in clone () from /lib64/libc.so.6</div>
<div><br>
</div>
<div>Thread 2 (Thread 0x7ff3bc3b8700 (LWP 28445)):</div>
<div>#0 0x00007ff3c0fe4746 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a>
() from /lib64/libpthread.so.0</div>
<div>#1 0x00007ff3c1978016 in virCondWait () from
/usr/lib64/libvirt.so.0</div>
<div>#2 0x00007ff3c1978b63 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#3 0x00007ff3c1977db8 in ?? () from
/usr/lib64/libvirt.so.0</div>
<div>#4 0x00007ff3c0fdd6b3 in start_thread () from
/lib64/libpthread.so.0</div>
<div>#5 0x00007ff3c0d1176f in clone () from /lib64/libc.so.6</div>
<div><br>
</div>
<div>Thread 1 (Thread 0x7ff3c265a800 (LWP 28444)):</div>
<div>#0 0x00007ff3c0d059cc in poll () from /lib64/libc.so.6</div>
<div>#1 0x00007ff3c1921c84 in virEventPollRunOnce () from
/usr/lib64/libvirt.so.0</div>
<div>#2 0x00007ff3c1920971 in virEventRunDefaultImpl () from
/usr/lib64/libvirt.so.0</div>
<div>#3 0x00007ff3c1a8d64d in virNetDaemonRun () from
/usr/lib64/libvirt.so.0</div>
<div>#4 0x0000560fd001a700 in ?? ()</div>
<div>#5 0x00007ff3c0c2efe1 in __libc_start_main () from
/lib64/libc.so.6</div>
<div>#6 0x0000560fd001affa in ?? ()</div>
</div>
</div>
</div>
</body>
</html>