<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi, <br>
Thank you Michal for your advice. <br>
I went ahead and used gdb, and this is the result I got:<br>
<blockquote>2014-02-14 00:54:11.109+0000: 34958: info :
remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
client pid:58466,uid:0<br>
2014-02-14 00:54:11.125+0000: 34958: error :
qemudDomainInterfaceStats:7865 : 無効な引数: 無効なパス、'Interface'
は不明なインターフェイスです<br>
2014-02-14 00:54:11.155+0000: 34960: info :
remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
client pid:58474,uid:0<br>
2014-02-14 00:54:11.218+0000: 34967: info :
remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
client pid:58489,uid:0<br>
2014-02-14 00:54:11.282+0000: 34961: info :
remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
client pid:58506,uid:0<br>
2014-02-14 00:54:11.392+0000: 34967: info :
remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
client pid:58514,uid:0<br>
2014-02-14 00:54:11.408+0000: 34967: error :
qemudDomainInterfaceStats:7865 : 無効な引数: 無効なパス、'Interface'
は不明なインターフェイスです<br>
2014-02-14 00:54:11.438+0000: 34960: info :
remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
client pid:58522,uid:0<br>
2014-02-14 00:54:11.502+0000: 34959: info :
remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
client pid:58536,uid:0<br>
2014-02-14 00:54:11.612+0000: 34961: info :
remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
client pid:58573,uid:0<br>
2014-02-14 00:54:11.707+0000: 34960: info :
remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
client pid:58600,uid:0<br>
2014-02-14 00:54:12.007+0000: 34962: info :
remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
client pid:58642,uid:0<br>
2014-02-14 00:54:12.072+0000: 34958: info :
remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
client pid:58653,uid:0<br>
2014-02-14 00:54:12.197+0000: 34960: info :
remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
client pid:58666,uid:0<br>
2014-02-14 00:54:12.264+0000: 34966: info :
remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
client pid:58678,uid:0<br>
2014-02-14 00:54:12.355+0000: 34967: info :
remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
client pid:58685,uid:0<br>
2014-02-14 00:54:12.508+0000: 34964: info :
remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
client pid:58693,uid:0<br>
2014-02-14 00:54:12.677+0000: 34961: info :
remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
client pid:58707,uid:0<br>
2014-02-14 00:54:12.744+0000: 34960: info :
remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
client pid:58719,uid:0<br>
2014-02-14 00:54:12.843+0000: 34965: info :
remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
client pid:58726,uid:0<br>
<br>
Program received signal SIGSEGV, Segmentation fault.<br>
[Switching to Thread 0x7f77cd80d700 (LWP 34958)]<br>
0x00007f77d81b696c in free () from /lib64/libc.so.6<br>
</blockquote>
Here's the result of the backtrace:<br>
<blockquote>(gdb) backtrace<br>
#0 0x00007f77d81b696c in free () from /lib64/libc.so.6<br>
#1 0x00007f77db790159 in virFree (ptrptr=0x7f77a0020088) at
util/memory.c:310<br>
#2 0x00007f77db7a1072 in virTypedParameterArrayClear
(params=<value optimized out>,<br>
nparams=5) at util/virtypedparam.c:58<br>
#3 0x0000000000425bec in remoteDispatchDomainBlockStatsFlags (<br>
server=<value optimized out>, client=<value
optimized out>,<br>
msg=<value optimized out>, rerr=0x7f77cd80cbc0,
args=<value optimized out>,<br>
ret=0x7f77a0021730) at remote.c:1186<br>
#4 remoteDispatchDomainBlockStatsFlagsHelper (server=<value
optimized out>,<br>
client=<value optimized out>, msg=<value optimized
out>, rerr=0x7f77cd80cbc0,<br>
args=<value optimized out>, ret=0x7f77a0021730) at
remote_dispatch.h:757<br>
#5 0x00007f77db856e65 in virNetServerProgramDispatchCall
(prog=0xf1f220,<br>
server=0xf13ce0, client=0xf1b570, msg=0xf1f3d0) at
rpc/virnetserverprogram.c:416<br>
#6 virNetServerProgramDispatch (prog=0xf1f220, server=0xf13ce0,
client=0xf1b570,<br>
msg=0xf1f3d0) at rpc/virnetserverprogram.c:289<br>
#7 0x00007f77db8580f1 in virNetServerHandleJob
(jobOpaque=<value optimized out>,<br>
opaque=0xf13ce0) at rpc/virnetserver.c:164<br>
#8 0x00007f77db79a45c in virThreadPoolWorker (opaque=<value
optimized out>)<br>
at util/threadpool.c:144<br>
#9 0x00007f77db799d79 in virThreadHelper (data=<value
optimized out>)<br>
at util/threads-pthread.c:161<br>
#10 0x00007f77d88dc851 in start_thread () from
/lib64/libpthread.so.0<br>
#11 0x00007f77d822367d in clone () from /lib64/libc.so.6<br>
</blockquote>
And util/virtypedparam.c:58 looks like this for libvirt-0.9.10:<br>
49 virTypedParameterArrayClear(virTypedParameterPtr params, int
nparams)<br>
50 {<br>
51 int i;<br>
52<br>
53 if (!params)<br>
54 return;<br>
55<br>
56 for (i = 0; i < nparams; i++) {<br>
57 if (params[i].type == VIR_TYPED_PARAM_STRING)<br>
58 VIR_FREE(params[i].value.s);<br>
59 }<br>
60 }<br>
61<br>
<br>
I'm very unfamiliar with C so I would again appreciate any input
on why this is happening, and what I can do to solve this issue.<br>
By the way this segfault can be 100% reproduced with CentOS 6.3
but not CentOS 6.2.<br>
<br>
Thank you and regards,<br>
Minami<br>
<br>
On 2014/02/07 18:30, Michal Privoznik wrote:<br>
</div>
<blockquote cite="mid:52F4A7C6.9030305@redhat.com" type="cite">On
07.02.2014 09:47, Minami Katsumata wrote:
<br>
<blockquote type="cite">Hi,
<br>
<br>
I'm having problems with libvirt crashing after a couple hours
when a
<br>
specific domain monitoring program is running.
<br>
<br>
I have pasted below the following:
<br>
1. libvirt version
<br>
2. qemu-kvm version
<br>
3. OS version
<br>
4. Kernel version
<br>
5. libvirt status post-crash
<br>
6. libvirtd.log (info level dump around crash; too long to post
<br>
everything so just the beginning and end. UTC)
<br>
7. custom.log (on what this domain monitoring program was doing
around
<br>
the time of the crash. JST)
<br>
8. FYI on the program being executed
<br>
9. other related server settings
<br>
<br>
Please, if anyone can look through these and give some insight
as to
<br>
what is causing libvirt to crash, that would be greatly
appreciated.
<br>
<br>
1.) libvirt version:
<br>
# rpm -q libvirt
<br>
libvirt-0.9.10-21.el6.x86_64
<br>
</blockquote>
<br>
This is rather ancient libvirt, can you please update and see if
the issue was fixed?
<br>
<br>
<blockquote type="cite">
<br>
2.) qemu-kvm version:
<br>
qemu-kvm-0.12.1.2-3.295.el6.10.x86_64
<br>
<br>
3.) OS version:
<br>
# cat /etc/redhat-release
<br>
CentOS release 6.3 (Final)
<br>
</blockquote>
<br>
Ah, this explains the libvirt version. AFAIK there's been
Centos-6.5 released.
<br>
<br>
<blockquote type="cite">
<br>
4.) Kernel version:
<br>
# uname -r
<br>
2.6.32-279.22.1.el6.x86_64
<br>
<br>
5.) libvirt status after crash:
<br>
# service libvirtd status
<br>
libvirtd dead but pid file exists
<br>
<br>
6.) libvirtd.log
<br>
<br>
2014-02-06 10:25:05.173+0000: 1187: info :
remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
client pid:58626,uid:0
<br>
2014-02-06 10:25:05.237+0000: 1184: info :
remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
client pid:58636,uid:0
<br>
2014-02-06 10:25:05.271+0000: 1185: info :
remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
client pid:58646,uid:0
<br>
2014-02-06 10:25:05.301+0000: 1184: info :
remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
client pid:58648,uid:0
<br>
2014-02-06 10:25:05.400+0000: 1184: info :
remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
client pid:58650,uid:0
<br>
Caught Segmentation violation dumping internal log buffer:
<br>
<br>
====== start of log =====
<br>
<br>
^@05.412+00001182: debug : virEventPollMakePollFDs:383 : Prepare
n=19 w=21, f=32 e=25 d=0
<br>
2014-02-06 10:25:05.412+00001182: debug :
virEventPollMakePollFDs:383 : Prepare n=20 w=22, f=34 e=25 d=0
<br>
2014-02-06 10:25:05.412+00001182: debug :
virEventPollMakePollFDs:383 : Prepare n=21 w=23, f=33 e=25 d=0
<br>
2014-02-06 10:25:05.412+00001182: debug :
virEventPollMakePollFDs:383 : Prepare n=22 w=24, f=36 e=25 d=0
<br>
2014-02-06 10:25:05.412+00001182: debug :
virEventPollMakePollFDs:383 : Prepare n=23 w=25, f=38 e=25 d=0
<br>
2014-02-06 10:25:05.412+00001182: debug :
virEventPollMakePollFDs:383 : Prepare n=24 w=26, f=39 e=25 d=0
<br>
2014-02-06 10:25:05.412+00001182: debug :
virEventPollMakePollFDs:383 : Prepare n=25 w=27, f=41 e=25 d=0
<br>
2014-02-06 10:25:05.412+00001182: debug :
virEventPollMakePollFDs:383 : Prepare n=26 w=28, f=40 e=25 d=0
<br>
<br>
(cut out due to length)
<br>
<br>
2014-02-06 10:25:05.423+00001182: debug :
virEventPollDispatchHandles:488 : EVENT_POLL_DISPATCH_HANDLE:
watch=2791 events=2
<br>
2014-02-06 10:25:05.423+00001182: debug : virNetMessageFree:75 :
msg=0x2326a20 nfds=0 cb=(nil)
<br>
2014-02-06 10:25:05.423+00001182: debug :
virNetServerClientCalculateHandleMode:137 : tls=(nil) hs=-1,
rx=0x2266390 tx=(nil)
<br>
2014-02-06 10:25:05.423+00001182: debug :
virNetServerClientCalculateHandleMode:167 : mode=1
<br>
2014-02-06 10:25:05.423+00001182: debug :
virEventPollUpdateHandle:151 : EVENT_POLL_UPDATE_HANDLE:
watch=2791 events=1
<br>
2014-02-06 10:25:05.423+00001182: debug :
virEventPollInterruptLocked:702 : Skip interrupt, 1 -1675536288
<br>
2014-02-06 10:25:05.423+00001182: debug :
virEventPollDispatchHandles:474 : i=33 w=2793
<br>
2014-02-06 10:25:05.423+00001182: debug :
virEventPollCleanupTimeouts:506 : Cleanup 12
<br>
2014-02-06 10:25:05.423+00001182: debug :
virEventPollCleanupHandles:554 : Cleanup 34
<br>
2014-02-06 10:25:05.423+00001182: debug :
virNetServerClientClose:632 : client=0x22e6860 refs=3
<br>
2014-02-06 10:25:05.423+00001182: debug : virKeepAliveStop:382 :
RPC_KEEPALIVE_STOP: ka=0x225bf20 client=0x22e6860
<br>
2014-02-06 10:25:05.423+00001182: debug :
virEventPollRemoveTimeout:293 : EVENT_POLL_REMOVE_TIMEOUT:
timer=8290
<br>
2014-02-06 10:25:05.423+00001182: debug :
virEventPollInterruptLocked:702 : Skip interrupt, 0 -1675536288
<br>
2014-02-06 10:25:05.423+00001182: debug :
virEventPollRemoveTimeout:293 : EVENT_POLL_REMOVE_TIMEOUT:
timer=8289
<br>
2014-02-06 10:25:05.423+00001182: debug :
virEventPollInterruptLocked:702 : Skip interrupt, 0 -1675536288
<br>
2014-02-06 10:25:05.423+00001182: debug : virKeepAliveFree:304 :
RPC_KEEPALIVE_FREE: ka=0x225bf20 client=0x22e6860 refs=3
<br>
2014-02-06 10:25:05.423+00001182: debug :
daemonRemoveAllClientStreams:493 : stream=(nil)
<br>
2014-02-06 10:25:05.423+00001182: debug :
virEventPollRemoveHandle:180 : EVENT_POLL_REMOVE_HANDLE:
watch=2791
<br>
2014-02-06 10:25:05.423+00001182: debug :
virEventPollRemoveHandle:193 : mark delete 32 50
<br>
2014-02-06 10:25:05.423+00001182: debug :
virEventPollInterruptLocked:702 : Skip interrupt, 0 -1675536288
<br>
2014-02-06 10:25:05.423+00001182: debug : virNetMessageFree:75 :
msg=0x2266390 nfds=0 cb=(nil)
<br>
2014-02-06 10:25:05.423+00001182: debug : virNetSocketFree:722 :
RPC_SOCKET_FREE: sock=0x22e66a0 refs=2
<br>
2014-02-06 10:25:05.423+00001182: debug :
virNetServerClientFree:591 : RPC_SERVER_CLIENT_FREE:
client=0x22e6860 refs=3
<br>
2014-02-06 10:25:05.423+00001182: debug :
virEventRunDefaultImpl:244 : running default event
implementation
<br>
2014-02-06 10:25:05.423+00001182: debug :
virEventPollCleanupTimeouts:506 : Cleanup 12
<br>
2014-02-06 10:25:05.423+00001182: debug :
virEventPollCleanupTimeouts:519 : EVENT_POLL_PURGE_TIMEOUT:
timer=8289
<br>
2014-02-06 10:25:05.423+00001182: debug : virKeepAliveFree:304 :
RPC_KEEPALIVE_FREE: ka=0x225bf20 client=0x22e6860 refs=2
<br>
2014-02-06 10:25:05.423+00001182: debug :
virEventPollCleanupTimeouts:519 : EVENT_POLL_PURGE_TIMEOUT:
timer=8290
<br>
2014-02-06 10:25:05.423+00001182: debug : virKeepAliveFree:304 :
RPC_KEEPALIVE_FREE: ka=0x225bf20 client=0x22e6860 refs=1
<br>
2014-02-06 10:25:05.423+00001182: debug :
virNetServerClientFree:591 : RPC_SERVER_CLIENT_FREE:
client=0x22e6860 refs=2
<br>
2014-02-06 10:25:05.423+00001182: debug :
virEventPollCleanupHandles:554 : Cleanup 34
<br>
2014-02-06 10:25:05.423+00001182: debug :
virEventPollCleanupHandles:567 : EVENT_POLL_PURGE_HANDLE:
watch=2791
<br>
2014-02-06 10:25:05.423+00001182: debug :
virNetServerClientFree:591 : RPC_SERVER_CLIENT_FREE:
client=0x22e6860 refs=1
<br>
2014-02-06 10:25:05.423+00001182: debug : virConnectClose:1462 :
conn=0x7f1b380c4630
<br>
2014-02-06 10:25:05.423+00001182: debug : virUnrefConnect:145 :
unref connection 0x7f1b380c4630 1
<br>
2014-02-06 10:25:05.423+00001182: debug : virReleaseConnect:94 :
release connection 0x7f1b380c4630
<br>
<br>
====== end of log =====
<br>
</blockquote>
<br>
These logs are pretty much useless (not your fault). On one hand,
they may help us to see what libvirt was doing just before the
crash. On the other hand:
<br>
a) it completely misses TID
<br>
b) it ends just before SIGSEGV occurs (so for example if
segmentation fault happens in one thread, the logs may as well
been showing completely unrelated thread).
<br>
<br>
Therefore I think attaching gdb to the libvirtd, then reproducing
the crash would gain more data, IMO.
<br>
<br>
Michal
<br>
</blockquote>
<br>
</body>
</html>