[libvirt-users] libvirt crashes with Caught Segmentation violation

Minami Katsumata princessrainicorndev at gmail.com
Fri Feb 14 04:19:22 UTC 2014


Hi,
Thank you Michal for your advice.
I went ahead and used gdb, and this is the result I got:

    2014-02-14 00:54:11.109+0000: 34958: info :
    remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
    client pid:58466,uid:0
    2014-02-14 00:54:11.125+0000: 34958: error :
    qemudDomainInterfaceStats:7865 : ?????: ??????'Interface' ??????????????
    2014-02-14 00:54:11.155+0000: 34960: info :
    remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
    client pid:58474,uid:0
    2014-02-14 00:54:11.218+0000: 34967: info :
    remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
    client pid:58489,uid:0
    2014-02-14 00:54:11.282+0000: 34961: info :
    remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
    client pid:58506,uid:0
    2014-02-14 00:54:11.392+0000: 34967: info :
    remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
    client pid:58514,uid:0
    2014-02-14 00:54:11.408+0000: 34967: error :
    qemudDomainInterfaceStats:7865 : ?????: ??????'Interface' ??????????????
    2014-02-14 00:54:11.438+0000: 34960: info :
    remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
    client pid:58522,uid:0
    2014-02-14 00:54:11.502+0000: 34959: info :
    remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
    client pid:58536,uid:0
    2014-02-14 00:54:11.612+0000: 34961: info :
    remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
    client pid:58573,uid:0
    2014-02-14 00:54:11.707+0000: 34960: info :
    remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
    client pid:58600,uid:0
    2014-02-14 00:54:12.007+0000: 34962: info :
    remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
    client pid:58642,uid:0
    2014-02-14 00:54:12.072+0000: 34958: info :
    remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
    client pid:58653,uid:0
    2014-02-14 00:54:12.197+0000: 34960: info :
    remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
    client pid:58666,uid:0
    2014-02-14 00:54:12.264+0000: 34966: info :
    remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
    client pid:58678,uid:0
    2014-02-14 00:54:12.355+0000: 34967: info :
    remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
    client pid:58685,uid:0
    2014-02-14 00:54:12.508+0000: 34964: info :
    remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
    client pid:58693,uid:0
    2014-02-14 00:54:12.677+0000: 34961: info :
    remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
    client pid:58707,uid:0
    2014-02-14 00:54:12.744+0000: 34960: info :
    remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
    client pid:58719,uid:0
    2014-02-14 00:54:12.843+0000: 34965: info :
    remoteDispatchAuthList:2091 : Bypass polkit auth for privileged
    client pid:58726,uid:0

    Program received signal SIGSEGV, Segmentation fault.
    [Switching to Thread 0x7f77cd80d700 (LWP 34958)]
    0x00007f77d81b696c in free () from /lib64/libc.so.6

Here's the result of the backtrace:

    (gdb) backtrace
    #0  0x00007f77d81b696c in free () from /lib64/libc.so.6
    #1  0x00007f77db790159 in virFree (ptrptr=0x7f77a0020088) at
    util/memory.c:310
    #2  0x00007f77db7a1072 in virTypedParameterArrayClear (params=<value
    optimized out>,
         nparams=5) at util/virtypedparam.c:58
    #3  0x0000000000425bec in remoteDispatchDomainBlockStatsFlags (
         server=<value optimized out>, client=<value optimized out>,
         msg=<value optimized out>, rerr=0x7f77cd80cbc0, args=<value
    optimized out>,
         ret=0x7f77a0021730) at remote.c:1186
    #4  remoteDispatchDomainBlockStatsFlagsHelper (server=<value
    optimized out>,
         client=<value optimized out>, msg=<value optimized out>,
    rerr=0x7f77cd80cbc0,
         args=<value optimized out>, ret=0x7f77a0021730) at
    remote_dispatch.h:757
    #5  0x00007f77db856e65 in virNetServerProgramDispatchCall
    (prog=0xf1f220,
         server=0xf13ce0, client=0xf1b570, msg=0xf1f3d0) at
    rpc/virnetserverprogram.c:416
    #6  virNetServerProgramDispatch (prog=0xf1f220, server=0xf13ce0,
    client=0xf1b570,
         msg=0xf1f3d0) at rpc/virnetserverprogram.c:289
    #7  0x00007f77db8580f1 in virNetServerHandleJob (jobOpaque=<value
    optimized out>,
         opaque=0xf13ce0) at rpc/virnetserver.c:164
    #8  0x00007f77db79a45c in virThreadPoolWorker (opaque=<value
    optimized out>)
         at util/threadpool.c:144
    #9  0x00007f77db799d79 in virThreadHelper (data=<value optimized out>)
         at util/threads-pthread.c:161
    #10 0x00007f77d88dc851 in start_thread () from /lib64/libpthread.so.0
    #11 0x00007f77d822367d in clone () from /lib64/libc.so.6

And util/virtypedparam.c:58 looks like this for libvirt-0.9.10:
  49 virTypedParameterArrayClear(virTypedParameterPtr params, int nparams)
  50 {
  51     int i;
  52
  53     if (!params)
  54         return;
  55
  56     for (i = 0; i < nparams; i++) {
  57         if (params[i].type == VIR_TYPED_PARAM_STRING)
  58             VIR_FREE(params[i].value.s);
  59     }
  60 }
  61

I'm very unfamiliar with C so I would again appreciate any input on why 
this is happening, and what I can do to solve this issue.
By the way this segfault can be 100% reproduced with CentOS 6.3 but not 
CentOS 6.2.

Thank you and regards,
Minami

On 2014/02/07 18:30, Michal Privoznik wrote:
> On 07.02.2014 09:47, Minami Katsumata wrote:
>> Hi,
>>
>> I'm having problems with libvirt crashing after a couple hours when a
>> specific domain monitoring program is running.
>>
>> I have pasted below the following:
>> 1. libvirt version
>> 2. qemu-kvm version
>> 3. OS version
>> 4. Kernel version
>> 5. libvirt status post-crash
>> 6. libvirtd.log (info level dump around crash; too long to post
>> everything so just the beginning and end. UTC)
>> 7. custom.log (on what this domain monitoring program was doing around
>> the time of the crash. JST)
>> 8. FYI on the program being executed
>> 9. other related server settings
>>
>> Please, if anyone can look through these and give some insight as to
>> what is causing libvirt to crash, that would be greatly appreciated.
>>
>> 1.) libvirt version:
>> # rpm -q libvirt
>> libvirt-0.9.10-21.el6.x86_64
>
> This is rather ancient libvirt, can you please update and see if the 
> issue was fixed?
>
>>
>> 2.) qemu-kvm version:
>> qemu-kvm-0.12.1.2-3.295.el6.10.x86_64
>>
>> 3.) OS version:
>> # cat /etc/redhat-release
>> CentOS release 6.3 (Final)
>
> Ah, this explains the libvirt version. AFAIK there's been Centos-6.5 
> released.
>
>>
>> 4.) Kernel version:
>> # uname -r
>> 2.6.32-279.22.1.el6.x86_64
>>
>> 5.) libvirt status after crash:
>> # service libvirtd status
>> libvirtd dead but pid file exists
>>
>> 6.) libvirtd.log
>>
>> 2014-02-06 10:25:05.173+0000: 1187: info : 
>> remoteDispatchAuthList:2091 : Bypass polkit auth for privileged 
>> client pid:58626,uid:0
>> 2014-02-06 10:25:05.237+0000: 1184: info : 
>> remoteDispatchAuthList:2091 : Bypass polkit auth for privileged 
>> client pid:58636,uid:0
>> 2014-02-06 10:25:05.271+0000: 1185: info : 
>> remoteDispatchAuthList:2091 : Bypass polkit auth for privileged 
>> client pid:58646,uid:0
>> 2014-02-06 10:25:05.301+0000: 1184: info : 
>> remoteDispatchAuthList:2091 : Bypass polkit auth for privileged 
>> client pid:58648,uid:0
>> 2014-02-06 10:25:05.400+0000: 1184: info : 
>> remoteDispatchAuthList:2091 : Bypass polkit auth for privileged 
>> client pid:58650,uid:0
>> Caught Segmentation violation dumping internal log buffer:
>>
>>      ====== start of log =====
>>
>> ^@05.412+00001182: debug : virEventPollMakePollFDs:383 : Prepare n=19 
>> w=21, f=32 e=25 d=0
>> 2014-02-06 10:25:05.412+00001182: debug : virEventPollMakePollFDs:383 
>> : Prepare n=20 w=22, f=34 e=25 d=0
>> 2014-02-06 10:25:05.412+00001182: debug : virEventPollMakePollFDs:383 
>> : Prepare n=21 w=23, f=33 e=25 d=0
>> 2014-02-06 10:25:05.412+00001182: debug : virEventPollMakePollFDs:383 
>> : Prepare n=22 w=24, f=36 e=25 d=0
>> 2014-02-06 10:25:05.412+00001182: debug : virEventPollMakePollFDs:383 
>> : Prepare n=23 w=25, f=38 e=25 d=0
>> 2014-02-06 10:25:05.412+00001182: debug : virEventPollMakePollFDs:383 
>> : Prepare n=24 w=26, f=39 e=25 d=0
>> 2014-02-06 10:25:05.412+00001182: debug : virEventPollMakePollFDs:383 
>> : Prepare n=25 w=27, f=41 e=25 d=0
>> 2014-02-06 10:25:05.412+00001182: debug : virEventPollMakePollFDs:383 
>> : Prepare n=26 w=28, f=40 e=25 d=0
>>
>> (cut out due to length)
>>
>> 2014-02-06 10:25:05.423+00001182: debug : 
>> virEventPollDispatchHandles:488 : EVENT_POLL_DISPATCH_HANDLE: 
>> watch=2791 events=2
>> 2014-02-06 10:25:05.423+00001182: debug : virNetMessageFree:75 : 
>> msg=0x2326a20 nfds=0 cb=(nil)
>> 2014-02-06 10:25:05.423+00001182: debug : 
>> virNetServerClientCalculateHandleMode:137 : tls=(nil) hs=-1, 
>> rx=0x2266390 tx=(nil)
>> 2014-02-06 10:25:05.423+00001182: debug : 
>> virNetServerClientCalculateHandleMode:167 : mode=1
>> 2014-02-06 10:25:05.423+00001182: debug : 
>> virEventPollUpdateHandle:151 : EVENT_POLL_UPDATE_HANDLE: watch=2791 
>> events=1
>> 2014-02-06 10:25:05.423+00001182: debug : 
>> virEventPollInterruptLocked:702 : Skip interrupt, 1 -1675536288
>> 2014-02-06 10:25:05.423+00001182: debug : 
>> virEventPollDispatchHandles:474 : i=33 w=2793
>> 2014-02-06 10:25:05.423+00001182: debug : 
>> virEventPollCleanupTimeouts:506 : Cleanup 12
>> 2014-02-06 10:25:05.423+00001182: debug : 
>> virEventPollCleanupHandles:554 : Cleanup 34
>> 2014-02-06 10:25:05.423+00001182: debug : virNetServerClientClose:632 
>> : client=0x22e6860 refs=3
>> 2014-02-06 10:25:05.423+00001182: debug : virKeepAliveStop:382 : 
>> RPC_KEEPALIVE_STOP: ka=0x225bf20 client=0x22e6860
>> 2014-02-06 10:25:05.423+00001182: debug : 
>> virEventPollRemoveTimeout:293 : EVENT_POLL_REMOVE_TIMEOUT: timer=8290
>> 2014-02-06 10:25:05.423+00001182: debug : 
>> virEventPollInterruptLocked:702 : Skip interrupt, 0 -1675536288
>> 2014-02-06 10:25:05.423+00001182: debug : 
>> virEventPollRemoveTimeout:293 : EVENT_POLL_REMOVE_TIMEOUT: timer=8289
>> 2014-02-06 10:25:05.423+00001182: debug : 
>> virEventPollInterruptLocked:702 : Skip interrupt, 0 -1675536288
>> 2014-02-06 10:25:05.423+00001182: debug : virKeepAliveFree:304 : 
>> RPC_KEEPALIVE_FREE: ka=0x225bf20 client=0x22e6860 refs=3
>> 2014-02-06 10:25:05.423+00001182: debug : 
>> daemonRemoveAllClientStreams:493 : stream=(nil)
>> 2014-02-06 10:25:05.423+00001182: debug : 
>> virEventPollRemoveHandle:180 : EVENT_POLL_REMOVE_HANDLE: watch=2791
>> 2014-02-06 10:25:05.423+00001182: debug : 
>> virEventPollRemoveHandle:193 : mark delete 32 50
>> 2014-02-06 10:25:05.423+00001182: debug : 
>> virEventPollInterruptLocked:702 : Skip interrupt, 0 -1675536288
>> 2014-02-06 10:25:05.423+00001182: debug : virNetMessageFree:75 : 
>> msg=0x2266390 nfds=0 cb=(nil)
>> 2014-02-06 10:25:05.423+00001182: debug : virNetSocketFree:722 : 
>> RPC_SOCKET_FREE: sock=0x22e66a0 refs=2
>> 2014-02-06 10:25:05.423+00001182: debug : virNetServerClientFree:591 
>> : RPC_SERVER_CLIENT_FREE: client=0x22e6860 refs=3
>> 2014-02-06 10:25:05.423+00001182: debug : virEventRunDefaultImpl:244 
>> : running default event implementation
>> 2014-02-06 10:25:05.423+00001182: debug : 
>> virEventPollCleanupTimeouts:506 : Cleanup 12
>> 2014-02-06 10:25:05.423+00001182: debug : 
>> virEventPollCleanupTimeouts:519 : EVENT_POLL_PURGE_TIMEOUT: timer=8289
>> 2014-02-06 10:25:05.423+00001182: debug : virKeepAliveFree:304 : 
>> RPC_KEEPALIVE_FREE: ka=0x225bf20 client=0x22e6860 refs=2
>> 2014-02-06 10:25:05.423+00001182: debug : 
>> virEventPollCleanupTimeouts:519 : EVENT_POLL_PURGE_TIMEOUT: timer=8290
>> 2014-02-06 10:25:05.423+00001182: debug : virKeepAliveFree:304 : 
>> RPC_KEEPALIVE_FREE: ka=0x225bf20 client=0x22e6860 refs=1
>> 2014-02-06 10:25:05.423+00001182: debug : virNetServerClientFree:591 
>> : RPC_SERVER_CLIENT_FREE: client=0x22e6860 refs=2
>> 2014-02-06 10:25:05.423+00001182: debug : 
>> virEventPollCleanupHandles:554 : Cleanup 34
>> 2014-02-06 10:25:05.423+00001182: debug : 
>> virEventPollCleanupHandles:567 : EVENT_POLL_PURGE_HANDLE: watch=2791
>> 2014-02-06 10:25:05.423+00001182: debug : virNetServerClientFree:591 
>> : RPC_SERVER_CLIENT_FREE: client=0x22e6860 refs=1
>> 2014-02-06 10:25:05.423+00001182: debug : virConnectClose:1462 : 
>> conn=0x7f1b380c4630
>> 2014-02-06 10:25:05.423+00001182: debug : virUnrefConnect:145 : unref 
>> connection 0x7f1b380c4630 1
>> 2014-02-06 10:25:05.423+00001182: debug : virReleaseConnect:94 : 
>> release connection 0x7f1b380c4630
>>
>>       ====== end of log =====
>
> These logs are pretty much useless (not your fault). On one hand, they 
> may help us to see what libvirt was doing just before the crash. On 
> the other hand:
> a) it completely misses TID
> b) it ends just before SIGSEGV occurs (so for example if segmentation 
> fault happens in one thread, the logs may as well been showing 
> completely unrelated thread).
>
> Therefore I think attaching gdb to the libvirtd, then reproducing the 
> crash would gain more data, IMO.
>
> Michal

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20140214/0be3f6c3/attachment.htm>


More information about the libvirt-users mailing list