[libvirt] Segfault in libxl_osevent_occurred_timeout [Was: Re: [libvirt-users] About debugging of libvirt.]
Dario Faggioli
dario.faggioli at citrix.com
Thu Dec 19 11:31:14 UTC 2013
[Moving this to libvir, libvir-users in Bcc. Also, added xen-devel]
Jim,
cooldharma06 reports having issues when destroying VMs with libvirt
1.2.0 and Xen 4.2.1 (is that the case, cooldharma06)?
Look at the stack trace at the end of this message, or here:
https://bugzilla.redhat.com/show_bug.cgi?id=1044838
Ian, since it looks like events are involved... Any idea?
cooldharma06, about the Xen 4.2.1, thing, is there a specific reason why
you use it instead, of, e.g., 4.3.x ? Since you are building from source
anyway, could you at least try Xen 4.3, to see whether the issue could
be a bug in 4.2's libxl? I'm asking because, with both Xen 4.3 and the
curret git tip, I don't have issues destroying domains with virsh.
Thanks and Regards,
Dario
On gio, 2013-12-19 at 15:42 +0530, cool dharma06 wrote:
> i did the debugging as you said. Kindly refer the following logs:
>
>
> (gdb) c
> Continuing.
> thread apply all bt
> [New Thread 0x7fc337c6b700 (LWP 29520)]
>
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x0000000000000000 in ?? ()
> (gdb) thread apply all bt
>
>
> Thread 12 (Thread 0x7fc337c6b700 (LWP 29520)):
> #0 0x00007fc33509f18d in read ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00007fc32b1555c4 in read_all (fd=26, data=0x7fc3380e0a70,
> data at entry=0x20, len=len at entry=16, nonblocking=nonblocking at entry=0) at
> xs.c:365
> #2 0x00007fc32b1556d8 in read_message (h=h at entry=0x7fc324013ee0,
> nonblocking=nonblocking at entry=0) at xs.c:1071
> #3 0x00007fc32b156005 in read_thread (arg=0x7fc324013ee0) at
> xs.c:1137
> #4 0x00007fc335097b50 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #5 0x00007fc3349daa3d in clone ()
> from /lib/x86_64-linux-gnu/libc.so.6
> #6 0x0000000000000000 in ?? ()
>
>
> Thread 11 (Thread 0x7fc330e7f700 (LWP 29170)):
> #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00007fc3371ae9ea in virCondWait (c=c at entry=0x7fc3380dbd80,
> m=m at entry=0x7fc3380dbd58) at util/virthreadpthread.c:117
> #2 0x00007fc3371af0cb in virThreadPoolWorker
> (opaque=opaque at entry=0x7fc3380db870) at util/virthreadpool.c:103
> #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at
> util/virthreadpthread.c:161
> #4 0x00007fc335097b50 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #5 0x00007fc3349daa3d in clone ()
> from /lib/x86_64-linux-gnu/libc.so.6
> #6 0x0000000000000000 in ?? ()
>
>
> Thread 10 (Thread 0x7fc33067e700 (LWP 29171)):
> #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00007fc3371ae9ea in virCondWait (c=c at entry=0x7fc3380dbd80,
> m=m at entry=0x7fc3380dbd58) at util/virthreadpthread.c:117
> #2 0x00007fc3371af0cb in virThreadPoolWorker
> (opaque=opaque at entry=0x7fc3380db640) at util/virthreadpool.c:103
> #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at
> util/virthreadpthread.c:161
> #4 0x00007fc335097b50 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #5 0x00007fc3349daa3d in clone ()
> from /lib/x86_64-linux-gnu/libc.so.6
> #6 0x0000000000000000 in ?? ()
>
>
> Thread 9 (Thread 0x7fc32fe7d700 (LWP 29172)):
> #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00007fc3371ae9ea in virCondWait (c=c at entry=0x7fc3380dbd80,
> m=m at entry=0x7fc3380dbd58) at util/virthreadpthread.c:117
> #2 0x00007fc3371af0cb in virThreadPoolWorker
> (opaque=opaque at entry=0x7fc3380db870) at util/virthreadpool.c:103
> #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at
> util/virthreadpthread.c:161
> #4 0x00007fc335097b50 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #5 0x00007fc3349daa3d in clone ()
> from /lib/x86_64-linux-gnu/libc.so.6
> #6 0x0000000000000000 in ?? ()
>
>
> Thread 8 (Thread 0x7fc32f67c700 (LWP 29173)):
> #0 0x00007fc33497cb9f in realloc ()
> from /lib/x86_64-linux-gnu/libc.so.6
> #1 0x00007fc3349eec85 in __vasprintf_chk ()
> from /lib/x86_64-linux-gnu/libc.so.6
> ---Type <return> to continue, or q <return> to quit---
> #2 0x00007fc3371aac0a in vasprintf (__ap=0x7fc32f67b6d0,
> __fmt=0x7fc337391017 "%llu: %s : %s:%d : %s\n", __ptr=0x7fc32f67b830)
> at /usr/include/x86_64-linux-gnu/bits/stdio2.h:199
> #3 virVasprintfInternal (report=report at entry=false, domcode=0,
> filename=0x0, funcname=0x0, linenr=0, strp=0x7fc32f67b830,
> fmt=fmt at entry=0x7fc337391017 "%llu: %s : %s:%d : %s\n",
> list=list at entry=0x7fc32f67b6d0) at util/virstring.c:337
> #4 0x00007fc3371aad1b in virAsprintfInternal
> (report=report at entry=false, domcode=domcode at entry=0,
> filename=filename at entry=0x0, funcname=funcname at entry=0x0,
> linenr=linenr at entry=0, strp=strp at entry=0x7fc32f67b830,
> fmt=fmt at entry=0x7fc337391017 "%llu: %s : %s:%d : %s\n") at
> util/virstring.c:358
> #5 0x00007fc33718cb59 in virLogFormatString (str=<optimized out>,
> priority=VIR_LOG_DEBUG, funcname=0x7fc337394734 "virObjectRef",
> linenr=293, msg=0x7fc32f67b830)
> at util/virlog.c:719
> #6 virLogVMessage (source=VIR_LOG_FROM_TRACE, priority=VIR_LOG_DEBUG,
> filename=0x7fc337394597 "util/virobject.c", linenr=293,
> funcname=0x7fc337394734 "virObjectRef",
> metadata=0x0, fmt=fmt at entry=0x7fc3373945f1 "OBJECT_REF: obj=%p",
> vargs=vargs at entry=0x7fc32f67b8c0) at util/virlog.c:846
> #7 0x00007fc33718cfc7 in virLogMessage
> (source=source at entry=VIR_LOG_FROM_TRACE,
> priority=priority at entry=VIR_LOG_DEBUG,
> filename=filename at entry=0x7fc337394597 "util/virobject.c",
> linenr=linenr at entry=293, funcname=funcname at entry=0x7fc337394734
> "virObjectRef",
> metadata=metadata at entry=0x0, fmt=fmt at entry=0x7fc3373945f1
> "OBJECT_REF: obj=%p") at util/virlog.c:778
> #8 0x00007fc33719c3d7 in virObjectRef (anyobj=0x7fc338102720) at
> util/virobject.c:293
> #9 0x00007fc32af1824f in libxlDriverConfigGet
> (driver=driver at entry=0x7fc338108930) at libxl/libxl_conf.c:1124
> #10 0x00007fc32af1a50e in libxlVmCleanup
> (driver=driver at entry=0x7fc338108930, vm=vm at entry=0x7fc32400f530,
> reason=reason at entry=VIR_DOMAIN_SHUTOFF_DESTROYED)
> at libxl/libxl_driver.c:264
> #11 0x00007fc32af20274 in libxlVmReap (driver=0x7fc338108930,
> vm=0x7fc32400f530, reason=VIR_DOMAIN_SHUTOFF_DESTROYED) at
> libxl/libxl_driver.c:338
> #12 0x00007fc32af207de in libxlDomainDestroyFlags (dom=0x7fc3380e3770,
> flags=<optimized out>) at libxl/libxl_driver.c:1507
> #13 0x00007fc33723cd37 in virDomainDestroy
> (domain=domain at entry=0x7fc3380e3770) at libvirt.c:2342
> #14 0x00007fc337cb9ba4 in remoteDispatchDomainDestroy (args=<optimized
> out>, rerr=0x7fc32f67bcd0, client=0x7fc32400ee60, server=<optimized
> out>, msg=<optimized out>)
> at remote_dispatch.h:3165
> #15 remoteDispatchDomainDestroyHelper (server=<optimized out>,
> client=0x7fc32400ee60, msg=<optimized out>, rerr=0x7fc32f67bcd0,
> args=<optimized out>,
> ret=<optimized out>) at remote_dispatch.h:3143
> #16 0x00007fc3372b4d79 in virNetServerProgramDispatchCall
> (msg=0x7fc324014070, client=0x7fc32400ee60, server=0x7fc3380dbba0,
> prog=0x7fc324007750)
> at rpc/virnetserverprogram.c:435
> #17 virNetServerProgramDispatch (prog=0x7fc324007750,
> server=server at entry=0x7fc3380dbba0, client=0x7fc32400ee60,
> msg=0x7fc324014070) at rpc/virnetserverprogram.c:305
> #18 0x00007fc3372aea78 in virNetServerProcessMsg (msg=<optimized out>,
> prog=<optimized out>, client=<optimized out>, srv=0x7fc3380dbba0) at
> rpc/virnetserver.c:165
> #19 virNetServerHandleJob (jobOpaque=<optimized out>,
> opaque=0x7fc3380dbba0) at rpc/virnetserver.c:186
> #20 0x00007fc3371af02e in virThreadPoolWorker
> (opaque=opaque at entry=0x7fc3380db640) at util/virthreadpool.c:144
> #21 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at
> util/virthreadpthread.c:161
> #22 0x00007fc335097b50 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #23 0x00007fc3349daa3d in clone ()
> from /lib/x86_64-linux-gnu/libc.so.6
> #24 0x0000000000000000 in ?? ()
>
>
> Thread 7 (Thread 0x7fc32ee7b700 (LWP 29174)):
> #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00007fc3371ae9ea in virCondWait (c=c at entry=0x7fc3380dbd80,
> m=m at entry=0x7fc3380dbd58) at util/virthreadpthread.c:117
> #2 0x00007fc3371af0cb in virThreadPoolWorker
> (opaque=opaque at entry=0x7fc3380db870) at util/virthreadpool.c:103
> #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at
> util/virthreadpthread.c:161
> ---Type <return> to continue, or q <return> to quit---
> #4 0x00007fc335097b50 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #5 0x00007fc3349daa3d in clone ()
> from /lib/x86_64-linux-gnu/libc.so.6
> #6 0x0000000000000000 in ?? ()
>
>
> Thread 6 (Thread 0x7fc32e67a700 (LWP 29175)):
> #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00007fc3371ae9ea in virCondWait (c=c at entry=0x7fc3380dbe18,
> m=m at entry=0x7fc3380dbd58) at util/virthreadpthread.c:117
> #2 0x00007fc3371af0eb in virThreadPoolWorker
> (opaque=opaque at entry=0x7fc3380db640) at util/virthreadpool.c:103
> #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at
> util/virthreadpthread.c:161
> #4 0x00007fc335097b50 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #5 0x00007fc3349daa3d in clone ()
> from /lib/x86_64-linux-gnu/libc.so.6
> #6 0x0000000000000000 in ?? ()
>
>
> Thread 5 (Thread 0x7fc32de79700 (LWP 29176)):
> #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00007fc3371ae9ea in virCondWait (c=c at entry=0x7fc3380dbe18,
> m=m at entry=0x7fc3380dbd58) at util/virthreadpthread.c:117
> #2 0x00007fc3371af0eb in virThreadPoolWorker
> (opaque=opaque at entry=0x7fc3380db870) at util/virthreadpool.c:103
> #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at
> util/virthreadpthread.c:161
> #4 0x00007fc335097b50 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #5 0x00007fc3349daa3d in clone ()
> from /lib/x86_64-linux-gnu/libc.so.6
> #6 0x0000000000000000 in ?? ()
>
>
> Thread 4 (Thread 0x7fc32d678700 (LWP 29177)):
> #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00007fc3371ae9ea in virCondWait (c=c at entry=0x7fc3380dbe18,
> m=m at entry=0x7fc3380dbd58) at util/virthreadpthread.c:117
> #2 0x00007fc3371af0eb in virThreadPoolWorker
> (opaque=opaque at entry=0x7fc3380db640) at util/virthreadpool.c:103
> #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at
> util/virthreadpthread.c:161
> #4 0x00007fc335097b50 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #5 0x00007fc3349daa3d in clone ()
> from /lib/x86_64-linux-gnu/libc.so.6
> #6 0x0000000000000000 in ?? ()
>
>
> Thread 3 (Thread 0x7fc32ce77700 (LWP 29178)):
> #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00007fc3371ae9ea in virCondWait (c=c at entry=0x7fc3380dbe18,
> m=m at entry=0x7fc3380dbd58) at util/virthreadpthread.c:117
> #2 0x00007fc3371af0eb in virThreadPoolWorker
> (opaque=opaque at entry=0x7fc3380db870) at util/virthreadpool.c:103
> #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at
> util/virthreadpthread.c:161
> #4 0x00007fc335097b50 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #5 0x00007fc3349daa3d in clone ()
> from /lib/x86_64-linux-gnu/libc.so.6
> #6 0x0000000000000000 in ?? ()
>
>
> ---Type <return> to continue, or q <return> to quit---
> Thread 2 (Thread 0x7fc32c676700 (LWP 29179)):
> #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00007fc3371ae9ea in virCondWait (c=c at entry=0x7fc3380dbe18,
> m=m at entry=0x7fc3380dbd58) at util/virthreadpthread.c:117
> #2 0x00007fc3371af0eb in virThreadPoolWorker
> (opaque=opaque at entry=0x7fc3380db640) at util/virthreadpool.c:103
> #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at
> util/virthreadpthread.c:161
> #4 0x00007fc335097b50 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #5 0x00007fc3349daa3d in clone ()
> from /lib/x86_64-linux-gnu/libc.so.6
> #6 0x0000000000000000 in ?? ()
>
>
> Thread 1 (Thread 0x7fc337c4d7c0 (LWP 29169)):
> #0 0x0000000000000000 in ?? ()
> #1 0x00007fc32ace881e in libxl_osevent_occurred_timeout
> (ctx=<optimized out>, for_libxl=0x7fc3380edf38) at libxl_event.c:1039
> #2 0x00007fc32af199c7 in libxlDomainObjTimerCallback
> (timer=<optimized out>, timer_info=0x7fc3380e3310) at
> libxl/libxl_domain.c:216
> #3 0x00007fc33717cf7d in virEventPollDispatchTimeouts () at
> util/vireventpoll.c:451
> #4 virEventPollRunOnce () at util/vireventpoll.c:644
> #5 0x00007fc33717b70b in virEventRunDefaultImpl () at
> util/virevent.c:274
> #6 0x00007fc3372b0065 in virNetServerRun (srv=0x7fc3380dbba0) at
> rpc/virnetserver.c:1112
> #7 0x00007fc337c9851c in main (argc=<optimized out>, argv=<optimized
> out>) at libvirtd.c:1513
> (gdb)
> (gdb)
>
>
> ****
>
>
> regards,
> cooldharma06.
>
>
> On Thu, Dec 19, 2013 at 3:30 PM, Daniel P. Berrange
> <berrange at redhat.com> wrote:
> On Thu, Dec 19, 2013 at 03:09:14PM +0530, cool dharma06 wrote:
> > I am newbie to this gdb debugging. now i did the bt in the
> debugging. In my
> > side still it showing the error. Following are the
> stacktrace which i got.
> > Kindly refer the following logs for full information:
> >
> > ****
> > [New Thread 0x7f8503695700 (LWP 20012)]
> >
> > Program received signal SIGSEGV, Segmentation fault.
> > 0x0000000000000000 in ?? ()
> > (gdb) bt
> > #0 0x0000000000000000 in ?? ()
> > #1 0x00007f84f671281e in libxl_osevent_occurred_timeout
> (ctx=<optimized
> > out>, for_libxl=0x7f8503bbbc38) at libxl_event.c:1039
> > #2 0x00007f84f69439c7 in libxlDomainObjTimerCallback
> (timer=<optimized
> > out>, timer_info=0x7f8503bb5930) at libxl/libxl_domain.c:216
> > #3 0x00007f8502ba6f7d in virEventPollDispatchTimeouts () at
> > util/vireventpoll.c:451
> > #4 virEventPollRunOnce () at util/vireventpoll.c:644
> > #5 0x00007f8502ba570b in virEventRunDefaultImpl () at
> util/virevent.c:274
> > #6 0x00007f8502cda065 in virNetServerRun
> (srv=0x7f8503ba6ba0) at
> > rpc/virnetserver.c:1112
> > #7 0x00007f85036c251c in main (argc=<optimized out>,
> argv=<optimized out>)
> > at libvirtd.c:1513
> > (gdb)
>
>
> In this case, that stack trace looks like it ought to be
> sufficient
> to identify the problem - a clear NULL pointer reference. In
> general
> though you want to capture a stack trace of all threads in a
> program
>
> eg instead of 'bt', do 'thread apply all bt' next time.
>
> Regards,
> Daniel
> --
> |: http://berrange.com -o-
> http://www.flickr.com/photos/dberrange/ :|
> |: http://libvirt.org -o-
> http://virt-manager.org :|
> |: http://autobuild.org -o-
> http://search.cpan.org/~danberr/ :|
> |: http://entangle-photo.org -o-
> http://live.gnome.org/gtk-vnc :|
>
>
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20131219/123e77bd/attachment-0001.sig>
More information about the libvir-list
mailing list