[libvirt] libvirtd crashes

Fri Dec 4 00:36:48 UTC 2009

FOr me, the error seems to vary.  I am going to show two versions.

This is one.
The gdb bt is
#0  0x00007f7ce92986b4 in pthread_mutex_unlock () from /lib/libpthread.so.0
#1  0x000000000042f661 in ?? ()
#2  0x000000000043da36 in ?? ()
#3  0x000000000043ef4b in ?? ()
#4  0x00007f7ce94f90fb in virDomainCreateXML () from /usr/lib/libvirt.so.0
#5  0x000000000041f228 in ?? ()
#6  0x0000000000420e41 in ?? ()
#7  0x00000000004211f3 in ?? ()
#8  0x000000000041478c in ?? ()
#9  0x00007f7ce9294a04 in start_thread () from /lib/libpthread.so.0
#10 0x00007f7ce8ffe7bd in clone () from /lib/libc.so.6
#11 0x0000000000000000 in ?? ()

Its corresponding debug is
17:09:46.140: debug : virEventRemoveHandleImpl:173 : Remove handle w=81
17:09:46.140: debug : virEventRemoveHandleImpl:186 : mark delete 38 68
17:09:46.140: debug : virEventInterruptLocked:658 : Skip interrupt, 1 -438118128
17:09:46.140: debug : qemuMonitorClose:532 : Mark monitor to be deleted 0x7f7cd80cf480
17:09:46.140: debug : qemuDomainSetFileOwnership:1971 : Setting ownership on /srv/cloud/one/var//6666/images/disk.0 to 0:0
17:09:46.140: debug : virEventUpdateHandleImpl:146 : Update handle w=81 e=12
17:09:46.140: debug : virEventInterruptLocked:662 : Interrupting
17:09:46.140: debug : qemuMonitorCommandWithHandler:271 : Receive command reply ret=-1 errno=104 0 bytes '(null)'
17:09:46.140: error : qemuMonitorCommandWithHandler:290 : cannot send monitor command 'info cpus': Connection reset by peer
17:09:46.140: error : qemuMonitorTextGetCPUInfo:436 : internal error cannot run monitor command to fetch CPU thread info

Here is another one:
(gdb) bt
#0  0x00007ff61ca026b4 in pthread_mutex_unlock () from /lib/libpthread.so.0
#1  0x000000000042f661 in qemuDomainObjExitMonitorWithDriver (driver=0x7ff61000c410, obj=0x1e8c310) at qemu/qemu_driver.c:318
#2  0x000000000043da36 in qemudStartVMDaemon (conn=<value optimized out>, driver=0x7ff61000c410, vm=0x1e8c310, 
    migrateFrom=<value optimized out>, stdin_fd=<value optimized out>) at qemu/qemu_driver.c:2327
#3  0x000000000043ef4b in qemudDomainCreate (conn=0x7ff610009a00, xml=<value optimized out>, flags=<value optimized out>)
    at qemu/qemu_driver.c:2881
#4  0x00007ff61cc630fb in virDomainCreateXML (conn=0x7ff610009a00, 
    xmlDesc=0x7ff610004d70 "<domain type='kvm'>\n\t<name>one-7238</name>\n\t<vcpu>1</vcpu>\n\t<memory>524288</memory>\n\t<os>\n\t\t<type>hvm</type>\n\t\t<boot dev='hd'/>\n\t</os>\n\t<devices>\n\t\t<emulator>/usr/bin/kvm</emulator>\n\t\t<disk type='file"..., flags=0) at libvirt.c:1745
#5  0x000000000041f228 in remoteDispatchDomainCreateXml (server=<value optimized out>, client=<value optimized out>, 
    conn=0x7ff610009a00, hdr=0x6572687420555043, rerr=0x6f666e69206461, args=0x616d6d6f6320726f, ret=0x7ff60e7fbed0)
    at remote.c:873
#6  0x0000000000420e41 in remoteDispatchClientCall (server=<value optimized out>, client=0x7ff6080c4960, msg=0x7ff6080ca910)
    at dispatch.c:506
#7  0x00000000004211f3 in remoteDispatchClientRequest (server=0x1e62070, client=0x7ff6080c4960, msg=0x7ff6080ca910)
    at dispatch.c:388
#8  0x000000000041478c in qemudWorker (data=<value optimized out>) at libvirtd.c:1518
#9  0x00007ff61c9fea04 in start_thread () from /lib/libpthread.so.0
#10 0x00007ff61c7687bd in clone () from /lib/libc.so.6
#11 0x0000000000000000 in ?? ()

and its debug:
17:33:35.424: debug : virEventUpdateHandleImpl:146 : Update handle w=1389 e=12
17:33:35.424: debug : virEventInterruptLocked:662 : Interrupting
17:33:35.424: debug : qemuMonitorCommandWithHandler:271 : Receive command reply ret=-1 errno=104 0 bytes '(null)'
17:33:35.424: error : qemuMonitorCommandWithHandler:290 : cannot send monitor command 'info cpus': Connection reset by peer
17:33:35.424: error : qemuMonitorTextGetCPUInfo:436 : internal error cannot run monitor command to fetch CPU thread info

Thanks a lot.
Shi
--
Shi Jin, PhD

--- On Thu, 12/3/09, Matthias Bolte <matthias.bolte at googlemail.com> wrote:

> From: Matthias Bolte <matthias.bolte at googlemail.com>
> Subject: Re: [libvirt] libvirtd crashes
> To: "Shi Jin" <jinzishuai at yahoo.com>
> Cc: libvir-list at redhat.com, jinzishuai at gmail.com
> Date: Thursday, December 3, 2009, 4:20 PM
> 2009/12/3 Shi Jin <jinzishuai at yahoo.com>:
> > Hi there,
> >
> > My libvirtd built from the latest git code keeps on
> crashing on all machines.
> > I turned on debugging and this is the information I
> have in the log file before crashing:
> > 14:31:50.828: debug : virEventUpdateHandleImpl:146 :
> Update handle w=110 e=12
> > 14:31:50.828: debug : virEventInterruptLocked:662 :
> Interrupting
> > 14:31:50.828: debug :
> qemuMonitorCommandWithHandler:271 : Receive command reply
> ret=-1 errno=104 0 bytes '(null)'
> > 14:31:50.828: error :
> qemuMonitorCommandWithHandler:290 : cannot send monitor
> command 'info cpus': Connection reset by peer
> > 14:31:50.828: error : qemuMonitorTextGetCPUInfo:436 :
> internal error cannot run monitor command to fetch CPU
> thread info
> >
> > I am not sure if there is any other information needed
> to help identify the problem.  My building options are:
> > ./autogen.sh  --prefix=/usr --sysconfdir=/etc
> --localstatedir=/var  --without-xen --with-qemu
> --with-qemu-user=oneadmin --with-qemu-group=oneadmin
> --without-uml --without-vbox --without-openvz --without-lxc
> >
> > Please help me here. I can accept the service failing
> queries from time to time since I have error handling
> written so that they can be re-tried. But a crashing
> libvirtd takes the whole thing down.
> >
> > Thanks a lot.
> >
> > Shi
> > --
> > Shi Jin, PhD
> >
> 
> A GDB backtrace would be helpful. Judging by the debug log
> alone it
> could be a known issue, see:
> 
> https://www.redhat.com/archives/libvir-list/2009-December/msg00063.html
> 
> Matthias
> 