[libvirt] libvirtd crash when attach-disk to VM

Sat Aug 18 10:57:16 UTC 2012

On 08/16/12 11:15, Wangpan wrote:
> Hi all,
> I got a depressed problem(libvirtd crash with SIGABRT or SIGSEGV sometimes) when attach a nbd disk to a VM by using cmd as follow:
> virsh attach-disk 228 --source /dev/nbd0 --target vdd --sourcetype block --driver qemu --subdriver raw
> or just using  virsh attach-disk 228 /dev/nbd0 vdd.
> and this problem occurs everytime when I attach a nbd disk to VM.
> 

...

> 
> Some info of packages' version is listed below:
> root at 114-113-199-15:/home/hzwangpan# dpkg -l | grep libvi
> ii  libvirt-bin                                    0.9.12-4                             programs for the libvirt library
> ii  libvirt-dev                                    0.9.12-4                             development files for the libvirt library
> ii  libvirt0                                       0.9.12-4                             library for interfacing with different virtualization systems
> ii  libvirt0-dbg                                   0.9.12-4                             library for interfacing with different virtualization systems
> 

You probably (looking at the version numbers) came across a known bug:
https://bugzilla.redhat.com/show_bug.cgi?id=822068

> 0x00007ffff45c5475 in raise () from /lib/x86_64-linux-gnu/libc.so.6
> (gdb) bt
> #0  0x00007ffff45c5475 in raise () from /lib/x86_64-linux-gnu/libc.so.6
> #1  0x00007ffff45c86f0 in abort () from /lib/x86_64-linux-gnu/libc.so.6
> #2  0x00007ffff45ff2fb in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #3  0x00007ffff4608b46 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #4  0x00007ffff460c428 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #5  0x00007ffff460d960 in malloc () from /lib/x86_64-linux-gnu/libc.so.6
> #6  0x00007ffff4612912 in strdup () from /lib/x86_64-linux-gnu/libc.so.6
> #7  0x00007ffff77b75c9 in virJSONValueObjectAppend (object=object at entry=0x91d810, key=key at entry=0x4fef81 "execute", value=value at entry=0x85de90)
>      at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/util/json.c:274
> #8  0x00007ffff77b7e87 in virJSONValueObjectAppendString (object=object at entry=0x91d810, key=key at entry=0x4fef81 "execute", value=value at entry=0x51196f "human-monitor-command")
>      at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/util/json.c:296
> #9  0x00000000004aa884 in qemuMonitorJSONMakeCommandRaw (wrap=wrap at entry=false, cmdname=cmdname at entry=0x51196f "human-monitor-command")
>      at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor_json.c:404
> #10 0x00000000004ac3a7 in qemuMonitorJSONHumanCommandWithFd (mon=mon at entry=0x7fffe80010b0, cmd_str=<optimized out>, scm_fd=-1, reply_str=0x7ffff1760920)
>      at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor_json.c:886
> #11 0x000000000049d303 in qemuMonitorHMPCommandWithFd (mon=mon at entry=0x7fffe80010b0, cmd=<optimized out>, scm_fd=scm_fd at entry=-1, reply=reply at entry=0x7ffff1760920)
>      at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor.c:910
> #12 0x00000000004a8bfe in qemuMonitorTextAddDrive (mon=mon at entry=0x7fffe80010b0, drivestr=drivestr at entry=0x7fffe0202020 "file=/dev/nbd0,if=none,id=drive-virtio-disk3,format=raw")
>      at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor_text.c:2836
> #13 0x00000000004b0814 in qemuMonitorJSONAddDrive (mon=0x7fffe80010b0, drivestr=0x7fffe0202020 "file=/dev/nbd0,if=none,id=drive-virtio-disk3,format=raw")
>      at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor_json.c:2979
> #14 0x00000000004a1bad in qemuMonitorAddDrive (mon=<optimized out>, drivestr=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor.c:2571
> #15 0x0000000000484a5d in qemuDomainAttachPciDiskDevice (conn=conn at entry=0x7fffe00111f0, driver=driver at entry=0x81fec0, vm=vm at entry=0x82f6b0, disk=disk at entry=0x7fffe02024d0)
>      at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_hotplug.c:250
> #16 0x0000000000461d9e in qemuDomainAttachDeviceDiskLive (vm=0x82f6b0, driver=0x81fec0, conn=0x7fffe00111f0, dev=<optimized out>)
>      at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_driver.c:5179
> #17 qemuDomainAttachDeviceLive (dev=0x7fffe001d5b0, vm=0x82f6b0, dom=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_driver.c:5238
> #18 qemuDomainModifyDeviceFlags (dom=<optimized out>, xml=0x7fffe001d5b0 "\001", flags=<optimized out>, action=<optimized out>)
>      at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_driver.c:5779
> #19 0x00007ffff7846f5d in virDomainAttachDevice (domain=domain at entry=0x7fffe0201fb0,
>      xml=0x7fffe0201e50 "<disk type='block'>\n  <driver name='qemu' type='raw'/>\n  <source dev='/dev/nbd0'/>\n  <target dev='vdd'/>\n</disk>\n")
>      at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/libvirt.c:9288
> #20 0x000000000043ccfe in remoteDispatchDomainAttachDevice (args=0x7fffe0201ff0, rerr=0x7ffff1760c90, client=<optimized out>, server=<optimized out>, msg=<optimized out>)
>      at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./daemon/remote_dispatch.h:320
> #21 remoteDispatchDomainAttachDeviceHelper (server=<optimized out>, client=<optimized out>, msg=<optimized out>, rerr=0x7ffff1760c90, args=0x7fffe0201ff0, ret=<optimized out>)
>      at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./daemon/remote_dispatch.h:298
> #22 0x00007ffff788a866 in virNetServerProgramDispatchCall (msg=0x7fffe8093d20, client=0x7fffe8053050, server=0x76e920, prog=0x778880)
>      at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/rpc/virnetserverprogram.c:416
> #23 virNetServerProgramDispatch (prog=0x778880, server=server at entry=0x76e920, client=0x7fffe8053050, msg=0x7fffe8093d20)
>      at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/rpc/virnetserverprogram.c:289
> #24 0x00007ffff78864d1 in virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x76e920) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/rpc/virnetserver.c:161
> #25 0x00007ffff77c373e in virThreadPoolWorker (opaque=opaque at entry=0x7789a0) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/util/threadpool.c:144
> #26 0x00007ffff77c31c9 in virThreadHelper (data=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/util/threads-pthread.c:161
> #27 0x00007ffff4d27b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
> #28 0x00007ffff466b6dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> #29 0x0000000000000000 in ?? ()
> (gdb) f 7

This backtrace is identical with that attached to the bug.

The bug is fixed by commit:
commit 0f4660c8787cc41fe67f869984c0ae11d680037e
Author: Peter Krempa <pkrempa at redhat.com>
Date:   Thu Jun 14 10:29:36 2012 +0200

    qemu: Fix off-by-one error while unescaping monitor strings

    While unescaping the commands the commands passed through to the monitor
    function qemuMonitorUnescapeArg() initialized lenght of the input string
    to strlen()+1 which is fine for alloc but not for iteration of the
    string.

That is included in the 0.9.13 release. To fix this issue please upgrade or
propose to backport that patch into your distro. At any rate thanks for the 
exhausting bug report, it definitely helped identifying the issue and would
be useful in fixing it.

Peter