[libvirt-users] libvrtd-1.1.0 crashes when attempting to start some (but not all) LXC containers
Dennis Jenkins
dennis.jenkins.75 at gmail.com
Fri Jul 12 18:32:16 UTC 2013
Update: I am able to edit the XML in "dwj-hfax-dev" such that libvirtd no
longer crashes, and edit the XML for "dwj-lnx-dev" such that it will crash.
The presents of "<seclabel type='none'/>" near the bottom causes libvirtd
to crash.
I do not recall ever manually adding that to my domain.
In any event, libvirtd should probably not crash due to the XML element
(which seems valid - or at least "virsh edit" allows it).
On Fri, Jul 12, 2013 at 11:44 AM, Dennis Jenkins <
dennis.jenkins.75 at gmail.com> wrote:
> The debug log ends with this:
>
>
> 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:708 : Make
> group /machine/dwj-hfax-dev.libvirt-lxc
> 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make
> controller /sys/fs/cgroup/cpu/machine/dwj-hfax-dev.libvirt-lxc/
> 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make
> controller /sys/fs/cgroup/cpuacct/machine/dwj-hfax-dev.libvirt-lxc/
> 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make
> controller /sys/fs/cgroup/cpuset/machine/dwj-hfax-dev.libvirt-lxc/
> 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make
> controller /sys/fs/cgroup/memory/machine/dwj-hfax-dev.libvirt-lxc/
> 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make
> controller /sys/fs/cgroup/devices/machine/dwj-hfax-dev.libvirt-lxc/
> 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make
> controller /sys/fs/cgroup/freezer/machine/dwj-hfax-dev.libvirt-lxc/
> 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make
> controller /sys/fs/cgroup/blkio/machine/dwj-hfax-dev.libvirt-lxc/
> 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:715 :
> Skipping unmounted controller net_cls
> 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make
> controller /sys/fs/cgroup/perf_event/machine/dwj-hfax-dev.libvirt-lxc/
> 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:779 : Done
> making controllers for group
> 2013-07-12 16:43:31.740+0000: 21365: debug : virFileMakePathHelper:1995 :
> path=/var/log/libvirt/lxc mode=0777
> 2013-07-12 16:43:31.740+0000: 21365: debug : virLXCProcessStart:1096 :
> Setting current domain def as transient
> 2013-07-12 16:43:31.741+0000: 21365: debug : virLXCProcessStart:1121 :
> Preparing host devices
> 2013-07-12 16:43:31.741+0000: 21365: debug : virLXCProcessStart:1139 :
> Generating domain security label (if required)
>
>
> ====== end of log =====
>
> Segmentation fault (core dumped)
>
>
>
> (gdb) bt
> #0 0x00007fe4750c5d76 in __strcmp_sse42 () from /lib64/libc.so.6
> #1 0x00007fe47578ad31 in virSecurityManagerGenLabel () from
> /usr/lib64/libvirt.so.0
> #2 0x00007fe46aa92979 in virLXCProcessStart () from
> /usr/lib64/libvirt/connection-driver/libvirt_driver_lxc.so
> #3 0x00007fe46aa9736e in lxcDomainCreateWithFlags () from
> /usr/lib64/libvirt/connection-driver/libvirt_driver_lxc.so
> #4 0x00007fe47569c067 in virDomainCreate () from /usr/lib64/libvirt.so.0
> #5 0x00007fe4760d5578 in remoteDispatchDomainCreateHelper ()
> #6 0x00007fe4756fcd78 in virNetServerProgramDispatch () from
> /usr/lib64/libvirt.so.0
> #7 0x00007fe4756f7302 in virNetServerProcessMsg () from
> /usr/lib64/libvirt.so.0
> #8 0x00007fe4756f7a93 in virNetServerHandleJob () from
> /usr/lib64/libvirt.so.0
> #9 0x00007fe47560f95e in virThreadPoolWorker () from
> /usr/lib64/libvirt.so.0
> #10 0x00007fe47560efc6 in virThreadHelper () from /usr/lib64/libvirt.so.0
> #11 0x00007fe475354da6 in start_thread () from /lib64/libpthread.so.0
> #12 0x00007fe47508d99d in clone () from /lib64/libc.so.6
> (gdb)
>
>
>
> On Fri, Jul 12, 2013 at 11:40 AM, Dennis Jenkins <
> dennis.jenkins.75 at gmail.com> wrote:
>
>> Hello all,
>>
>> I have two issues:
>>
>> 1) I am unable to start a seemingly correct LXC domain (I cloned it from
>> a working domain).
>>
>> 2) I am able to crash "libvirtd" by attempting to start the cloned
>> domain, but starting the original works just fine.
>>
>> I humbly submit that item #2 is a bug - the "libvirtd" daemon should
>> never crash due to anything the "libvirt" client throws at it. As for
>> item #1, I'm not sure where I went wrong. A full walk-through is below
>> (ending with a DIFF of the XML from the two domains).
>>
>> I created by original domain ("dwj-lnx-dev") a long time ago. Today
>> I created the new domain ("dwj-hfax-dev") as follows:
>>
>> 1) Shutdown "dwj-lnx-dev"
>> 2) Clone the root file system: "cd /vm/lxc/; cp -a dwj-lnx-dev
>> dwj-hfax-dev" (2.5GB, ~5 min)
>> 3) "libvirt -c lxc:/// dumpxml dwj-lnx-dev > a.xml"
>> 4) ${EDITOR} a.xml
>> a) changed MAC address, name, memory, source directory for "/"
>> 5) "libvirt -c lxc:/// define a.xml"
>> 6) Edit "/etc/bind/pri/*" and "/etc/dhcp/dhcpd.conf" on my host.
>>
>>
>> It does not matter is "dwj-lnx-dev" is running or not. Any attempt
>> to start "dwj-hfax-dev" will crash libvirtd.
>>
>>
>> In the past I was asked to turn on some debugging and capture a
>> detailed log (
>> https://www.redhat.com/archives/libvirt-users/2013-May/msg00076.html).
>> I will do this soon and post my results as a follow up.
>>
>>
>>
>> ostara ~ # uname -a
>> Linux ostara 3.8.13-gentoo #1 SMP PREEMPT Mon Jun 3 17:10:56 CDT 2013
>> x86_64 Intel(R) Core(TM) i5 CPU 760 @ 2.80GHz GenuineIntel GNU/Linux
>>
>>
>> ostara ~ # equery l libvirt
>> * Searching for libvirt ...
>> [IP-] [ ] app-emulation/libvirt-1.1.0-r1:0
>>
>>
>> ostara ~ # virsh -c lxc:/// version
>> Compiled against library: libvirt 1.1.0
>> Using library: libvirt 1.1.0
>> Using API: LXC 1.1.0
>> Running hypervisor: LXC 3.8.13
>>
>>
>> ostara ~ # /etc/init.d/libvirtd restart
>> * Caching service dependencies
>> ...
>> [ ok ]
>> * Stopping libvirtd ...
>> * Shutting down network(s):
>> *
>> default
>> [ ok ]
>> * Starting libvirtd
>> ...
>> [ ok ]
>>
>>
>> ostara ~ # virsh -c lxc:/// list --all
>> Id Name State
>> ----------------------------------------------------
>> - dwj-hfax-dev shut off
>> - dwj-lnx-dev shut off
>> - vm1 shut off
>>
>> ostara ~ # virsh -c lxc:/// start dwj-lnx-dev
>> Domain dwj-lnx-dev started
>>
>>
>> ostara ~ # virsh -c lxc:/// list --all
>> Id Name State
>> ----------------------------------------------------
>> 9441 dwj-lnx-dev running
>> - dwj-hfax-dev shut off
>> - vm1 shut off
>>
>>
>> ostara ~ # virsh -c lxc:/// start dwj-hfax-dev
>> error: Failed to start domain dwj-hfax-dev
>> error: End of file while reading data: Input/output error
>> error: One or more references were leaked after disconnect from the
>> hypervisor
>> error: Failed to reconnect to the hypervisor
>>
>>
>> ostara ~ # virsh -c lxc:/// list --all
>> error: failed to connect to the hypervisor
>> error: no valid connection
>> error: Failed to connect socket to '/var/run/libvirt/libvirt-sock':
>> Connection refused
>>
>>
>> ostara ~ # ls -l /var/run/libvirt/libvirt-sock
>> srwx------ 1 root root 0 Jul 12 11:21 /var/run/libvirt/libvirt-sock
>>
>>
>> ostara ~ # ps axfw | grep libvirt
>> 9997 pts/2 S+ 0:00 \_ grep --colour=auto
>> libvirt
>> 8446 ? S 0:00 /usr/sbin/dnsmasq
>> --conf-file=/var/lib/libvirt/dnsmasq/default.conf
>> 9441 ? Ss 0:00 /usr/libexec/libvirt_lxc --name dwj-lnx-dev
>> --console 19 --security=none --handshake 23 --background --veth veth1
>>
>>
>> ostara ~ # /etc/init.d/libvirtd restart
>> * Stopping libvirtd ...
>> * start-stop-daemon: no matching processes
>> found
>> [ ok ]
>> * Starting libvirtd
>> ...
>> [ ok ]
>>
>>
>> ostara ~ # ps axfw | grep libvirt
>> 10130 pts/2 S+ 0:00 \_ grep --colour=auto
>> libvirt
>> 8446 ? S 0:00 /usr/sbin/dnsmasq
>> --conf-file=/var/lib/libvirt/dnsmasq/default.conf
>> 9441 ? Ss 0:00 /usr/libexec/libvirt_lxc --name dwj-lnx-dev
>> --console 19 --security=none --handshake 23 --background --veth veth1
>> 10033 ? Sl 0:00 /usr/sbin/libvirtd -d --listen
>>
>>
>> ostara ~ # virsh -c lxc:/// list --all
>> Id Name State
>> ----------------------------------------------------
>> 9441 dwj-lnx-dev running
>> - dwj-hfax-dev shut off
>> - vm1 shut off
>>
>>
>> ostara ~ # virsh -c lxc:/// dumpxml dwj-hfax-dev
>> <domain type='lxc'>
>> <name>dwj-hfax-dev</name>
>> <uuid>681410de-7b56-41bd-b38d-3c66ce97e7b3</uuid>
>> <memory unit='KiB'>4194304</memory>
>> <currentMemory unit='KiB'>4194304</currentMemory>
>> <vcpu placement='static'>4</vcpu>
>> <resource>
>> <partition>/machine</partition>
>> </resource>
>> <os>
>> <type arch='x86_64'>exe</type>
>> <init>/sbin/init</init>
>> </os>
>> <clock offset='utc'/>
>> <on_poweroff>destroy</on_poweroff>
>> <on_reboot>restart</on_reboot>
>> <on_crash>destroy</on_crash>
>> <devices>
>> <emulator>/usr/libexec/libvirt_lxc</emulator>
>> <filesystem type='mount' accessmode='passthrough'>
>> <source dir='/vm/lxc/dwj-hfax-dev'/>
>> <target dir='/'/>
>> </filesystem>
>> <filesystem type='mount' accessmode='passthrough'>
>> <source dir='/usr/portage'/>
>> <target dir='/usr/portage'/>
>> </filesystem>
>> <filesystem type='mount' accessmode='passthrough'>
>> <source dir='/usr/src'/>
>> <target dir='/usr/src'/>
>> </filesystem>
>> <filesystem type='mount' accessmode='passthrough'>
>> <source dir='/home'/>
>> <target dir='/home'/>
>> </filesystem>
>> <interface type='bridge'>
>> <mac address='82:00:00:00:01:01'/>
>> <source bridge='br0'/>
>> <target dev='veth0'/>
>> </interface>
>> <console type='pty'>
>> <target type='lxc' port='0'/>
>> </console>
>> </devices>
>> <seclabel type='none'/>
>> </domain>
>>
>>
>> ostara ~ # virsh -c lxc:/// dumpxml dwj-lnx-dev
>> <domain type='lxc' id='9441'>
>> <name>dwj-lnx-dev</name>
>> <uuid>fbcd8c3a-9939-12b4-727d-5d3526bc448f</uuid>
>> <memory unit='KiB'>500000</memory>
>> <currentMemory unit='KiB'>500000</currentMemory>
>> <vcpu placement='static'>2</vcpu>
>> <resource>
>> <partition>/machine</partition>
>> </resource>
>> <os>
>> <type arch='x86_64'>exe</type>
>> <init>/sbin/init</init>
>> </os>
>> <clock offset='utc'/>
>> <on_poweroff>destroy</on_poweroff>
>> <on_reboot>restart</on_reboot>
>> <on_crash>destroy</on_crash>
>> <devices>
>> <emulator>/usr/libexec/libvirt_lxc</emulator>
>> <filesystem type='mount' accessmode='passthrough'>
>> <source dir='/vm/lxc/dwj-lnx-dev'/>
>> <target dir='/'/>
>> </filesystem>
>> <filesystem type='mount' accessmode='passthrough'>
>> <source dir='/usr/portage'/>
>> <target dir='/usr/portage'/>
>> </filesystem>
>> <filesystem type='mount' accessmode='passthrough'>
>> <source dir='/usr/src'/>
>> <target dir='/usr/src'/>
>> </filesystem>
>> <filesystem type='mount' accessmode='passthrough'>
>> <source dir='/home'/>
>> <target dir='/home'/>
>> </filesystem>
>> <interface type='bridge'>
>> <mac address='82:00:00:00:01:00'/>
>> <source bridge='br0'/>
>> <target dev='veth0'/>
>> </interface>
>> <console type='pty' tty='/dev/pts/3'>
>> <source path='/dev/pts/3'/>
>> <target type='lxc' port='0'/>
>> <alias name='console0'/>
>> </console>
>> </devices>
>> <seclabel type='none'/>
>> </domain>
>>
>>
>> ostara ~ # virsh -c lxc:/// dumpxml dwj-lnx-dev > lnx.xml
>>
>>
>> ostara ~ # virsh -c lxc:/// dumpxml dwj-hfax-dev > hfax.xml
>>
>>
>> ostara ~ # diff lnx.xml hfax.xml
>> 1,6c1,6
>> < <domain type='lxc' id='9441'>
>> < <name>dwj-lnx-dev</name>
>> < <uuid>fbcd8c3a-9939-12b4-727d-5d3526bc448f</uuid>
>> < <memory unit='KiB'>500000</memory>
>> < <currentMemory unit='KiB'>500000</currentMemory>
>> < <vcpu placement='static'>2</vcpu>
>> ---
>> > <domain type='lxc'>
>> > <name>dwj-hfax-dev</name>
>> > <uuid>681410de-7b56-41bd-b38d-3c66ce97e7b3</uuid>
>> > <memory unit='KiB'>4194304</memory>
>> > <currentMemory unit='KiB'>4194304</currentMemory>
>> > <vcpu placement='static'>4</vcpu>
>> 21c21
>> < <source dir='/vm/lxc/dwj-lnx-dev'/>
>> ---
>> > <source dir='/vm/lxc/dwj-hfax-dev'/>
>> 37c37
>> < <mac address='82:00:00:00:01:00'/>
>> ---
>> > <mac address='82:00:00:00:01:01'/>
>> 41,42c41
>> < <console type='pty' tty='/dev/pts/3'>
>> < <source path='/dev/pts/3'/>
>> ---
>> > <console type='pty'>
>> 44d42
>> < <alias name='console0'/>
>>
>>
>> (After reseting everything, and attempting to boot hfax with dev offline,
>> libvirtd still crashes)
>>
>> ostara ~ # virsh -c lxc:/// list --all
>> Id Name State
>> ----------------------------------------------------
>> - dwj-hfax-dev shut off
>> - dwj-lnx-dev shut off
>> - vm1 shut off
>>
>>
>>
>> ostara ~ # virsh -c lxc:/// start dwj-hfax-dev
>> error: Failed to start domain dwj-hfax-dev
>> error: End of file while reading data: Input/output error
>> error: One or more references were leaked after disconnect from the
>> hypervisor
>> error: Failed to reconnect to the hypervisor
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20130712/cd407e79/attachment.htm>
More information about the libvirt-users
mailing list