[libvirt] Cannot start domain w/xenner emulator
Gerry Reno
greno at verizon.net
Wed Dec 16 04:01:03 UTC 2009
Gerry Reno wrote:
> Gerry Reno wrote:
>> Daniel P. Berrange wrote:
>>> On Tue, Dec 15, 2009 at 12:02:50AM -0500, Gerry Reno wrote:
>>>> # virsh start Ubuntu-domU-x86-1
>>>> Connecting to uri: qemu:///system
>>>> error: Failed to start domain Ubuntu-domU-x86-1
>>>> error: internal error unable to start guest: libvir: error : cannot
>>>> execute binary /usr/bin/xenner: Permission denied
>>>>
>>>> # ls -l /usr/bin/xenner
>>>> -rwxr-xr-x 1 root root 118936 2009-12-14 17:05 /usr/bin/xenner
>>>>
>>>> Sure looks executable. And there's no SELinux on the box.
>>>>
>>>> libvirt is 0.7.0
>>>>
>>>> What the..?
>>>
>>> Seriously bizarre ! What distro ? You might want to strace the
>>> libvirtd daemon
>>>
>>> strace -f -o trace.log $PID-OF-LIBVIRTD
>>>
>>> and then look in trace.log for EPERM / EACCESS on any syscall.
>>>
>>> Daniel
>>
>>
>> The host is Ubuntu 9.10. And apparmor has been disabled but what's
>> strange is that it looks like the kernel is loading a profile and
>> then denying access and then removing the profile.
>>
>> # virsh start Ubuntu-domU-x86-1
>> Connecting to uri: qemu:///system
>> error: Failed to start domain Ubuntu-domU-x86-1
>> error: internal error unable to start guest: libvir: error : cannot
>> execute binary /usr/bin/xenner: Permission denied
>>
>> # ls -l /usr/bin/xenner
>> -rwxr-xr-x 1 root root 118936 2009-12-14 17:05 /usr/bin/xenner
>>
>> + apparmor_status
>> apparmor module is loaded.
>> 0 profiles are loaded.
>> 0 profiles are in enforce mode.
>> 0 profiles are in complain mode.
>> 0 processes have profiles defined.
>> 0 processes are in enforce mode :
>> 0 processes are in complain mode.
>> 0 processes are unconfined but have a profile defined.
>>
>>
>> # /var/log/kern.log:
>> Dec 15 11:28:35 grp-01-23-02 kernel: [213217.260223] type=1503
>> audit(1260894515.683:73): operation="exec" pid=16770 parent=16769
>> profile="libvirt-74367128-9bd6-3264-3833-f661c47b464e"
>> requested_mask="x::" denied_mask="x::" fsuid=0 ouid=0
>> name="/usr/bin/xenner"
>> Dec 15 11:28:35 grp-01-23-02 kernel: [213217.305593] br0: port
>> 2(vnet0) entering disabled state
>> Dec 15 11:28:35 grp-01-23-02 kernel: [213217.344700] device vnet0
>> left promiscuous mode
>> Dec 15 11:28:35 grp-01-23-02 kernel: [213217.344703] br0: port
>> 2(vnet0) entering disabled state
>> Dec 15 11:28:35 grp-01-23-02 kernel: [213217.560461] type=1505
>> audit(1260894515.976:74): operation="profile_remove" pid=16772
>> name=libvirt-74367128-9bd6-3264-3833-f661c47b464e namespace=default
>> Dec 15 11:30:20 grp-01-23-02 kernel: [213322.133717] type=1505
>> audit(1260894620.556:75): operation="profile_load" pid=16995
>> name=libvirt-74367128-9bd6-3264-3833-f661c47b464e
>> Dec 15 11:30:20 grp-01-23-02 kernel: [213322.137277] device vnet0
>> entered promiscuous mode
>> Dec 15 11:30:20 grp-01-23-02 kernel: [213322.138430] br0: port
>> 2(vnet0) entering learning state
>> Dec 15 11:30:20 grp-01-23-02 kernel: [213322.139715] type=1503
>> audit(1260894620.556:76): operation="exec" pid=16999 parent=16998
>> profile="libvirt-74367128-9bd6-3264-3833-f661c47b464e"
>> requested_mask="x::" denied_mask="x::" fsuid=0 ouid=0
>> name="/usr/bin/xenner"
>> Dec 15 11:30:20 grp-01-23-02 kernel: [213322.181318] br0: port
>> 2(vnet0) entering disabled state
>> Dec 15 11:30:20 grp-01-23-02 kernel: [213322.220628] device vnet0
>> left promiscuous mode
>> Dec 15 11:30:20 grp-01-23-02 kernel: [213322.220631] br0: port
>> 2(vnet0) entering disabled state
>> Dec 15 11:30:20 grp-01-23-02 kernel: [213322.359241] type=1505
>> audit(1260894620.773:77): operation="profile_remove" pid=17001
>> name=libvirt-74367128-9bd6-3264-3833-f661c47b464e namespace=default
>>
>>
>> -Gerry
>>
>>
> For the moment I got by the apparmor profile issue by completely
> removing the package.
>
> The domU guest still does not start however:
>
> + virsh start Ubuntu-domU-x86-1
> Connecting to uri: qemu:///system
> error: Failed to start domain Ubuntu-domU-x86-1
> error: internal error unable to start guest: [xenner,1]
> qemu_monitor_config:
> unix:/var/run/libvirt/qemu/Ubuntu-domU-x86-1.monitor,server,nowait
> [xenner,1] qemu_serial_config: pty
> [xenner,1] qemu_disk_config_blkbackd:
> file=/var/lib/libvirt/images/ubuntu.9-10.x86.img,if=scsi,index=0
> [xenner,1] qemu_disk_config_blkbackd: if != xen, ignoring disk
> [xenner,1] qemu_net_config_netbackd: nic,macaddr=54:52:07:2d:e1:81,vlan=0
> [xenner,1] qemu_net_config_netbackd: 0: mac="54:52:07:2d:e1:81"
> [xenner,1] qemu_net_config_netbackd: tap,fd=16,vlan=0
> [xenner,1] qemu_net_config_netbackd: 0: if="vnet2"
> started as: "/usr/bin/xenner" "-S" "-M" "pc" "-m" "2048" "-smp" "4"
> "-name" "Ubuntu-domU-x86-1" "-uuid"
> "74367128-9bd6-3264-3833-f661c47b464e" "-domid" "6" "-nographic"
> "-monitor"
> "unix:/var/run/libvirt/qemu/Ubuntu-domU-x86-1.monitor,server,nowait"
> "-boot" "c" "-kernel"
> "/home/greno/xen/domU/x86/ubuntu-9.10-karmic/kernel/boot/vmlinuz-2.6.31.6"
> "-append" "xencons=xvc console=tty1 console=xvc0" "-drive"
> "file=/var/lib/libvirt/images/ubuntu.9-10.x86.img,i
>
> It cuts the command line off as you can see in the output but if I
> take that xenner command line from the log and run it manually then
> the domU guest starts fine. So is there a workaround to this
> qemu_monitor_config error?
>
> -Gerry
>
>
Ok, I set bus='xen' and it no longer ignores the disk but now xenstored
segfaults. Could this be because the image device is 'sda' and the bus
is 'xen'?
Here is the session and logs:
####################################################################
+ virsh start Ubuntu-domU-x86-1
Connecting to uri: qemu:///system
error: Failed to start domain Ubuntu-domU-x86-1
error: internal error unable to start guest: [xenner,1]
qemu_monitor_config:
unix:/var/run/libvirt/qemu/Ubuntu-domU-x86-1.monitor,server,nowait
[xenner,1] qemu_serial_config: pty
[xenner,1] qemu_disk_config_blkbackd:
file=/var/lib/libvirt/images/ubuntu.9-10.x86.img,if=xen,index=0
[xenner,1] qemu_net_config_netbackd: nic,macaddr=54:52:07:2d:e1:81,vlan=0
[xenner,1] qemu_net_config_netbackd: 0: mac="54:52:07:2d:e1:81"
[xenner,1] qemu_net_config_netbackd: tap,fd=16,vlan=0
[xenner,1] qemu_net_config_netbackd: 0: if="vnet3"
started as: "/usr/bin/xenner" "-S" "-M" "pc" "-m" "2048" "-smp" "4"
"-name" "Ubuntu-domU-x86-1" "-uuid"
"4027c80e-b5d5-a986-18ed-5887d623d0bb" "-domid" "9" "-nographic"
"-monitor"
"unix:/var/run/libvirt/qemu/Ubuntu-domU-x86-1.monitor,server,nowait"
"-boot" "c" "-kernel"
"/home/greno/xen/domU/x86/ubuntu-9.10-karmic/kernel/boot/vmlinuz-2.6.31.6"
"-append" "xencons=xvc console=tty1 console=xvc0" "-drive"
"file=/var/lib/libvirt/images/ubuntu.9-10.x86.img,if=xen,index=0" "-net"
"nic,macaddr=54:52:07:2d:e1:81,vlan=0" "-n
## /var/log/messages: xenstored segfaults when we 'virsh start...'
Dec 15 19:47:47 grp-01-23-02 kernel: [10087.283379] xenstored[23440]:
segfault at 0 ip 000000000040672c sp 00007fff3576e190 error 4 in
xenstored[400000+12000]
## domain log:
LC_ALL=C
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
/usr/bin/xenner -S -M pc -m 2048 -smp 4 -name Ubuntu-domU-x86-1 -uuid
4027c80e-b5d5-a986-18ed-5887d623d0bb -domid 8 -nographic -monitor
unix:/var/run/libvirt/qemu/Ubuntu-domU-x86-1.monitor,server,nowait -boot
c -kernel
/home/greno/xen/domU/x86/ubuntu-9.10-karmic/kernel/boot/vmlinuz-2.6.31.6
-append xencons=xvc console=tty1 console=xvc0 -drive
file=/var/lib/libvirt/images/ubuntu.9-10.x86.img,if=xen,index=0 -net
nic,macaddr=54:52:07:2d:e1:81,vlan=0 -net tap,fd=16,vlan=0 -serial pty
-parallel none -usb
[xenner,1] qemu_monitor_config:
unix:/var/run/libvirt/qemu/Ubuntu-domU-x86-1.monitor,server,nowait
[xenner,1] qemu_serial_config: pty
[xenner,1] qemu_disk_config_blkbackd:
file=/var/lib/libvirt/images/ubuntu.9-10.x86.img,if=xen,index=0
[xenner,1] qemu_net_config_netbackd: nic,macaddr=54:52:07:2d:e1:81,vlan=0
[xenner,1] qemu_net_config_netbackd: 0: mac="54:52:07:2d:e1:81"
[xenner,1] qemu_net_config_netbackd: tap,fd=16,vlan=0
[xenner,1] qemu_net_config_netbackd: 0: if="vnet2"
started as: "/usr/bin/xenner" "-S" "-M" "pc" "-m" "2048" "-smp" "4"
"-name" "Ubuntu-domU-x86-1" "-uuid"
"4027c80e-b5d5-a986-18ed-5887d623d0bb" "-domid" "8" "-nographic"
"-monitor"
"unix:/var/run/libvirt/qemu/Ubuntu-domU-x86-1.monitor,server,nowait"
"-boot" "c" "-kernel"
"/home/greno/xen/domU/x86/ubuntu-9.10-karmic/kernel/boot/vmlinuz-2.6.31.6"
"-append" "xencons=xvc console=tty1 console=xvc0" "-drive"
"file=/var/lib/libvirt/images/ubuntu.9-10.x86.img,if=xen,index=0" "-net"
"nic,macaddr=54:52:07:2d:e1:81,vlan=0" "-net" "tap,fd=16,vlan=0"
"-serial" "pty" "-parallel" "none" "-usb"
kvm capability clocksource: ok
kvm capability nop-iodelay: ok
kvm capability mmu-op: not supported
==================== setup ====================
xc_dom_allocate: cmdline="xencons=xvc console=tty1 console=xvc0",
features="(null)"
xen_guest_parse: ----- parse kernel -----
xc_dom_kernel_file:
filename="/home/greno/xen/domU/x86/ubuntu-9.10-karmic/kernel/boot/vmlinuz-2.6.31.6"
xc_dom_malloc_filemap : 1820 kB
xc_dom_parse_image: called
xc_dom_find_loader: trying ELF-generic loader ... failed
xc_dom_find_loader: trying Linux bzImage loader ... xc_dom_malloc : 3620 kB
xc_dom_do_gunzip: unzip ok, 0x1bf9e6 -> 0x3891a4
OK
elf_parse_binary: phdr: paddr=0x1000000 memsz=0x2a0000
elf_parse_binary: phdr: paddr=0x12a0000 memsz=0x1f3000
elf_parse_binary: memory: 0x1000000 -> 0x1493000
elf_xen_parse_note: GUEST_OS = "linux"
elf_xen_parse_note: GUEST_VERSION = "2.6"
elf_xen_parse_note: XEN_VERSION = "xen-3.0"
elf_xen_parse_note: VIRT_BASE = 0xc0000000
elf_xen_parse_note: ENTRY = 0xc12de000
elf_xen_parse_note: HYPERCALL_PAGE = 0xc1002000
elf_xen_parse_note: FEATURES = "!writable_page_tables|pae_pgdir_above_4gb"
elf_xen_parse_note: PAE_MODE = "yes"
elf_xen_parse_note: LOADER = "generic"
elf_xen_parse_note: unknown xen elf note (0xd)
elf_xen_parse_note: SUSPEND_CANCEL = 0x1
elf_xen_parse_note: HV_START_LOW = 0xf5800000
elf_xen_parse_note: PADDR_OFFSET = 0x0
elf_xen_addr_calc_check: addresses:
virt_base = 0xc0000000
elf_paddr_offset = 0x0
virt_offset = 0xc0000000
virt_kstart = 0xc1000000
virt_kend = 0xc1493000
virt_entry = 0xc12de000
xc_dom_parse_elf_kernel: xen-3.0-x86_32p: 0xc1000000 -> 0xc1493000
[xenner,1] domain_builder: memory: emu 4 MB, m2p 4 MB, guest 2040 MB
xen_emu_load: ----- load xen emu -----
[xenner,1] xen_load_emu_file: loading /usr/lib/xenner/emu32.elf (186892
bytes)
elf_parse_binary: phdr: paddr=0xff000000 memsz=0x2fdc0
elf_parse_binary: memory: 0xff000000 -> 0xff02fdc0
elf_load_binary: phdr 0 at 0x0x7f2f51979000 -> 0x0x7f2f519a5760
xen_emu_setup: ----- memory info -----
xen_emu_setup: ----- emu pgd setup -----
xen_guest_setup: ----- memory setup -----
xc_dom_mem_init: mem 2040 MB, pages 0x7f800 pages, 4k each
xc_dom_mem_init: 0x7f800 pages
xc_dom_malloc : 4080 kB
xen_guest_setup: ----- create start-of-day -----
xc_dom_build_image: called
xc_dom_alloc_segment: kernel : 0xc1000000 -> 0xc1493000 (pfn 0x1000 +
0x493 pages)
xc_dom_pfn_to_ptr: anonymous memory: pfn 0x1000+0x493 at 0x7f2f50b31000
elf_load_binary: phdr 0 at 0x0x7f2f50b31000 -> 0x0x7f2f50dd1000
elf_load_binary: phdr 1 at 0x0x7f2f50dd1000 -> 0x0x7f2f50e61000
xc_dom_alloc_segment: phys2mach : 0xc1493000 -> 0xc1691000 (pfn 0x1493 +
0x1fe pages)
xc_dom_pfn_to_ptr: anonymous memory: pfn 0x1493+0x1fe at 0x7f2f50933000
xc_dom_alloc_page : start info : 0xc1691000 (pfn 0x1691)
xc_dom_alloc_page : xenstore : 0xc1692000 (pfn 0x1692)
xc_dom_alloc_page : console : 0xc1693000 (pfn 0x1693)
nr_page_tables: 0x00000000ffffffff/32: 0x0000000000000000 ->
0x00000000ffffffff, 1 table(s)
nr_page_tables: 0x000000003fffffff/30: 0x00000000c0000000 ->
0x00000000ffffffff, 1 table(s)
nr_page_tables: 0x00000000001fffff/21: 0x00000000c0000000 ->
0x00000000c17fffff, 12 table(s)
xc_dom_alloc_segment: page tables : 0xc1694000 -> 0xc16a2000 (pfn 0x1694
+ 0xe pages)
xc_dom_pfn_to_ptr: anonymous memory: pfn 0x1694+0xe at 0x7f2f50925000
xc_dom_alloc_page : boot stack : 0xc16a2000 (pfn 0x16a2)
xc_dom_build_image : virt_alloc_end : 0xc16a3000
xc_dom_build_image : virt_pgtab_end : 0xc1800000
xc_dom_update_guest_p2m: dst 32bit, pages 0x7f800
xc_dom_pfn_to_ptr: anonymous memory: pfn 0x1693+0x1 at 0x7f2f50924000
xc_dom_pfn_to_ptr: anonymous memory: pfn 0x1692+0x1 at 0x7f2f50923000
xc_dom_pfn_to_ptr: anonymous memory: pfn 0x1691+0x1 at 0x7f2f50922000
start_info_x86_32: called
xen_guest_setup: ----- setup xen hypercall page -----
xen_guest_setup: ----- setup vcpu context -----
vcpu_x86_32: called
vcpu_x86_32: cr3: pfn 0x1694 mfn 0x1e94
domain_builder: ----- kvm: state setup -----
xc_dom_release: called
domain_builder: ----- all done -----
[xenner,1] xenstore_device_net: vnic 0: mac="54:52:07:2d:e1:81"
if="vnet2" br=""
calibrate tsc ... LC_ALL=C
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
/usr/bin/xenner -S -M pc -m 2048 -smp 4 -name Ubuntu-domU-x86-1 -uuid
4027c80e-b5d5-a986-18ed-5887d623d0bb -domid 9 -nographic -monitor
unix:/var/run/libvirt/qemu/Ubuntu-domU-x86-1.monitor,server,nowait -boot
c -kernel
/home/greno/xen/domU/x86/ubuntu-9.10-karmic/kernel/boot/vmlinuz-2.6.31.6
-append xencons=xvc console=tty1 console=xvc0 -drive
file=/var/lib/libvirt/images/ubuntu.9-10.x86.img,if=xen,index=0 -net
nic,macaddr=54:52:07:2d:e1:81,vlan=0 -net tap,fd=16,vlan=0 -serial pty
-parallel none -usb
[xenner,1] qemu_monitor_config:
unix:/var/run/libvirt/qemu/Ubuntu-domU-x86-1.monitor,server,nowait
[xenner,1] qemu_serial_config: pty
[xenner,1] qemu_disk_config_blkbackd:
file=/var/lib/libvirt/images/ubuntu.9-10.x86.img,if=xen,index=0
[xenner,1] qemu_net_config_netbackd: nic,macaddr=54:52:07:2d:e1:81,vlan=0
[xenner,1] qemu_net_config_netbackd: 0: mac="54:52:07:2d:e1:81"
[xenner,1] qemu_net_config_netbackd: tap,fd=16,vlan=0
[xenner,1] qemu_net_config_netbackd: 0: if="vnet3"
started as: "/usr/bin/xenner" "-S" "-M" "pc" "-m" "2048" "-smp" "4"
"-name" "Ubuntu-domU-x86-1" "-uuid"
"4027c80e-b5d5-a986-18ed-5887d623d0bb" "-domid" "9" "-nographic"
"-monitor"
"unix:/var/run/libvirt/qemu/Ubuntu-domU-x86-1.monitor,server,nowait"
"-boot" "c" "-kernel"
"/home/greno/xen/domU/x86/ubuntu-9.10-karmic/kernel/boot/vmlinuz-2.6.31.6"
"-append" "xencons=xvc console=tty1 console=xvc0" "-drive"
"file=/var/lib/libvirt/images/ubuntu.9-10.x86.img,if=xen,index=0" "-net"
"nic,macaddr=54:52:07:2d:e1:81,vlan=0" "-net" "tap,fd=16,vlan=0"
"-serial" "pty" "-parallel" "none" "-usb"
kvm capability clocksource: ok
kvm capability nop-iodelay: ok
kvm capability mmu-op: not supported
==================== setup ====================
[xenner,0] xenstore_init_early: can't connect to
/var/run/xenstored/socket (xenstored not running?)
xenner_cleanup: ----- cleaning up -----
####################################################################
And again if I run this command from the command line it works and
xenstored does not crash.
-Gerry
More information about the libvir-list
mailing list