[libvirt-users] VMs fail to start with NUMA configuration

[Osier Yang]

On 2013年01月24日 14:26, Doug Goldstein wrote:
> On Wed, Jan 23, 2013 at 11:02 PM, Osier Yang<jyang at redhat.com>  wrote:
>> On 2013年01月24日 12:11, Doug Goldstein wrote:
>>>
>>> On Wed, Jan 23, 2013 at 3:45 PM, Doug Goldstein<cardoe at gentoo.org>   wrote:
>>>>
>>>> I am using libvirt 0.10.2.2 and qemu-kvm 1.2.2 (qemu-kvm 1.2.0 + qemu
>>>> 1.2.2 applied on top plus a number of stability patches). Having issue
>>>> where my VMs fail to start with the following message:
>>>>
>>>> kvm_init_vcpu failed: Cannot allocate memory
>>
>>
>> Smell likes we have problem on setting the NUMA policy (perhaps
>> caused by the incorrect host NUMA topology), given that the system
>> still has enough memory. Or numad (if it's installed) is doing
>> something wrong.
>>
>> Can you see if there is something about the Nodeset used to set
>> the policy in debug log?
>>
>> E.g.
>>
>> % cat libvirtd.debug | grep Nodeset
>
> Well I don't see anything but its likely because I didn't do something
> correct. I had LIBVIRT_DEBUG=1 exported and ran libvirtd --verbose
> from the command line.

If the process is in background, it's expected you can't see anything

My /etc/libvirt/libvirtd.conf had:
> log_outputs="3:syslog:libvirtd 1:file:/tmp/libvirtd.log" But I didn't
> get any debug messages.

log_level=1 has to be set.

Anyway, let's simply do this:

% service libvirtd stop
% LIBVIRT_DEBUG=1 /usr/sbin/libvirtd 2>&1 | tee -a libvirtd.debug


>
>> % cat /var/log/libvirt/qemu/$guest.log
>
> Nothing in here. Just the command line:
>
> 2013-01-24 04:00:49.989+0000: starting up
> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/
> bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3 HOME=/root USER=root LOGNAME=root QEM
> U_AUDIO_DRV=none /usr/bin/qemu-kvm -name bb-2.6.18-128.el5.x86_64 -S -M pc-1.2 -
> cpu Opteron_G4,+perfctr_nb,+perfctr_core,+topoext,+nodeid_msr,+lwp,+wdt,+skinit,
> +ibs,+osvw,+cr8legacy,+extapic,+cmp_legacy,+fxsr_opt,+mmxext,+osxsave,+monitor,+
> ht,+vme -enable-kvm -m 2048 -smp 2,sockets=1,cores=2,threads=1 -uuid
> 2b5990f0-649f-ae25-99f0-dc4b05f682e1 -no-user-config -nodefaults
> -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/bb-2.6.18-128.el5.x86_64.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc
> base=utc,driftfix=slew -no-shutdown -boot menu=off -device
> piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
> file=/dev/disk/by-path/ip-192.168.200.20:3260-iscsi-iqn.2011-07.lab.san-1:2.6.18-128.el5-x86_64-lun-0,if=none,id=drive-ide0-0-0,format=raw,cache=none,werror=stop,rerror=stop,aio=native
> -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
> -netdev tap,fd=31,id=hostnet0 -device
> rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:7c:b2:81,bus=pci.0,addr=0x7
> -chardev pty,id=charserial0 -device
> isa-serial,chardev=charserial0,id=serial0 -vnc 0.0.0.0:10,password
> -vga cirrus -device AC97,id=sound0,bus=pci.0,addr=0x4 -device
> i6300esb,id=watchdog0,bus=pci.0,addr=0x6 -watchdog-action reset
> -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
> char device redirected to /dev/pts/12
> CPU feature perfctr_nb not found
> CPU feature perfctr_core not found
> CPU feature topoext not found
> CPU feature lwp not found
> kvm_init_vcpu failed: Cannot allocate memory
> 2013-01-24 04:00:50.194+0000: shutting down
>
>>
>> On the other hand, I'm wondering if there is a chance to see the
>> full error msg kernel throws out.
>>
>
> That's what I've been trying to figure out. Trying to get some more
> info out of libvirt as well. If you've got any ideas I'll give them a
> shot.
>