[libvirt-users] VMs fail to start with NUMA configuration

[Osier Yang]

On 2013年01月29日 00:17, Doug Goldstein wrote:
> On Sun, Jan 27, 2013 at 10:46 PM, Osier Yang<jyang at redhat.com>  wrote:
>> On 2013年01月28日 11:47, Osier Yang wrote:
>>>
>>> On 2013年01月28日 11:44, Osier Yang wrote:
>>>>
>>>> On 2013年01月26日 01:07, Doug Goldstein wrote:
>>>>>
>>>>> On Thu, Jan 24, 2013 at 12:58 AM, Osier Yang<jyang at redhat.com>  wrote:
>>>>>>
>>>>>> On 2013年01月24日 14:26, Doug Goldstein wrote:
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jan 23, 2013 at 11:02 PM, Osier Yang<jyang at redhat.com>  wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2013年01月24日 12:11, Doug Goldstein wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Jan 23, 2013 at 3:45 PM, Doug Goldstein<cardoe at gentoo.org>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I am using libvirt 0.10.2.2 and qemu-kvm 1.2.2 (qemu-kvm 1.2.0 +
>>>>>>>>>> qemu
>>>>>>>>>> 1.2.2 applied on top plus a number of stability patches). Having
>>>>>>>>>> issue
>>>>>>>>>> where my VMs fail to start with the following message:
>>>>>>>>>>
>>>>>>>>>> kvm_init_vcpu failed: Cannot allocate memory
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Smell likes we have problem on setting the NUMA policy (perhaps
>>>>>>>> caused by the incorrect host NUMA topology), given that the system
>>>>>>>> still has enough memory. Or numad (if it's installed) is doing
>>>>>>>> something wrong.
>>>>>>>>
>>>>>>>> Can you see if there is something about the Nodeset used to set
>>>>>>>> the policy in debug log?
>>>>>>>>
>>>>>>>> E.g.
>>>>>>>>
>>>>>>>> % cat libvirtd.debug | grep Nodeset
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Well I don't see anything but its likely because I didn't do something
>>>>>>> correct. I had LIBVIRT_DEBUG=1 exported and ran libvirtd --verbose
>>>>>>> from the command line.
>>>>>>
>>>>>>
>>>>>>
>>>>>> If the process is in background, it's expected you can't see anything
>>>>>>
>>>>>>
>>>>>> My /etc/libvirt/libvirtd.conf had:
>>>>>>>
>>>>>>>
>>>>>>> log_outputs="3:syslog:libvirtd 1:file:/tmp/libvirtd.log" But I didn't
>>>>>>> get any debug messages.
>>>>>>
>>>>>>
>>>>>>
>>>>>> log_level=1 has to be set.
>>>>>>
>>>>>> Anyway, let's simply do this:
>>>>>>
>>>>>> % service libvirtd stop
>>>>>> % LIBVIRT_DEBUG=1 /usr/sbin/libvirtd 2>&1 | tee -a libvirtd.debug
>>>>>>
>>>>>
>>>>> That's what I was doing, minus the tee just to the console and nothing
>>>>> was coming out. Which is why I added the 1:file:/tmp/libvirtd.log,
>>>>> which also didn't get any debug messages. Turns out this instance must
>>>>> have been built with --disable-debug,
>>>>>
>>>>> All I've got in the log is:
>>>>>
>>>>> # grep -i 'numa' libvirtd.debug
>>>>> 2013-01-25 16:50:15.287+0000: 417: debug : virCommandRunAsync:2200 :
>>>>> About to run /usr/bin/numad -w 2:2048
>>>>> 2013-01-25 16:50:17.295+0000: 417: debug : qemuProcessStart:3614 :
>>>>> Nodeset returned from numad: 1
>>>>
>>>>
>>>> This looks right.
>>>>
>>>>>
>>>>> Immediately below that is
>>>>>
>>>>> 2013-01-25 16:50:17.295+0000: 417: debug : qemuProcessStart:3622 :
>>>>> Setting up domain cgroup (if required)
>>>>> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupNew:619 : New
>>>>> group /libvirt/qemu/bb-2.6.35.9-i686
>>>>> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupDetect:273 :
>>>>> Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpuacct in
>>>>> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupDetect:273 :
>>>>> Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:537 :
>>>>> Make group /libvirt/qemu/bb-2.6.35.9-i686
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:562 :
>>>>> Make controller /sys/fs/cgroup/cpuacct/libvirt/qemu/bb-2.6.35.9-i686/
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:562 :
>>>>> Make controller /sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:469
>>>>> : Setting up inheritance /libvirt/qemu ->
>>>>> /libvirt/qemu/bb-2.6.35.9-i686
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupGetValueStr:361 :
>>>>> Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.cpus
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed
>>>>> fd 39
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:482
>>>>> : Inherit cpuset.cpus = 0-63
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 :
>>>>> Set value
>>>>> '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.cpus'
>>>>> to '0-63'
>>>>
>>>>
>>>> This looks not right, it should be 0-7 instead.
>>>>
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed
>>>>> fd 39
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupGetValueStr:361 :
>>>>> Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.mems
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed
>>>>> fd 39
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:482
>>>>> : Inherit cpuset.mems = 0-7
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 :
>>>>> Set value
>>>>> '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems'
>>>>> to '0-7'
>>>>
>>>>
>>>> This is right.
>>>>
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed
>>>>> fd 39
>>>>> 2013-01-25 16:50:17.296+0000: 417: warning : qemuSetupCgroup:388 :
>>>>> Could not autoset a RSS limit for domain bb-2.6.35.9-i686
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 :
>>>>> Set value
>>>>> '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems'
>>>>> to '1'
>>>>
>>>>
>>>> And it's strange that the cpuset.mems is changed to '1' here.
>>
>>
>> Oh, actually this is right, cpuset.mems is about the memory nodes.
>>
>>
>>>>
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed
>>>>> fd 39
>>>>>
>>>>> Could the RSS issue be related? Some kernel related option not playing
>>>>> nice or enabled?
>>>
>>>
>>> Instead, I'm wondering if the problem is caused by the mismatch
>>> (from libvirt p.o.v) between cpuset.cpus and cpuset.mems, which
>>> thus cause the problem for kernel memory management?
>>
>>
>> So, the simple method to prove the guess is to use static placement
>> like:
>>
>> <vcpu placement='static' cpuset='0-63'>2</vcpu>
>> <numatune>
>>    <memory placement='static' nodeset='1'/>
>> </numatune>
>>
>> Osier
>
> Same error. Which I don't know if you expected or didn't expect.
>

It's expected. as "0-63" is the final result when using "auto"
placement.