[libvirt-users] VMs fail to start with NUMA configuration

[Osier Yang]

On 2013年01月28日 11:44, Osier Yang wrote:
> On 2013年01月26日 01:07, Doug Goldstein wrote:
>> On Thu, Jan 24, 2013 at 12:58 AM, Osier Yang<jyang at redhat.com> wrote:
>>> On 2013年01月24日 14:26, Doug Goldstein wrote:
>>>>
>>>> On Wed, Jan 23, 2013 at 11:02 PM, Osier Yang<jyang at redhat.com> wrote:
>>>>>
>>>>> On 2013年01月24日 12:11, Doug Goldstein wrote:
>>>>>>
>>>>>>
>>>>>> On Wed, Jan 23, 2013 at 3:45 PM, Doug Goldstein<cardoe at gentoo.org>
>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> I am using libvirt 0.10.2.2 and qemu-kvm 1.2.2 (qemu-kvm 1.2.0 +
>>>>>>> qemu
>>>>>>> 1.2.2 applied on top plus a number of stability patches). Having
>>>>>>> issue
>>>>>>> where my VMs fail to start with the following message:
>>>>>>>
>>>>>>> kvm_init_vcpu failed: Cannot allocate memory
>>>>>
>>>>>
>>>>>
>>>>> Smell likes we have problem on setting the NUMA policy (perhaps
>>>>> caused by the incorrect host NUMA topology), given that the system
>>>>> still has enough memory. Or numad (if it's installed) is doing
>>>>> something wrong.
>>>>>
>>>>> Can you see if there is something about the Nodeset used to set
>>>>> the policy in debug log?
>>>>>
>>>>> E.g.
>>>>>
>>>>> % cat libvirtd.debug | grep Nodeset
>>>>
>>>>
>>>> Well I don't see anything but its likely because I didn't do something
>>>> correct. I had LIBVIRT_DEBUG=1 exported and ran libvirtd --verbose
>>>> from the command line.
>>>
>>>
>>> If the process is in background, it's expected you can't see anything
>>>
>>>
>>> My /etc/libvirt/libvirtd.conf had:
>>>>
>>>> log_outputs="3:syslog:libvirtd 1:file:/tmp/libvirtd.log" But I didn't
>>>> get any debug messages.
>>>
>>>
>>> log_level=1 has to be set.
>>>
>>> Anyway, let's simply do this:
>>>
>>> % service libvirtd stop
>>> % LIBVIRT_DEBUG=1 /usr/sbin/libvirtd 2>&1 | tee -a libvirtd.debug
>>>
>>
>> That's what I was doing, minus the tee just to the console and nothing
>> was coming out. Which is why I added the 1:file:/tmp/libvirtd.log,
>> which also didn't get any debug messages. Turns out this instance must
>> have been built with --disable-debug,
>>
>> All I've got in the log is:
>>
>> # grep -i 'numa' libvirtd.debug
>> 2013-01-25 16:50:15.287+0000: 417: debug : virCommandRunAsync:2200 :
>> About to run /usr/bin/numad -w 2:2048
>> 2013-01-25 16:50:17.295+0000: 417: debug : qemuProcessStart:3614 :
>> Nodeset returned from numad: 1
>
> This looks right.
>
>>
>> Immediately below that is
>>
>> 2013-01-25 16:50:17.295+0000: 417: debug : qemuProcessStart:3622 :
>> Setting up domain cgroup (if required)
>> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupNew:619 : New
>> group /libvirt/qemu/bb-2.6.35.9-i686
>> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupDetect:273 :
>> Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpuacct in
>> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupDetect:273 :
>> Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in
>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:537 :
>> Make group /libvirt/qemu/bb-2.6.35.9-i686
>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:562 :
>> Make controller /sys/fs/cgroup/cpuacct/libvirt/qemu/bb-2.6.35.9-i686/
>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:562 :
>> Make controller /sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/
>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:469
>> : Setting up inheritance /libvirt/qemu ->
>> /libvirt/qemu/bb-2.6.35.9-i686
>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupGetValueStr:361 :
>> Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.cpus
>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed fd 39
>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:482
>> : Inherit cpuset.cpus = 0-63
>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 :
>> Set value
>> '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.cpus'
>> to '0-63'
>
> This looks not right, it should be 0-7 instead.
>
>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed fd 39
>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupGetValueStr:361 :
>> Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.mems
>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed fd 39
>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:482
>> : Inherit cpuset.mems = 0-7
>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 :
>> Set value
>> '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems'
>> to '0-7'
>
> This is right.
>
>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed fd 39
>> 2013-01-25 16:50:17.296+0000: 417: warning : qemuSetupCgroup:388 :
>> Could not autoset a RSS limit for domain bb-2.6.35.9-i686
>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 :
>> Set value
>> '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems'
>> to '1'
>
> And it's strange that the cpuset.mems is changed to '1' here.
>
>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed fd 39
>>
>> Could the RSS issue be related? Some kernel related option not playing
>> nice or enabled?

Instead, I'm wondering if the problem is caused by the mismatch
(from libvirt p.o.v) between cpuset.cpus and cpuset.mems, which
thus cause the problem for kernel memory management?

Osier