[libvirt] [PATCH v2 0/2] qemu: Honor memory mode='strict'

[Daniel Henrique Barboza]

On 4/12/19 6:10 AM, Michal Privoznik wrote:
> On 4/11/19 7:29 PM, Daniel Henrique Barboza wrote:
>>
>>
>> On 4/11/19 11:56 AM, Michal Privoznik wrote:
>>> On 4/11/19 4:23 PM, Daniel Henrique Barboza wrote:
>>>> Hi,
>>>>
>>>> I've tested these patches again, twice, in similar setups like I 
>>>> tested
>>>> the first version (first in a Power8, then in a Power9 server).
>>>>
>>>> Same results, though. Libvirt will not avoid the launch of a 
>>>> pseries guest,
>>>> with numanode=strict, even if the numa node does not have available
>>>> RAM. If I stress test the memory of the guest to force the allocation,
>>>> QEMU exits with an error as soon as the memory of the host numa node
>>>> is exhausted.
>>>
>>> Yes, this is expected. I mean, by default qemu doesn't allocate 
>>> memory for the guest fully. You'd have to force it:
>>>
>>> <memoryBacking>
>>>   <allocation mode='immediate'/>
>>> </memoryBacking>
>>>
>>
>> Tried with this extra setting, still no good. Domain still boots, 
>> even if
>> there is not enough memory to load up all its ram in the NUMA node
>> I am setting. For reference, this is the top of the guest XML:
>>
>>
>>    <name>vm1</name>
>>    <uuid>f48e9e35-8406-4784-875f-5185cb4d47d7</uuid>
>>    <memory unit='KiB'>314572800</memory>
>>    <currentMemory unit='KiB'>314572800</currentMemory>
>>    <memoryBacking>
>>      <allocation mode='immediate'/>
>>    </memoryBacking>
>>    <vcpu placement='static'>16</vcpu>
>>    <numatune>
>>      <memory mode='strict' nodeset='0'/>
>>    </numatune>
>>    <os>
>>      <type arch='ppc64' machine='pseries'>hvm</type>
>>      <boot dev='hd'/>
>>    </os>
>>    <clock offset='utc'/>
>>
>> While doing this test, I recalled that some of my IBM peers recently
>> mentioned that they were unable to do a pre-allocation of the RAM
>> of a pseries guest using Libvirt, but they were able to do it using QEMU
>> directly (using -realtime mlock=on). In fact, I just tried it out 
>> with command
>> line QEMU and the guest allocated all the memory at boot.
>
> Ah, so looks like -mem-prealloc doesn't work at Power? Can you please 
> check:
>
> 1) that -mem-prealloc is on the qemu command line

Yes. This is the cmd line generated:

/usr/bin/qemu-system-ppc64 \
-name guest=vm1,debug-threads=on \
-S \
-object 
secret,id=masterKey0,format=raw,file=/home/user/usr/var/lib/libvirt/qemu/domain-2-vm1/master-key.aes 
\
-machine pseries-2.11,accel=kvm,usb=off,dump-guest-core=off \
-bios /home/user/boot_rom.bin \
-m 307200 \
-mem-prealloc \
-realtime mlock=off \
-smp 16,sockets=16,cores=1,threads=1 \
-uuid f48e9e35-8406-4784-875f-5185cb4d47d7 \
-display none \
-no-user-config \
-nodefaults \
-chardev 
socket,id=charmonitor,path=/home/user/usr/var/lib/libvirt/qemu/domain-2-vm1/monitor.sock,server,nowait 
\
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-shutdown \
-boot strict=on \
-device spapr-pci-host-bridge,index=1,id=pci.1 \
-device qemu-xhci,id=usb,bus=pci.0,addr=0x3 \
-drive 
file=/home/user/nv2-vm1.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 \
-device 
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 
\
-chardev pty,id=charserial0 \
-device spapr-vty,chardev=charserial0,id=serial0,reg=0x30000000 \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x1 \
-sandbox 
on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on




> 2) how much memory qemu allocates right after it started the guest? I 
> mean, before you start some mem stress test which causes it to 
> allocate the memory fully.

It starts with 300Gb. It depletes its assigned NUMA node (that has 256Gb),
then it takes ~70Gb from another NUMA node to complete the 300Gb.


>
>>
>> This means that the pseries guest is able to do mem pre-alloc. I'd 
>> say that
>> there might be something missing somewhere (XML, host setup, libvirt
>> config ...) or perhaps even a bug that is preventing Libvirt from doing
>> this pre-alloc. This explains why I can't verify this patch series. 
>> I'll see if
>> I dig it further to understand why when I have the time.
>
> Yeah, I don't know Power well enough to help you. Sorry.


No problem. One question: Libvirt is supposed to let the VM do the full
allocation of its RAM using -mem-prealloc and with -realtime mlock=off,
is that correct?


Thanks,

DHB


>
> Michal