[libvirt] 1GB huge pages and incompatible VM memory size

[Jim Fehlig]

On 02/23/2017 08:50 AM, Michal Privoznik wrote:
> On 02/23/2017 03:26 PM, Matt Fleming wrote:
>> On Tue, 21 Feb, at 11:23:52AM, Michal Privoznik wrote:
>>>
>>> Jim,
>>>
>>> what is the actual problem? I've tried to reproduce this by running vm
>>> with 3.5G RAM backed by 1GB huge pages and the guest runs just fine. I
>>> mean kvm guest. This is because at the cmd line level there is aligned
>>> value:
>>>
>>> -m size=4194304k,slots=16,maxmem=8388608k
>>>
>>> This is result of qemuDomainAlignMemorySizes(). So perhaps there's a bug
>>> somewhere in the function?
>>
>> Quite possibly. Some memory values work fine and appear to be rounded
>> to the next gigabyte boundary. One KiB values that fails for me is,
>>
>>   <memory unit='KiB'>52428801</memory>
>>   <currentMemory unit='KiB'>52428801</currentMemory>
>>
>> Which results in qemu-kvm ... -m 51201
>
> I currently don't have access to a host with 50+GB of RAM, so I start small:
>
>   <memory unit='KiB'>3145739</memory>
>   <currentMemory unit='KiB'>3145739</currentMemory>
>   <memoryBacking>
>     <hugepages>
>       <page size='1048576' unit='KiB'/>
>     </hugepages>
>   </memoryBacking>
>
>
>  -m 3073
>
>
> And this works just fine. One thing that I've noticed is that we don't
> take into account sizes of huge pages in qemuDomainAlignMemorySizes().
> But then again - what is the scenario you're seeing? What's the error
> message?

Sorry for the delay. I was having problems reproducing it until Matt reminded me 
that some vNUMA was needed in the mix. E.g. config containing

   <memory unit='KiB'>10000000</memory>
   <currentMemory unit='KiB'>10000000</currentMemory>
   <memoryBacking>
     <hugepages>
       <page size='1048576' unit='KiB'/>
     </hugepages>
   </memoryBacking>
   <cpu mode='host-passthrough'>
     <topology sockets='2' cores='20' threads='2'/>
     <numa>
       <cell id='0' cpus='0-39' memory='5000000' unit='KiB'/>
       <cell id='1' cpus='40-79' memory='5000000' unit='KiB'/>
     </numa>
   </cpu>

fails with

2017-03-14T21:50:01.228693Z qemu-system-x86_64: -object 
memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,size=5120196608,host-nodes=0-1,policy=bind: 
cannot bind memory to host NUMA nodes: Invalid argument

Changing the config by s/10000000/10485760/ and s/5000000/5242880/ works. It was 
only then that I remembered the machine was running libvirt 2.0.0 :-/. I updated 
to 3.1.0 and the original, unaligned config works. So there are some fixes in 
the alignment code in the meanwhile. I'll test some of the other problematic 
configurations, which in some cases resulted in different qemu errors, and 
report back if I find any problems with current code.

Regards,
Jim