[libvirt] Problem with setting up KVM guests to use HugePages

Clarylin L clearasu at gmail.com
Thu Jun 11 21:54:33 UTC 2015


With older qemu-1.5.3, I can start the guest by removing the following
lines from the xml config file and hugepages are correctly used.

    <numa>

      <cell id='0' cpus='0-15' memory='67108864'/>

      <cell id='1' cpus='16-31' memory='67108864'/>

    </numa>


I believe these lines were used to create NUMA nodes for the guest.
Removing them means no NUMA nodes for the guest I guess.


There is one thing I don't quite understand though. In the xml file, I kept


  <numatune>

    <memory mode='strict' nodeset='0-1'/>

  </numatune>

Since no NUMA nodes were created, how come this section can pass? Then what
is the nodeset here?



On Thu, Jun 11, 2015 at 10:27 AM, Clarylin L <clearasu at gmail.com> wrote:

> Hi Michal,
>
> I also tried the other option you mentioned "The other option you have is
> to not use guest NUMA nodes, in which case global -mem-path can be used."
> by removing from xml
>
>   <numatune>
>
>     <memory mode='strict' nodeset='0-1'/>
>
>   </numatune>
>
> while keeping older qemu-1.5.3 which does not support <page> element, it
> still complained no enough memory when starting guest.
>
> Did I miss something?
>
> On Thu, Jun 11, 2015 at 12:00 AM, Michal Privoznik <mprivozn at redhat.com>
> wrote:
>
>> [please keep the list CC'ed]
>>
>> On 10.06.2015 20:09, Clarylin L wrote:
>> > Hi Michal,
>> >
>> > Thanks a lot.
>> >
>> > If 100 hugepages are pre-allocated, the guest can start without
>> decreasing
>> > number of hugepages. Since the guest requires 128 hugepages, it's kind
>> of
>> > expected that the guest would not take memory from hugepages.
>> >
>> > Before guest start,
>> >
>> > [root at local ~]# cat /proc/meminfo | grep uge
>> >
>> > AnonH*uge*Pages:         0 kB
>> >
>> > H*uge*Pages_Total:     100
>> >
>> > H*uge*Pages_Free:      100
>> >
>> > H*uge*Pages_Rsvd:        0
>> >
>> > H*uge*Pages_Surp:        0
>> > H*uge*pagesize:    1048576 kB
>> >
>> > After:
>> >
>> > [root at local ~]# cat /proc/meminfo | grep uge
>> >
>> > AnonH*uge*Pages:  134254592 kB
>> >
>> > H*uge*Pages_Total:     100
>> >
>> > H*uge*Pages_Free:      100
>> >
>> > H*uge*Pages_Rsvd:        0
>> >
>> > H*uge*Pages_Surp:        0
>> > H*uge*pagesize:    1048576 kB
>> >
>> > There is no -mem-prealloc and -mem-path options in qemu command
>>
>> And there can't be. From the command line below, you are defining 2 NUMA
>> nodes for your guest. In order to instruct qemu to back their memory by
>> huge pages you need it to support memory-backend-file object which was
>> introduced in qemu-2.1.0.
>> The other option you have is to not use guest NUMA nodes, in which case
>> global -mem-path can be used.
>>
>> >
>> > [root at local ~]# ps -ef | grep qemu
>> >
>> > *qemu*      3403     1 99 17:42 ?        00:36:42
>> /usr/libexec/*qemu*-kvm
>> > -name qvpc-di-03-sf -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off
>> -cpu
>> > host -m 131072 -realtime mlock=off -smp 32,sockets=2,cores=16,threads=1
>> > -numa node,nodeid=0,cpus=0-15,mem=65536 -numa
>> > node,nodeid=1,cpus=16-31,mem=65536 -uuid
>> > e1b72349-4a0b-4b91-aedc-fd34e92251e4 -smbios type=1,serial=SCALE-SLOT-03
>> > -no-user-config -nodefaults -chardev
>> >
>> socket,id=charmonitor,path=/var/lib/libvirt/*qemu*/qvpc-di-03-sf.monitor,server,nowait
>> > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
>> -no-shutdown
>> > -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
>> -drive
>> >
>> file=/var/lib/libvirt/images/asr5700/qvpc-di-03-sf-hda.img,if=none,id=drive-ide0-0-0,format=qcow2
>> > -device
>> > ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2
>> -drive
>> > if=none,id=drive-ide0-1-0,readonly=on,format=raw -device
>> > ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
>> > -chardev pty,id=charserial0 -device
>> > isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1
>> > -device isa-serial,chardev=charserial1,id=serial1 -vnc 127.0.0.1:0 -vga
>> > cirrus -device i6300esb,id=watchdog0,bus=pci.0,addr=0x3 -watchdog-action
>> > reset -device vfio-pci,host=08:00.0,id=hostdev0,bus=pci.0,addr=0x5
>> -device
>> > vfio-pci,host=09:00.0,id=hostdev1,bus=pci.0,addr=0x6 -device
>> > vfio-pci,host=0a:00.0,id=hostdev2,bus=pci.0,addr=0x7 -device
>> > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on
>> >
>> >
>> > If 140 hugepages are preallocated, the guest cannot start and it
>> complained
>> > not enough memory.
>> >
>> >
>> > The libvirt version is shown as follows:
>> >
>> > virsh # version
>> >
>> > Compiled against library: libvirt 1.2.8
>> >
>> > Using library: libvirt 1.2.8
>> >
>> > Using API: QEMU 1.2.8
>> >
>> > Running hypervisor: QEMU 1.5.3
>> >
>> >
>>
>>
>>
>> > Also in guest configuration contains numa session. The hugepages are
>> > uniformly distributed to two nodes. In this case, do I need to make
>> > additional configurations to enable usage of hugepages?
>> >
>> >   <numatune>
>> >
>> >     <memory mode='strict' nodeset='0-1'/>
>> >
>> >   </numatune>
>> >
>>
>> This says that all the memory for you guests should be pinned onto host
>> nodes 0-1. If you want to be more specific, you can explicitly wire
>> guest NUMA nodes onto host NUMA nodes in 1:N relationship (where N can
>> be even 1, in which case you will get 1:1), e.g.:
>>
>> <memnode cellid='0' mode='preferred' nodeset='0'/>
>>
>> Michal
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20150611/dbdf0387/attachment-0001.htm>


More information about the libvir-list mailing list