[libvirt] memory pinning problem
David Weber
wb at munzinger.de
Wed Sep 11 08:47:08 UTC 2013
Am Freitag, 6. September 2013, 12:10:04 schrieb Daniel P. Berrange:
> On Tue, Aug 27, 2013 at 09:09:25AM +0200, David Weber wrote:
> > Hi,
> >
> > we try to use vcpu pinning on a 2 socket server with Intel Xeon E5620
> > cpus, HT enabled and 2*6*16GiB Ram but experience problems if we try to
> > start a guest on the second socket:
> > error: Failed to start domain test
> > error: internal error: process exited while connecting to monitor:
> > kvm_init_vcpu failed: Cannot allocate memory
> >
> > Libvirt version 1.1.1
> > Linux 3.11-rc7
> >
> > Because I coudn't find any other service which allowed a 7M file upload, I
> > put the log file and everything else which could perhabs be relevant into
> > a github repository:
> > https://github.com/David-Weber/vcpu-pinning
> >
> > When we try to start a guest on the first node it runs fine:
> > <vcpu placement='static' cpuset='0-3,8-11'>4</vcpu>
> > <numatune>
> >
> > <memory mode='strict' nodeset='0'/>
> >
> > </numatune>
> >
> > Starting it on the second node fails
> >
> > <vcpu placement='static' cpuset='4-7,12-15'>4</vcpu>
> > <numatune>
> >
> > <memory mode='strict' nodeset='1'/>
> >
> > </numatune>
> >
> > Even more strange, starting it with the CPUs of the second node and the
> > memory>
> > of the first node works:
> > <vcpu placement='static' cpuset='4-7,12-15'>4</vcpu>
> > <numatune>
> >
> > <memory mode='strict' nodeset='0'/>
> >
> > </numatune>
> >
> > The log file contains these three cases.
> >
> > Using the placement='auto' parameter leads to the same problem. If numad
> > return the second node, the guest won't start.
> >
> > Is this a configuration, a libvirt or a cgroup problem? :)
>
> With mode='strict' you are telling QEMU that if it can't allocate
> memory from the requested node, it should fail. Is it possible
> that some of your Numa nodes have insufficient memory free ?
>
> The combination of 'virsh capabilities' output and the results
> of 'virsh freecell NODENUM' for each NUMA node will give an
> indication of the allocation state.
>
>
> Daniel
Thank you for your response.
There should be plenty of free memory be available.
# virsh capabilities
<capabilities>
<host>
<uuid>91ecf0d9-c821-c9ca-c7fa-00259064e5c6</uuid>
<cpu>
<arch>x86_64</arch>
<model>Westmere</model>
<vendor>Intel</vendor>
<topology sockets='1' cores='4' threads='2'/>
<feature name='rdtscp'/>
<feature name='pdpe1gb'/>
<feature name='dca'/>
<feature name='pcid'/>
<feature name='pdcm'/>
<feature name='xtpr'/>
<feature name='tm2'/>
<feature name='est'/>
<feature name='smx'/>
<feature name='vmx'/>
<feature name='ds_cpl'/>
<feature name='monitor'/>
<feature name='dtes64'/>
<feature name='pclmuldq'/>
<feature name='pbe'/>
<feature name='tm'/>
<feature name='ht'/>
<feature name='ss'/>
<feature name='acpi'/>
<feature name='ds'/>
<feature name='vme'/>
</cpu>
<power_management>
<suspend_mem/>
<suspend_disk/>
<suspend_hybrid/>
</power_management>
<migration_features>
<live/>
<uri_transports>
<uri_transport>tcp</uri_transport>
</uri_transports>
</migration_features>
<topology>
<cells num='2'>
<cell id='0'>
<memory unit='KiB'>99003008</memory>
<cpus num='8'>
<cpu id='0' socket_id='0' core_id='0' siblings='0,8'/>
<cpu id='1' socket_id='0' core_id='1' siblings='1,9'/>
<cpu id='2' socket_id='0' core_id='9' siblings='2,10'/>
<cpu id='3' socket_id='0' core_id='10' siblings='3,11'/>
<cpu id='8' socket_id='0' core_id='0' siblings='0,8'/>
<cpu id='9' socket_id='0' core_id='1' siblings='1,9'/>
<cpu id='10' socket_id='0' core_id='9' siblings='2,10'/>
<cpu id='11' socket_id='0' core_id='10' siblings='3,11'/>
</cpus>
</cell>
<cell id='1'>
<memory unit='KiB'>99088544</memory>
<cpus num='8'>
<cpu id='4' socket_id='1' core_id='0' siblings='4,12'/>
<cpu id='5' socket_id='1' core_id='1' siblings='5,13'/>
<cpu id='6' socket_id='1' core_id='9' siblings='6,14'/>
<cpu id='7' socket_id='1' core_id='10' siblings='7,15'/>
<cpu id='12' socket_id='1' core_id='0' siblings='4,12'/>
<cpu id='13' socket_id='1' core_id='1' siblings='5,13'/>
<cpu id='14' socket_id='1' core_id='9' siblings='6,14'/>
<cpu id='15' socket_id='1' core_id='10' siblings='7,15'/>
</cpus>
</cell>
</cells>
</topology>
<secmodel>
<model>none</model>
<doi>0</doi>
</secmodel>
<secmodel>
<model>dac</model>
<doi>0</doi>
</secmodel>
</host>
<guest>
<os_type>hvm</os_type>
<arch name='i686'>
<wordsize>32</wordsize>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<machine canonical='pc-i440fx-1.5' maxCpus='255'>pc</machine>
<machine maxCpus='255'>pc-q35-1.4</machine>
<machine canonical='pc-q35-1.5' maxCpus='255'>q35</machine>
<machine maxCpus='1'>isapc</machine>
<machine maxCpus='255'>pc-0.10</machine>
<machine maxCpus='255'>pc-0.11</machine>
<machine maxCpus='255'>pc-0.12</machine>
<machine maxCpus='255'>pc-0.13</machine>
<machine maxCpus='255'>pc-0.14</machine>
<machine maxCpus='255'>pc-0.15</machine>
<machine maxCpus='255'>pc-1.0</machine>
<machine maxCpus='255'>pc-1.1</machine>
<machine maxCpus='255'>pc-1.2</machine>
<machine maxCpus='255'>pc-1.3</machine>
<machine maxCpus='255'>pc-i440fx-1.4</machine>
<machine maxCpus='1'>none</machine>
<domain type='qemu'>
</domain>
<domain type='kvm'>
<emulator>/usr/bin/qemu-kvm</emulator>
<machine canonical='pc-i440fx-1.5' maxCpus='255'>pc</machine>
<machine maxCpus='255'>pc-q35-1.4</machine>
<machine canonical='pc-q35-1.5' maxCpus='255'>q35</machine>
<machine maxCpus='1'>isapc</machine>
<machine maxCpus='255'>pc-0.10</machine>
<machine maxCpus='255'>pc-0.11</machine>
<machine maxCpus='255'>pc-0.12</machine>
<machine maxCpus='255'>pc-0.13</machine>
<machine maxCpus='255'>pc-0.14</machine>
<machine maxCpus='255'>pc-0.15</machine>
<machine maxCpus='255'>pc-1.0</machine>
<machine maxCpus='255'>pc-1.1</machine>
<machine maxCpus='255'>pc-1.2</machine>
<machine maxCpus='255'>pc-1.3</machine>
<machine maxCpus='255'>pc-i440fx-1.4</machine>
<machine maxCpus='1'>none</machine>
</domain>
</arch>
<features>
<cpuselection/>
<deviceboot/>
<acpi default='on' toggle='yes'/>
<apic default='on' toggle='no'/>
<pae/>
<nonpae/>
</features>
</guest>
<guest>
<os_type>hvm</os_type>
<arch name='x86_64'>
<wordsize>64</wordsize>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<machine canonical='pc-i440fx-1.5' maxCpus='255'>pc</machine>
<machine maxCpus='255'>pc-q35-1.4</machine>
<machine canonical='pc-q35-1.5' maxCpus='255'>q35</machine>
<machine maxCpus='1'>isapc</machine>
<machine maxCpus='255'>pc-0.10</machine>
<machine maxCpus='255'>pc-0.11</machine>
<machine maxCpus='255'>pc-0.12</machine>
<machine maxCpus='255'>pc-0.13</machine>
<machine maxCpus='255'>pc-0.14</machine>
<machine maxCpus='255'>pc-0.15</machine>
<machine maxCpus='255'>pc-1.0</machine>
<machine maxCpus='255'>pc-1.1</machine>
<machine maxCpus='255'>pc-1.2</machine>
<machine maxCpus='255'>pc-1.3</machine>
<machine maxCpus='255'>pc-i440fx-1.4</machine>
<machine maxCpus='1'>none</machine>
<domain type='qemu'>
</domain>
<domain type='kvm'>
<emulator>/usr/bin/qemu-kvm</emulator>
<machine canonical='pc-i440fx-1.5' maxCpus='255'>pc</machine>
<machine maxCpus='255'>pc-q35-1.4</machine>
<machine canonical='pc-q35-1.5' maxCpus='255'>q35</machine>
<machine maxCpus='1'>isapc</machine>
<machine maxCpus='255'>pc-0.10</machine>
<machine maxCpus='255'>pc-0.11</machine>
<machine maxCpus='255'>pc-0.12</machine>
<machine maxCpus='255'>pc-0.13</machine>
<machine maxCpus='255'>pc-0.14</machine>
<machine maxCpus='255'>pc-0.15</machine>
<machine maxCpus='255'>pc-1.0</machine>
<machine maxCpus='255'>pc-1.1</machine>
<machine maxCpus='255'>pc-1.2</machine>
<machine maxCpus='255'>pc-1.3</machine>
<machine maxCpus='255'>pc-i440fx-1.4</machine>
<machine maxCpus='1'>none</machine>
</domain>
</arch>
<features>
<cpuselection/>
<deviceboot/>
<acpi default='on' toggle='yes'/>
<apic default='on' toggle='no'/>
</features>
</guest>
</capabilities>
# virsh freecell 0
0: 86071624 KiB
# virsh freecell 1
1: 75258628 KiB
# virsh edit test
<domain type='kvm'>
<name>test</name>
<uuid>08cdc389-78bf-450c-89f4-b4728edabdbf</uuid>
<memory unit='KiB'>1048576</memory>
<currentMemory unit='KiB'>1048576</currentMemory>
<vcpu placement='static' cpuset='4-7'>1</vcpu>
<numatune>
<memory mode='strict' nodeset='1'/>
</numatune>
<os>
<type arch='x86_64' machine='pc-i440fx-1.5'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
</features>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<emulator>/usr/bin/qemu-kvm</emulator>
<controller type='usb' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<controller type='ide' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x1'/>
</controller>
<input type='mouse' bus='ps2'/>
<graphics type='vnc' port='-1' autoport='yes'/>
<video>
<model type='cirrus' vram='9216' heads='1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/>
</video>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
</memballoon>
</devices>
</domain>
# virsh start test
error: Failed to start domain test
error: internal error: process exited while connecting to monitor:
kvm_init_vcpu failed: Cannot allocate memory
Allocating memory on this node with numactl works fine
# numactl --cpubind=1 --membind=1 -- dd if=/dev/zero of=/dev/null bs=2G
count=1
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB) copied, 0.60816 s, 3.5 GB/s
David
More information about the libvir-list
mailing list