[libvirt] cpuset / numa and qemu in TCG mode

[Martin Kletzander]

On Mon, May 11, 2015 at 10:52:46AM +1000, Tony Breeds wrote:
>
>Hello all,
>    This is with reference to OpenStack (nova) bug 1439280 [1].
>
>The symptom is when nova(-compute) tries to launch an instance/VM it errors out with:
>    "libvirtError: Requested operation is not valid: cpu affinity is not supported"
>This only happens with using qemu in TCG mode.
>

I can see where the error might have come from.  QEMU doesn't have
vcpu threads if ran in TCG mode, so if libvirt tries to pin some of
them, this will happen.  *But* upstream handles this perfectly.  I
just tried and it works.  I'll explain why a few lines down the road.

>After looking at the domain XML and the docs at [2]  It seems to me the problem is either:
>1) the libvirt driver in nova is generating valid XML that is an invalid
>   domain description ; or
>2) NUMA support in qemu (TCG) mode is broken.
>

I don't get what this has to do with NUMA, but anyway I just think
this is a bug in older libvirt actually.

>Exploring these options a little further:
>
>Option 1:  the libvirt driver in nova is generating valid XML that is an
>invalid domain description
>====
>
>This is the relevant section of the domain XML
>---
>  <vcpu placement='static' cpuset='0-1'>1</vcpu>
>  <cputune>
>    <shares>1024</shares>
>  </cputune>
>---
>
>To my reading of [2] the domain XML should either avoid specifying the 'cpuset'
>in the vcpu node ; or include 'emulatorpin' in the cputune node.
>

Yes, that's because we can't differentiate which threads do what with
TCG accel.  Because of the you must only specify one pinning for all
threads (I/O threads, CPU threads and the emulator thread), which is
done exactly using the 'cpuset' attribute as that is valid for the
whole machine.  Thatnsk to that we don't have to differentiate
anything and just use the cpuset for the whole machine -> no error
should occur there.

>Have I understood the documentation correctly?  If so it would seem that the
>correct fix is in nova to teach the libvirt driver to generate the correct XML
>for this virtulisation type.
>

You understood the documentation correctly, it was just libvirt acting
up a bit.

>Option 2: NUMA support in qemu (TCG) mode is broken.
>====
>
>Is cpuset and NUMA architecture expected to work in qemu when running in TCG
>mode?  If it is then clearly we need to dig into this and fix qemu.
>

I'm still not sure why are you bringing NUMA into the mix as there is
nothing NUMA related in the bug or this mail.  But to complete my
answer, even though there is no practical benefit to using NUMA on TCG
accelerated QEMU machine, it should still work.

Here's the XML I just fired up using libvirt 1.2.15 just to try it
out.  And it simply works.

<domain type='qemu'>
  <name>dummy-tcg</name>
  <uuid>946f637c-dcaa-4392-8926-b54476eddd43</uuid>
  <memory unit='KiB'>131072</memory>
  <currentMemory unit='KiB'>65536</currentMemory>
  <vcpu placement='static' cpuset='3-5,7'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-2.3'>hvm</type>
    <boot dev='hd'/>
  </os>
  <cpu>
    <numa>
      <cell id='0' cpus='0,2' memory='65536' unit='KiB'/>
      <cell id='1' cpus='1,3' memory='65536' unit='KiB'/>
    </numa>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
    <memballoon model='none'/>
  </devices>
</domain>

>
>Yours Tony.
>
>[1] https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1439280
>[2] https://libvirt.org/formatdomain.html#elementsCPUAllocation
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20150511/11e6192f/attachment-0001.sig>