[libvirt-users] NUMA issues on virtualized hosts

Thu Sep 20 14:28:01 UTC 2018

Hello,

so final working solution for 8-NUMA node configuration is:

<cpu mode='host-passthrough'><topology sockets='8' cores='4' threads='1'/><numa>
<cell id='0' cpus='0-3' memory='62000000'>
<distances>
<sibling id='0' value='10'/>
<sibling id='1' value='16'/>
<sibling id='2' value='16'/>
<sibling id='3' value='16'/>
<sibling id='4' value='32'/>
<sibling id='5' value='32'/>
<sibling id='6' value='32'/>
<sibling id='7' value='32'/>
</distances>
</cell>
<cell id='1' cpus='4-7' memory='62000000'>
<distances>
<sibling id='0' value='16'/>
<sibling id='1' value='10'/>
<sibling id='2' value='16'/>
<sibling id='3' value='16'/>
<sibling id='4' value='32'/>
<sibling id='5' value='32'/>
<sibling id='6' value='32'/>
<sibling id='7' value='32'/>
</distances>
</cell>
<cell id='2' cpus='8-11' memory='62000000'>
<distances>
<sibling id='0' value='16'/>
<sibling id='1' value='16'/>
<sibling id='2' value='10'/>
<sibling id='3' value='16'/>
<sibling id='4' value='32'/>
<sibling id='5' value='32'/>
<sibling id='6' value='32'/>
<sibling id='7' value='32'/>
</distances>
</cell>
<cell id='3' cpus='12-15' memory='62000000'>
<distances>
<sibling id='0' value='16'/>
<sibling id='1' value='16'/>
<sibling id='2' value='16'/>
<sibling id='3' value='10'/>
<sibling id='4' value='32'/>
<sibling id='5' value='32'/>
<sibling id='6' value='32'/>
<sibling id='7' value='32'/>
</distances>
</cell>
<cell id='4' cpus='16-19' memory='62000000'>
<distances>
<sibling id='0' value='32'/>
<sibling id='1' value='32'/>
<sibling id='2' value='32'/>
<sibling id='3' value='32'/>
<sibling id='4' value='10'/>
<sibling id='5' value='16'/>
<sibling id='6' value='16'/>
<sibling id='7' value='16'/>
</distances>
</cell>
<cell id='5' cpus='20-23' memory='62000000'>
<distances>
<sibling id='0' value='32'/>
<sibling id='1' value='32'/>
<sibling id='2' value='32'/>
<sibling id='3' value='32'/>
<sibling id='4' value='16'/>
<sibling id='5' value='10'/>
<sibling id='6' value='16'/>
<sibling id='7' value='16'/>
</distances>
</cell>
<cell id='6' cpus='24-27' memory='62000000'>
<distances>
<sibling id='0' value='32'/>
<sibling id='1' value='32'/>
<sibling id='2' value='32'/>
<sibling id='3' value='32'/>
<sibling id='4' value='16'/>
<sibling id='5' value='16'/>
<sibling id='6' value='10'/>
<sibling id='7' value='16'/>
</distances>
</cell>
<cell id='7' cpus='28-31' memory='62000000'>
<distances>
<sibling id='0' value='32'/>
<sibling id='1' value='32'/>
<sibling id='2' value='32'/>
<sibling id='3' value='32'/>
<sibling id='4' value='16'/>
<sibling id='5' value='16'/>
<sibling id='6' value='16'/>
<sibling id='7' value='10'/>
</distances>
</cell>
</numa></cpu>
<cputune><vcpupin vcpu='0' cpuset='0' /><vcpupin vcpu='1' cpuset='1' /><vcpupin vcpu='2' cpuset='2' /><vcpupin vcpu='3' cpuset='3' /><vcpupin vcpu='4' cpuset='4' /><vcpupin vcpu='5' cpuset='5' /><vcpupin vcpu='6' cpuset='6' /><vcpupin vcpu='7' cpuset='7' /><vcpupin vcpu='8' cpuset='8' /><vcpupin vcpu='9' cpuset='9' /><vcpupin vcpu='10' cpuset='10' /><vcpupin vcpu='11' cpuset='11' /><vcpupin vcpu='12' cpuset='12' /><vcpupin vcpu='13' cpuset='13' /><vcpupin vcpu='14' cpuset='14' /><vcpupin vcpu='15' cpuset='15' /><vcpupin vcpu='16' cpuset='16' /><vcpupin vcpu='17' cpuset='17' /><vcpupin vcpu='18' cpuset='18' /><vcpupin vcpu='19' cpuset='19' /><vcpupin vcpu='20' cpuset='20' /><vcpupin vcpu='21' cpuset='21' /><vcpupin vcpu='22' cpuset='22' /><vcpupin vcpu='23' cpuset='23' /><vcpupin vcpu='24' cpuset='24' /><vcpupin vcpu='25' cpuset='25' /><vcpupin vcpu='26' cpuset='26' /><vcpupin vcpu='27' cpuset='27' /><vcpupin vcpu='28' cpuset='28' /><vcpupin vcpu='29' cpuset='29' /><vcpupin vcpu='30' cpuset='30' /><vcpupin vcpu='31' cpuset='31' /></cputune>
<numatune>
<memnode cellid='0' mode='strict' nodeset='0'/>
<memnode cellid='1' mode='strict' nodeset='1'/>
<memnode cellid='2' mode='strict' nodeset='2'/>
<memnode cellid='3' mode='strict' nodeset='3'/>
<memnode cellid='4' mode='strict' nodeset='4'/>
<memnode cellid='5' mode='strict' nodeset='5'/>
<memnode cellid='6' mode='strict' nodeset='6'/>
<memnode cellid='7' mode='strict' nodeset='7'/>
</numatune>

With this configuration, virtualized Debian 9 even slightly outperforms the same Debian
9 on the bare metal. 

As of iozone and cache none case. It seems that the problem is with KVM which
stops the iozone running for the first time when not all memory pages are
puppulated or something. Number of pages is not small with 512GB machine.
However, letting kvm to poppulate pages and running the iozone again, does not
bring any performance loss. 

On Tue, Sep 18, 2018 at 09:50:44AM +0200, Michal Privoznik wrote:
> On 09/17/2018 04:59 PM, Lukas Hejtmanek wrote:
> > Hello,
> > 
> > so the current domain configuration:
> 
> 
> Yes, looking good.
> 
> > 
> > Good news is, that spec benchmark looks promising. The first test bwaves
> > finished in 1003 seconds compared to 1700 seconds in the previous wrong case.
> > So far so good.
> 
> Very well, this means that the config above is correct.
> 
> > 
> > Bad news is, that iozone is still the same. There might be some
> > misunderstanding. 
> > 
> > I have to cases:
> > 
> > 1) cache=unsafe. In this case, I can see that hypervizor is prone to swap.
> > Swap a lot. It usually eats whole swap partition and kswapd is running on 100%
> > CPU. swappines, dirty_ration and company do not improve things at all.
> > However, I believe, this is just wrong option for scratch disks where one can
> > expect huge I/O load. Moreover, the hypevizor is poor machine with only low
> > memory left (ok, in my case about 10GB available), so it does not make sense
> > to use that memory for additional cache/disk buffers.
> 
> One thing that just occurred to me - is the qcow2 file fully allocated?
> 
> # qemu-img info /var/lib/libvirt/images/fedora.qcow2
> ..
> virtual size: 20G (21474836480 bytes)
> disk size: 7.0G
> ..
> 
> This is NOT a fully allocated qcow2.
> 
> > 
> > 2) cache=none. In this case, performance is better (only few percent behind
> > baremetal). However, as soon as the size of stored data is about the size of
> > memory of the virtual, writes stops and iozone is eating whole CPU, it looks like
> > it is searching more free pages and it is harder and harder. But not sure,
> > I am not skilled in this area.
> 
> Hmm. Could it be that SSD doesn't have enough free blocks and thus
> writes are throttled? Can you fstrim it and see if that helps?
> 
> > 
> > here, you can clearly see, that it starts writes, doing the writes, then it
> > takes a pause, writes again, and so on, but the pauses are longer and longer..
> > https://pastebin.com/2gfPFgb9
> > The output is until the very end of iozone (I cancelled it by ctrl-c).
> > 
> > It seems that this is not happening on 2-NUMA node with rotational disks only.
> > It is partly happening on 2-NUMA node with 2 NVME SSDs. The partly means, that
> > there are also pauses in writes but it finishes, speed is reduced though. On
> > 1-NUMA node, with the same test, I can see steady writes from the very
> > beginning to the very end at roughly the same speed.
> > 
> > Maybe it could be related to the fact, that NVME is PCI device that is linked
> > to one NUMA node only?
> 
> Can be. I don't know qemu internals that much to know if its capable of
> doing zero copy disk writes.
> 
> > 
> > 
> > As of iothreads, I have only 1 disk (the vde) that is exposed to high i/o
> > load, so I believe more I/O threads is not applicable here. If I understand
> > correctly, I cannot set more iothreads to a single device.. And it does not
> > seem to be iothreads linked as the same scenario in 1-NUMA configuration works
> > OK (I mean that memory penalties can be huge as it does not reflect real NUMA
> > topology, but disk speed it ok anyway.)
> 
> Ah, since it's only one disk then iothreads will not help much here.
> Still worth giving it a shot ;-) Remember, iothreads are for all I/O,
> not disk I/O only.
> 
> Anyway, this is the point where I have to say "I don't know". Sorry. Try
> contacting qemu guys:
> 
> qemu-discuss at nongnu.org
> qemu-devel at nongnu.org
> 
> Michal
> 

-- 
Lukáš Hejtmánek

Linux Administrator only because
  Full Time Multitasking Ninja 
  is not an official job title