[vfio-users] Brutal DPC Latency - how is yours? check it please and report back

[Rokas Kupstys]

Yes currently i am actually booted with vanilla archlinux kernel, no
NO_HZ and other stuff.
> Why does 2 core for the host is unacceptable? You plan to use it
> making hard workloads while gaming?
Problem with isolcpus is that it exempts cores from linux cpu scheduler.
This means even if VM is offline they will stand idle. While i dont do
anything on host while gaming i do plenty when not gaming and just
throwing away 6 cores of already disadvantaged AMD cpu is a real waste.

> This config is not good actually.
Well.. It indeed looks bad on paper, however it is the only one that
yields bearable DPC latency. I tried what you mentioned, various
combinations. Pinning 0,2,4,6 cores to vm, 1,3 to emulator, 5,7 for io /
1,3,5,7 cores to vm, 0,2 to emulator, 4,6 for io / 0,1,2,3 cores to vm,
4,5 to emulator, 6,7 for io / 4,5,6,7 cores to vm, 0,1 to emulator, 2,3
for io. All of them yield terrible latency.

Would be interesting to hear someone who has AMD build, how (if) he
solved this.

On 2016.02.29 11:10, Bronek Kozicki wrote:
> Two things you can improve, IMO
>
> * disable NO_HZ
>
> * use isolcpus to dedicate your pinned CPUs to guest only - this
> will also ensure they are not used for guest  IO.
>
> B.
>
> On 29/02/2016 08:45, Rokas Kupstys wrote:
>
>
>>
>>
>> Yesterday i figured out my latency problem. All things listed
>> everywhere on internet failed. Last thing i tried was pinning one
>> vcpu to two physical cores and it brought latency down. Now i have
>> FX-8350 CPU which has shared FPU for each two cores so maybe thats
>> why. With just this pinning latency now is most of the time just
>> above 1000μs. However under load latency increases. I threw out
>> iothreads and emulator pinning and it did not affect much.
>> Superior latency could be achieved using isolcpus=2-7, however
>> leaving just two cores to host is unacceptable. With that setting
>> latency was around 500μs without load. Good part is that
>> Battlefield3 no longer lags, although i observed increased loading
>> times on textures compared to bare metal. Not so good part is that
>> there still is minor sound skipping/cracking since latency is
>> spiking up under load. That is very disappointing. I also tried
>> performance with two VM cores pinned to 4 host cores - bf3 lagged
>> enough to be unplayable. 3 vm cores pinned to 6 host cores was
>> already playable but sound was still cracking. I noticed little
>> difference between that and 4 vm cores pinned to 8 host cores. Be
>> nice if sound could be cleaned up. If anyone have any ideas im all
>> ears. Libvirt xml i use now:
>>
>>
>>
>>>   <vcpu
>>> placement='static'>4</vcpu>
>>>
>>>   <cputune>
>>>
>>>     <vcpupin vcpu='0' cpuset='0-1'/>
>>>
>>>     <vcpupin vcpu='1' cpuset='2-3'/>
>>>
>>>     <vcpupin vcpu='2' cpuset='4-5'/>
>>>
>>>     <vcpupin vcpu='3' cpuset='6-7'/>
>>>
>>>   </cputune>
>>>
>>>   <features>
>>>
>>>     <acpi/>
>>>
>>>     <apic/>
>>>
>>>     <pae/>
>>>
>>>     <hap/>
>>>
>>>     <viridian/>
>>>
>>>     <hyperv>
>>>
>>>       <relaxed state='on'/>
>>>
>>>       <vapic state='on'/>
>>>
>>>       <spinlocks state='on' retries='8191'/>
>>>
>>>     </hyperv>
>>>
>>>     <kvm>
>>>
>>>       <hidden state='on'/>
>>>
>>>     </kvm>
>>>
>>>     <pvspinlock state='on'/>
>>>
>>>   </features>
>>>
>>>   <cpu mode='host-passthrough'>
>>>
>>>     <topology sockets='1' cores='4' threads='1'/>
>>>
>>>   </cpu>
>>>
>>>   <clock offset='utc'>
>>>
>>>     <timer name='rtc' tickpolicy='catchup'/>
>>>
>>>     <timer name='pit' tickpolicy='delay'/>
>>>
>>>     <timer name='hpet' present='no'/>
>>>
>>>     <timer name='hypervclock' present='yes'/>
>>>
>>>   </clock>
>>>
>>>
>>>
>>
>> Kernel configs
>>
>>> CONFIG_NO_HZ_FULL=y
>>>
>>> CONFIG_RCU_NOCB_CPU_ALL=y
>>>
>>> CONFIG_HZ_1000=y
>>>
>>> CONFIG_HZ=1000
>>
>> I am not convinced 1000 hz tickrate is needed. Default one (300)
>> seems to perform equally as well from looking at latency charts.
>> Did not get chance to test it with bf3 yet however.
>>
>>
>>
>>
>>
>> On 2016.01.12 11:12, thibaut noah
>> wrote:
>>
>>
>>
>>
>
>
>
> [cut]
>
>
>
>
>
>
>
>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160229/707211a5/attachment.htm>