[vfio-users] MSI Interrupt balance in guests

Fri Apr 29 14:43:10 UTC 2016

So I’ve made a little more progress here-

Ubuntu 16 - doing the full upgrade with the newest kernel seemed to give me
massive performance increase with multiple different configurations. While
this has fixed a lot of my issues, it raises even further questions about
interrupt management. Switching to the lowlatency kernel also seems to have
 made a big difference- I believe the linux kernel is more able to keep up
with the timer demands of the guest. But at this point I have 0 audio
stutter, and latency jitters at +/- 5ms with a frequency of about 20Hz
(which is better than I’ve seen on some bare metal streaming hosts!)

The other thing I did was force x2apic mode in windows 10:
https://support.microsoft.com/en-us/kb/2303458

So this is where i get a lot of confusion… recent posts are saying to turn
of hyper-v APIC because APICv on newer intel is faster (and AFAIK this is
done through x2apic emulation). KVM does not seem to enable APICv for me,
so I get more stuttering with hyper-v apic off. Now, where the magic really
seems to come in…. forcing windows to use X2APIC AND enabling hyper-v gives
me the best performance. From what I’ve read in the KVM code, qemu should
be giving hints to Windows on what APIC to use. From what i understand,
windows should really choose one or the other, but in my case it seems to
be using both X2APIC and hyper-v APIC. i think hyper-v APIC is just fast
EOI emulation, I can see via kvm_stat that I get paravirtual EOIs with it
on, but not with it off.

The other magic, is that with X2APIC, I magically don’t have to force MSI
interrupts anymore. I get almost identical performance either way (probably
some loss for shared edge interrupts with non-msi). So at this point I’m
really confused how these different types are being
emulated/passed-through. Without x2apic I seem to have no ability to use
edge interrupts, but MSI is fine. With x2APIC it all seems… fine, and with
really low overhead even though linux tells me I dont have APICv enabled.

I posted another thread trying to clear up the difference between the
interrupt types:
https://www.redhat.com/archives/vfio-users/2016-April/msg00219.html

On Thu, Apr 28, 2016 at 10:43 PM Muted Bytes <mutedbytes at gmail.com> wrote:

> It would be great to get information on all the points your brought up.
> Regarding your last few points, specifically
>
> "Also, how does x2APIC relate to MSI and/or VFIO? I’m overriding my BIOS
> opt-out to force x2APIC. Is there any reason to not do this? Does this
> interfere with MSI at all? Why would my bios want to disable a much larger
> APIC? Can qemu/kvm even utilize this?"
>
> what I have found so far, a lot of people recently have been suggesting to
> remove the hyperv vapic enlightenment so that guests can use APICv hardware
> support of more recent Intel CPUs. However, it seems this actually requires
> x2apic support in order for guest to actually use apicv. In my case, my
> motherboard also requires the optout to be overridden for x2apic to be
> enabled. Upon booting with this override option, I have observed a
> significant negative performance impact in the host. Browsers lag, and even
> ramspeed program shows 15% or worse decrease in ram throughput, among
> others.
> On Mar 31, 2016 10:57 AM, "Colin Godsey" <crgodsey at gmail.com> wrote:
>
>> I had some questions about IRQ balancing with MSI interrupts from
>> passthrough devices using VFIO.
>>
>> I have a system that is designed to serve as a gaming streaming server
>> for 2 windows VMs each using PCIe passthrough. So far the recurring problem
>> has been interrupt management. This system ends up being a perfect storm
>> (no pun indented) for interrupts, having to manage: high performance timers
>> for video and audio, high data transfer from the GPU (more than normal
>> because we’re passing 50mbps video stream from the GPU to the system), then
>> 50mbps per VM transfer back to the host via virtio networking. I’m giving
>> full CPU mapping to both VMs, with staggered affinity (VM0 VCPU0=CPU0, VM1
>> VCPU0=CPU2). It is a non-HT intel SMP CPU.
>>
>> MSI so far seems to be the key to getting this all to work right.
>> Enabling MSI was the first big thing, guests not detecting it led to really
>> poor interrupt management, reading more about it explained why.
>>
>> The low latency kernel seems to be the other part of this- games like to
>> run on an accurate clock, and context switching seems minimal when the GPU
>> can drive frame composition like normal, and in the end i think it wastes
>> less CPU cycles due to accurate media response. Even though this system
>> deals with large amounts of throughput, latency and device-responsiveness
>> are still mostly preferred.
>>
>> This leads to the last piece of the puzzle. My mysterious performance
>> drop across the board when both systems are under GPU load. Both MSI GPUs
>> seem to be piling the majority of their interrupts onto the same CPU. The
>> MSI based ethernet adapter seems to fall on another CPU, so that’s great.
>> But the GPU seems to just tank overall guest performance and the GPU
>> interrupts dont seem to want to budge from CPU1 no matter what. There is
>> some distribution of the GPU interrupts across CPUs, but the bulk seems to
>> always land on CPU1.
>>
>> Ultimately what im wondering:
>> * How does MSI balance? Via VFIO, are the MSI interrupts
>> assigned/mapped/masked on a guest level, host level, or hardware level?
>> * Also, how does x2APIC relate to MSI and/or VFIO? I’m overriding my BIOS
>> opt-out to force x2APIC. Is there any reason to not do this? Does this
>> interfere with MSI at all? Why would my bios want to disable a much larger
>> APIC? Can qemu/kvm even utilize this?
>> * Could my performance issues just be related to context-switching and
>> CPU/cache bounds? Would the low latency kernel (1000Mhz) make that worse?
>> If so, is there any way to force MSI affinity?
>>
>> Thanks tons!
>>
>> -Colin G
>>
>> _______________________________________________
>> vfio-users mailing list
>> vfio-users at redhat.com
>> https://www.redhat.com/mailman/listinfo/vfio-users
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160429/c4458470/attachment.htm>