[vfio-users] Debugging IRQ latency

[Michael Slade]

I have been attempting to get passed through sound cards working and 
have been encountering issues.  Apart from the extra complication of 
windows, I've been testing with a real-time audio app like yoshimi, and 
I get audio lag/glitches.  I've been tracking down the causes of these 
glitches and have found several things that contribute.  So far, on two 
setups I have, I've found:

  *  >1ms long SMIs on a GA-990FX motherboard, caused exclusively by
    parallel disk writes (I am running a 5-disk array on it)
  * The linux radeon driver (on the host) occasionally locking up a core
    for 10-35ms at a time (I think this has been fixed since the 4.19
    series)
  * When console messages get sent to a serial port, it locks up a core
    for the duration of the message. (This is a known problem that is
    still being worked on)
  * qemu's emulated (uhci) USB seems to hold up a vCPU for a few ms
    somewhere in the guest's interrupt handler

I've been able to track these down with some combination of

- the SMI (hwlat) detector in ftrace
- the irqsoff tracer in ftrace
- cyclictest
- guesswork

I have found workarounds for the above, however I am still having 
trouble with one box - I get glitches every minute or two. My 
investigation is so far stuck at hits from the irqsoff tracer that 
contain lines like this:

alsa-sin-31611     1d...    0us#: _raw_spin_lock_irq
alsa-sin-31611     1d... 4448us : snd_pcm_group_unlock_irq

So, questions:

- when I get output like the above from an irqsoff trace, how do I probe 
further?

- are there any other tools that could help with debugging these sorts 
of problems?

- I understand that for IRQs to get routed properly from host to guest, 
the device needs to have a per-device (per-function?) IRQ status bit 
that can be used to aid in routing.  How do I confirm that the device in 
question has this bit?

- how do I check that IRQs are being routed to the guest optimally?  
What do I watch out for?  How do I check if it's using irqfd?

- does MSI really help?  Will it save any more more than, say, 100us in 
IRQ response latency?

(I am able to swap in a sound card that works; I have a cheap USB dongle 
that hasn't given me any problems yet.  I still want to see what options 
I have though.)


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20210216/9796ba6c/attachment.htm>