[vfio-users] Not enough IRQs in 82093AA IOAPIC

[Zir Blazer]

I have something that could be useful for you, albeit chances are that it is a bit too little too late:
https://habr.com/ru/post/501912/




I think that everything related to PCI Interrupts gets rather chaotic when you get to PCI Express. As PCIe is a superset of PCI that is backwards compatible with it, it is mandatory for the entire PCIe interface to support the old PCI Interrupt subsystem as the baseline lowest common denominator. However, PCIe Slots do not have physical Pins for an out-of-band Interrupt subsystem like PCI did, instead, all Interrupts are passed as in-band messages though the PCIe Lanes, just like MSI on legacy PCI. On pure PCIe Chipsets that don't have built-in PCI support, the whole Interrupt subsystem could be entirely virtual wire with no externally exposed PIRQ Pins at all.
Note that while MSI Interrupts are always better, in many cases they are not enabled by default or are broken somehow. I see often in PCI Passthrough scenarios for Windows with people complaining about some type of audio stuttering that in many cases gets solved by explicitly enabling MSI, as the OS or the Drivers appears to not always do so. So even when using modern OSes legacy PCI Interrupts performance may be important.

What I found highly interesing is that all the PCIe legacy PCI Interrupts virtual wire stuff can be far more versatile than what it initially appears to be, even on actual bare metal. Because there aren't physical lines any more, depending on what the Chipset allows you to do, everything about the PCIe legacy PCI Interrupt allocation can be potentially configured via Software. On platform initialization, the Firmware can configure which PCIe Device uses which legacy PCI Interrupt Pin, which Motherboard Interrupt line (PIR PIRQ) it will be mapped to, and how the PCIe Device ends up getting an IRQ in the IOAPIC. The only limitation seems to be a finite amount of IOAPIC logical IRQs. Now, I understand than on the old PCI system where more IRQs means more physical lines on the Motherboard you couldn't easily scale up the IOAPIC amount of IRQs, but this doesn't seems to be a problem on pure PCIe platforms, as all them are entirely virtual, so I don't know why we're still limited to 24 IRQs. At least on paper, legacy PCI Interrupts could have been optimized if Chipset vendors had wanted to, which is your goal inside a VM.


If anything, what is dissapointing is when you have to deal with actual implementations of all this tech babble. While at the Chipset level the legacy PCI Interrupt mapping could be highly configurable, your regular Motherboard vendor uses propietary Firmware where these things are entirely hardcoded in a non optimum way. As examples, you can see how ASUS mapped the PCIe Slots and Chipset integrated PCIe Devices to PCI Interrupts lines in three different Z170 Motherboards, each on its own way:
https://rog.asus.com/forum/showthread.php?78072-IRQ-assignment-explaint
Notice how bad the spread is, there are several virtual PIRQ lines that are entirely unused, yet you get one with a heavy load. This is yet another case where an open source Firmware like Coreboot could be excellent because it can potentially expose all what the Chipset is able to do and let the end user configure it.

Where taking control of legacy PCI Interrupts could be of major importance is when you're dealing with PCI Passthrough of legacy PCI Devices behind PCIe-to-PCI Bridges or anything that doesn't support Interrupt sharing, like these examples:
https://www.reddit.com/r/VFIO/comments/8j903p/sound_card_interupt_is_in_conflict_with_a_usb/
https://www.reddit.com/r/VFIO/comments/8aj11f/failed_to_setup_intx_fd_operation_not_permitted/

On bare metal, I would consider to multiplex all the Chipset integrated PCIe Devices that can use MSI to the same PIRQ then leave the rest of virtual PIRQs as exclusive for expansion cards, as these seems to be the problematic ones. I'm really annoyed of the current "try in another PCIe Slot or try with another Motherboard" when it should be "change the default PIRQ mapping". I don't like that you have to throw the Hardware because the Firmware sucks. At least you may manage to have fun configuring all this VM side.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20200919/e0a9ba06/attachment.htm>