[vfio-users] Not enough IRQs in 82093AA IOAPIC

[Zir Blazer]

Note that the IOAPIC being used depends on the Chipset. 440FX may use an optional 82093AA IOAPIC, which is what is in the PDF that you linked earlier (It used to be a standalone chip and usually only found on Dual Processor Motherboards. IOAPIC was extremely rare on consumer 440FX Single Processor Motherboards. QEMU uses it by default because otherwise you wouldn't have working APIC, thus no SMP/SMT), whereas Q35 includes an IOAPIC integrated in the Southbridge component. The Q35 Chipset IOAPIC is also more advanced and provides a total of 24 usable IRQs (The 82093AA has a total of 24, but not all are usable), check Slide 15-17 here: https://www.linux-kvm.org/images/0/06/2012-forum-Q35.pdf
So, if you're complaining about the 440FX IOAPIC but didn't tested the Q35 one already, give it a try.


I agree than it isn't a good idea to arbitrarily increase the amount of IRQs supported by the current IOAPICs, as you may be breaking something else. I mean, these IOAPICs are supposed to identify themselves as preexisting chips with known functions, so they have to mimic the original chips capabilities so that Software that are intended to interact with them like Drivers behaves the same in the emulated ones. If you change the emulated IOAPICs capabilities, they are not a reasonable emulation of the real chips anymore. Same applies to multiple IOAPICs. Actually, do the 82093AA and Q35 Southbridge even supported such combination? I know that the APIC specification allows for multiple IOAPICs, but I doubt these two were intended for that.
How would the OS and Chipset Drivers react to such changes? I know that virtualization has allowed to try physically imposible combinations like plugging real passthroughed PCIe Devices directly onto an emulated PCI Host Bridge and other weird combinations and for the most part they worked, so maybe it is less worse than appears.





If anything, I'm interesed on whenever you can create from scratch an emulated function that has no physical Hardware counterpart. It was done some years ago with the Super Nintendo and the MSU-1 enchantment chip, that was born in an emulator: http://nerdlypleasures.blogspot.com/2018/01/sd2snes-and-msu-1.html
I would believe that a custom, virtual platform implementation may be better in the long run than trying to match the capabilities of existing Hardware like with the current emulated Chipsets. At the end of the day, emulating Hardware puts you hard limits of what you should be able to do.

APIC seems to support some form of detection of one or multiple IOAPICs and how many IRQs each of them manages via the ACPI MADT Table: https://wiki.osdev.org/IOAPIC#Detecting_I.2FO_APIC
It may be possible than OS side it uses that to autodetect changes and act accordingly, so you may throw it something new and get it to work. But I don't know whenever you can replace with a brand new IOAPIC the one of the existing Chipsets, or if there are any assumptions by the Firmware/OS/Drivers side besides the ACPI Table one.

About old PCI, each PCI Slot has 4 Pins for 4 Interrupt lines (INTA#, INTB#, INTC#, INTD#). A PCI Device (Or Function?) may only use a single Interrupt, and a PCI Card with a PCI Device that uses Interrupts always begins using the INTA# line, the other 3 are used only if there are more Devices that uses Interrupts. Motherboard side, there are usually 4 lines (LNKA, LNKB, LNKC, LNKD) that are shared between multiple PCI Slots, usually with a zigzagging pattern, which coupled with cards always using INTA# first gives the layout an inherent load balancing. For example, LNKA goes to INTA# in first slot, then INTD# on second, INTC# on third and INTB# on fourth, and would repeat INTA# in the fifth slot, LNKB goes to INTB# in first slot, INTA# in second slot, and so on.
These 4 Motherboard lines end up wired to a PIR (Programmable Interrupt Router). The PIR is wired to the IOAPIC (Actually, I think that on every case, the PIR is part of the IOAPIC chip itself), and maps each of the Motherboard lines to an IOAPIC IRQ. However, a PIR may have more than 4 inputs, usually 8 (LNKE, LNKF, LNKG, LNKH). You could have an arrangement where if you have 8 PCI Slots, you could have 4 Motherboard lines (LNKA-D) take care of the total 16 Interrupt Pins in the first set of 4 slots, then let the other 4 lines (LNKE-H) take care of all the 16 pins from the other 4 slots. So, contrary to what AW said, you CAN have more than 4 PCI Interrupts.

Fore reference, here is a log from a Dell PowerEdge 1655MC that has a massive 16 Motherboard PCI Interrupt lines (From LNKA to LNKP) using a ServerWorks ServerSet LE Chipset that has 2 IOAPICs (Not sure if it means that they're two chips or just logically separated): http://old-list-archives.xenproject.org/archives/html/xen-users/2006-01/msg00185.html
And here is an ASUS PR-DLS with the ServerWorks Grand Champion LE Chipset that has a total of 3 IOAPICs and a truckload 32 lines (LNKA to LNK5): https://sourceforge.net/p/acpi/mailman/message/12179397/


I don't recall anything that said that if you had a wide enough PIR/IOAPIC, you couldn't get all the 4 Interrupt lines from a single PCI Slot mapped to a dedicated IRQ each (Heck, you could even eliminate the PIR if going that way). It isn't done on physical Hardware because the costs would be ridiculous (Multiply those 4 lines by the amount of Slots, plus two or three IOAPICs to wire them all, and Motherboard complexity), which is the reason why PCI has some form of Interrupt load balancing with the zigzagging pattern of the Interrupt lines between Slots, and the lines are connected to a PIR instead of the IOAPIC directly, so that you can both balance Interrupt load and map a Device to a specific IRQ. But since we are talking about virtualization, I don't see why not go nuts and see if it works. At least on W2000/XP, it should slighty decrease latency and increase performance if you don't have to multiplex several Devices onto the same IRQ.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20200901/6451abc7/attachment.htm>