[vfio-users] Double check before I buy another USB card

[Alex Williamson]

On Thu, 2016-01-14 at 22:42 -0800, Frederick N. Brier wrote:
> I think I faux pas'd.  I just bought a 4-port USB PCI card.  I
> determined the Ids and updated the /etc/modprobe.d/local.conf as per
> the blog tutorial.  When I check with "lspci -nnk", the "Kernel
> driver in use" shows "uhci_hcd", not "vfio-pci" as the secondary ATI
> Video card does.  I just saw Alex's post about conventional PCI not
> including a requester ID in bus transactions.  I must have missed
> that in the technology overview.  So even though I only have a single
> PCI slot on my Asus M5A99FX Pro and it is in its own IOMMU group, it
> still is not going to work, correct?  I just wanted to double check
> whether I need to buy a PCI-E card and wait till Monday to get it, or
> if I might just be missing a piece of configuration.  The VM's
> dedicated USB card is going to provide keyboard and mouse via a 4-
> port KVM.  I am trying to have a completely isolated USB hub for the
> VM.  Thank you.

Being conventional PCI doesn't automatically exclude the device from
device assignment, it just often comes with complications.  The device
itself doesn't have a requester ID for the iommu, but the PCIe-to-PCI
bridge adds one.  The side-effect is that the iommu cannot distinguish
between separate devices behind the same bridge, so if you had multiple
devices in the iommu group and tried to use the ACS override patch to
split them apart, this would fail miserably.  You don't have that
situation apparently.

Another issue can be the PCIe-to-PCI bridge itself, some of these just
horrible pieces of hardware, such as the Asmedia bridge on my
motherboard.  Interrupts don't work right across it, on bare metal or
assigned.  If you end up with that kind of bad hardware, those
conventional PCI slots are really just decoration.

Then of course comes the actual endpoint.  It's really hard to find
conventional PCI cards that support MSI, so you're likely stuck with
legacy interrupts.  That's not necessarily a problem, but the preferred
way to do interrupt masking at the device level is with a feature that
was adding in the PCI v2.3 spec.  Lots of conventional PCI cards pre-
date this spec and lots of others were very broken with this method of
interrupt masking.  If that method is not available, then we need to
mask interrupts at the APIC.  This unfortunately requires an exclusive
interrupt for the device, which is not always easy or possible to
achieve.

So, try it and see, but it may be more trouble than it's worth.  Even
if you only need USB1/2, you may see better overall VM performance with
USB3 hardware.  The older OHCI/UHCI/EHCI standards seem to use sub-page 
resources and some of them imply I/O port usage.  These incur higher
overhead because they cannot be directly mapped into the VM address
space.  The throughput for the device may be fine, but it'll take more
CPU power on the host to do it and may result poor overall performance.
Personally I like the TI USB3 cards, more expensive, but not crap.
YMMV.  Thanks,

Alex