[vfio-users] ACS capabitities for 1366 sockets

[Alex Williamson]

That's an interesting combination.  This system is old enough that you have
something more like a traditional North Bridge, South Bridge setup.  Intel
moved the memory controller to the processor in Nehalem & Westmere, but
your high speed Root Ports are on the IOH, part of chipset, not the
processor.  These are root ports 00:01.0, 00:02.0, 00:03.0, and 00:07.0.
These root ports do have ACS support, so things below them are isolated
from one another.  Rather than a PCH, you have an IOH.  Those root ports
are 00:1c.0, 00:1c.2, and 00:1c.4.  Those do not support ACS, do not have
quirks, and are part of a multifunction device, so they and everything
behind them is grouped together, that's your IOMMU group 9.

Now, back to your IOH paths, devices 15:00.*  are grouped together because
they lack ACS between the functions (I believe that Marvell SATA controller
is one of the ones that uses the wrong requester ID for DMA too).  14:00.0
is in it's own group.  Then come the PCIe switches.  The downstream ports
on the switches do not support ACS, so everything downstream of them will
be grouped together.  This is why group 13 contains not only the GTX 770,
but also the downstream port at 10:00.0.  That's fine for a GPU assignment
use case, but would not work well for things like multi-port NICs or SR-IOV
devices where you want to assign each function to a different VM.  The only
other interesting one is 08:00.0, where you've got a second layer of PCIe
switches.  Everything in that hierarchy is already grouped together due to
the lack of ACS at 06:01.0, making group 21 pretty large.  (I think that
other Marvell SATA controller also has DMA issues, but it should already
have quirks in the kernel)

So it looks ok to me.  The isolation isn't perfect, but it's ok for GPU
assignment.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160107/24aa150a/attachment.htm>