[vfio-users] Linux iommu issue with peer-to-peer dma transfers between NVidia GTX 1080s

[Zir Blazer]

I forgot to add this Thread, it has far more info regarding the nVidia cards P2P issue when ACS is enabled:


https://devtalk.nvidia.com/default/topic/883054/cuda-programming-and-performance/multi-gpu-peer-to-peer-access-failing-on-tesla-k80-/1

<https://devtalk.nvidia.com/default/topic/883054/cuda-programming-and-performance/multi-gpu-peer-to-peer-access-failing-on-tesla-k80-/1>


I'm extremely curious on how you are using dual cards on an ASRock X99-E-ITX/ac since it has only a single PCIe Slot. Are you using a riser card or something, or the model of your Motherboard is wrong?

Regardless, use lspci -tvnn to see the PCI Topology of your Motherboard, and use setpci -s bb:dd.f f2a.w=0000 (Where bb:dd.f should be the PCI Address of a PCI Bridge) to disable one by one the ACS across all the PCI Bridges involved from the Root Port and below it (In case that you have a Switch, besides the Root Port, it should have at least an upstream and downstream Bridge), then try again. setpci changes are temporal for the current session, so you need to re do them on reboot.

There are people that reports mixed success, but they are tackling the PLX Switches blindly and seems unaware of the PCI Topology as a whole, else, they would actually bother to check with lspci -t what is the tree section of the PCI Topology that is related to that card. The card could be plugged to a slot that goes to the Root Port directly, or the slot could go to a Switch in the Motherboard, or the card itself may have a switch (Dual GPU), or both. I'm pretty sure than that can allow you to both enable the IOMMU then wreck isolation in the name of PCIe P2P.


BTW, I think that the guy didn't saw my previous mail since I'm mailing the mailing list and not him, if he isn't suscribed and didn't look at the archives, he is missing this. Can someone forward this and the previous mails to him? Leave that task to a single guy or you will spam him, heh.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20170925/0ced6444/attachment.htm>