[vfio-users] GTX 580 Passthrough - Successes and Failures

[vfio]

I have been experiencing the precise symptoms that you describe in this 
thread and your previous thread. I have resolved the issue of the card 
not resetting after running the VM once, but I have not investigated the 
I/O issues yet. Have you since identified the issue with I/O?

I thought I should reply since I have very similar hardware to you, but 
with some differences. I've discovered that some of the GPU issues are 
caused by a feature of our motherboards.

My current setup:
Processor: i7-4960X
Graphics:
	GeForce GTX 580 (EVGA GeForce GTX 580 FTW Hydro Copper 2)
	GeForce 210 (ASUS GeForce 210 SILENT)
	GeForce GTX TITAN X (EVGA GeForce GTX TITAN X HYDRO COPPER)
RAM: 4x Corsair Vengeance Black, total 32 GB, set to 1333 MHz
Monitors:
	Monitor [1] hooked up to 580 DVI 1, and TITAN DVI
	Monitor [2] hooked up to 580 DVI 2
	Monitor [3] hooked up to 210 DVI
Motherboard: ASUS Rampage IV Extreme
Host OS: Debian (sid)
Guest OS: Windows 10 Education 64 Bit
Kernel: 4.4.0-1-amd64
Host Graphics Driver: nouveau 1.0.12

Many of my problems with VGA passthrough have been caused by the ASUS 
Rampage IV Extreme motherboard. The board has the following PCI slots:
	PCIE_X16_1
	PCIE_X8_2A
	PCIE_X8_2B
	PCIE_X16/8_3
	PCIEX1_1
	PCIE_X8_4

One year ago, I had two GTX 580 cards---one in PCIE_X16_1 and another in 
PXIE_X16/8_3. I put the 210 in PCIE_X8_4 and set it as the primary 
adapter, with the intention of passing both 580s in SLi to the VM. 
Unfortunately, this board CANNOT perform VGA passthrough with any card 
placed in PCIE_X16_1 (we established this as part of the discussion in 
the Arch forum). Since the motherboard does not have integrated 
graphics, it always performs some initialization on the card in the 
first PCIE slot; even pci-stub will not prevent the first card from 
being "claimed". To address this problem, I bought the TITAN X and 
discarded one of the 580s (having all three in at once was not 
physically possible).

In my current setup, I have the 580 in PCIE_X16_1, the 210 in 
PCIE_X8_2A, the TITAN in PCIE_X16/8_3, and a Blackmagic Intensity Pro 
(capture card) in PCIE_X8_4. PCIE_X8_2B and PCIEX1_1 are empty.

I mention all of these details because they are relevant to my problem 
with the TITAN not being reset after launching the VM. The Rampage IV 
Extreme includes a very handy feature: DIP switches that allow the PCI 
lanes to be turned on and off. I was using these switches to quickly 
"unplug" graphics cards while I was testing various configurations; 
since my system uses open-loop liquid cooling, it is impractical to 
frequently move the cards between slots.

It turns out that VGA passthrough does not work well with the Rampage IV 
Extreme's PCI lane switches.

When I left all of the lanes turned on except for PCIE_X8_4 (I haven't 
used the Blackmagic card for some time, so I just leave it off), I 
experienced similar issues: "Invalid ROM contents" messages, errors 
about not bringing the card out of the D3 power state, and a VM that 
only worked a fraction of the time. Additionally, my TITAN card would 
appear to switch between bus 03:00 and bus 04:00, even within the same 
session if I "removed" the device and then ran /sys/bus/pci/rescan. 
These issues stopped as soon as I turned all of the PCI lane DIP 
switches to the "on" position---both with the Blackmagic card plugged in 
and with it physically removed. The TITAN stopped switching between 
buses, and the VGA passthrough started working 100% of the time, even 
without reboots. "systemctl suspend" never resolved the issue for me.

Hopefully this helps other people who may have had strange problems with 
this motherboard. I am planning to reinstall the guest shortly, but I 
HAVE experienced the same I/O issues as you have (i.e., the Windows 
installer freezing on multiple "expanding files" percentages, and 
freezes when performing Windows update). However, I have always been 
assigning a physical HDD to the guest using "-drive 
file=/dev/sdf,format=raw". I would be interested to hear if you identify 
the cause of this problem.