[vfio-users] GTX 580 Passthrough - Successes and Failures

Tue Feb 23 21:21:42 UTC 2016

I wasn't even aware of those PCI switches. I'll look into it. To be clear,
if I'm only passing through the card in the second slot, I shouldn't have
any issues? What do those switches look like and where are they?

I fixed my IO for the most part by switching over to a raw partition on my
SSD and installing the redhat virtio drivers(if=virtio). I also set
cache=none which removes some extra IO that you shouldn't need when passing
through a drive. The freezing issue was on the latest virtio driver, when I
switched over to stable my installs stopped pausing so I suspect there was
something off with the latest driver.

I also tried to pass the Asmedia SATA controller directly to the VM to get
native performance but apparently it doesn't support boot drives at all.
Anyway, once I installed the virtio drivers I got fairly close to the
expected speeds on the drive, although my random read is still sitting in
the 20s. I've since resolved the Windows Update issue by letting it run for
around 40 hours straight, it's just extremely bloated but if you wait long
enough it will eventually install the updates. I don't know why you'd have
that issue on 10 though, I've only heard of it on 7.

Thanks for the advice!
On Feb 23, 2016 4:01 PM, "vfio" <vfio at taintedbit.com> wrote:

> I have been experiencing the precise symptoms that you describe in this
> thread and your previous thread. I have resolved the issue of the card not
> resetting after running the VM once, but I have not investigated the I/O
> issues yet. Have you since identified the issue with I/O?
>
> I thought I should reply since I have very similar hardware to you, but
> with some differences. I've discovered that some of the GPU issues are
> caused by a feature of our motherboards.
>
> My current setup:
> Processor: i7-4960X
> Graphics:
>         GeForce GTX 580 (EVGA GeForce GTX 580 FTW Hydro Copper 2)
>         GeForce 210 (ASUS GeForce 210 SILENT)
>         GeForce GTX TITAN X (EVGA GeForce GTX TITAN X HYDRO COPPER)
> RAM: 4x Corsair Vengeance Black, total 32 GB, set to 1333 MHz
> Monitors:
>         Monitor [1] hooked up to 580 DVI 1, and TITAN DVI
>         Monitor [2] hooked up to 580 DVI 2
>         Monitor [3] hooked up to 210 DVI
> Motherboard: ASUS Rampage IV Extreme
> Host OS: Debian (sid)
> Guest OS: Windows 10 Education 64 Bit
> Kernel: 4.4.0-1-amd64
> Host Graphics Driver: nouveau 1.0.12
>
> Many of my problems with VGA passthrough have been caused by the ASUS
> Rampage IV Extreme motherboard. The board has the following PCI slots:
>         PCIE_X16_1
>         PCIE_X8_2A
>         PCIE_X8_2B
>         PCIE_X16/8_3
>         PCIEX1_1
>         PCIE_X8_4
>
> One year ago, I had two GTX 580 cards---one in PCIE_X16_1 and another in
> PXIE_X16/8_3. I put the 210 in PCIE_X8_4 and set it as the primary adapter,
> with the intention of passing both 580s in SLi to the VM. Unfortunately,
> this board CANNOT perform VGA passthrough with any card placed in
> PCIE_X16_1 (we established this as part of the discussion in the Arch
> forum). Since the motherboard does not have integrated graphics, it always
> performs some initialization on the card in the first PCIE slot; even
> pci-stub will not prevent the first card from being "claimed". To address
> this problem, I bought the TITAN X and discarded one of the 580s (having
> all three in at once was not physically possible).
>
> In my current setup, I have the 580 in PCIE_X16_1, the 210 in PCIE_X8_2A,
> the TITAN in PCIE_X16/8_3, and a Blackmagic Intensity Pro (capture card) in
> PCIE_X8_4. PCIE_X8_2B and PCIEX1_1 are empty.
>
> I mention all of these details because they are relevant to my problem
> with the TITAN not being reset after launching the VM. The Rampage IV
> Extreme includes a very handy feature: DIP switches that allow the PCI
> lanes to be turned on and off. I was using these switches to quickly
> "unplug" graphics cards while I was testing various configurations; since
> my system uses open-loop liquid cooling, it is impractical to frequently
> move the cards between slots.
>
> It turns out that VGA passthrough does not work well with the Rampage IV
> Extreme's PCI lane switches.
>
> When I left all of the lanes turned on except for PCIE_X8_4 (I haven't
> used the Blackmagic card for some time, so I just leave it off), I
> experienced similar issues: "Invalid ROM contents" messages, errors about
> not bringing the card out of the D3 power state, and a VM that only worked
> a fraction of the time. Additionally, my TITAN card would appear to switch
> between bus 03:00 and bus 04:00, even within the same session if I
> "removed" the device and then ran /sys/bus/pci/rescan. These issues stopped
> as soon as I turned all of the PCI lane DIP switches to the "on"
> position---both with the Blackmagic card plugged in and with it physically
> removed. The TITAN stopped switching between buses, and the VGA passthrough
> started working 100% of the time, even without reboots. "systemctl suspend"
> never resolved the issue for me.
>
> Hopefully this helps other people who may have had strange problems with
> this motherboard. I am planning to reinstall the guest shortly, but I HAVE
> experienced the same I/O issues as you have (i.e., the Windows installer
> freezing on multiple "expanding files" percentages, and freezes when
> performing Windows update). However, I have always been assigning a
> physical HDD to the guest using "-drive file=/dev/sdf,format=raw". I would
> be interested to hear if you identify the cause of this problem.
>
> _______________________________________________
> vfio-users mailing list
> vfio-users at redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160223/d637eb2a/attachment.htm>