[vfio-users] NVidia code 43 with Winddows 10 guest - does NVidia have another trick up their sleeve?

[Alex Williamson]

On Wed, 3 Jan 2018 17:28:39 -0600
Robert Dahlstrom <robert.j.dahlstrom at gmail.com> wrote:

> Hello, I have been trying to setup a Windows 10 guest on my Linux host,
> passing through an nvidia 1060. I've been having a very difficult time,
> getting the dreaded code 43. I haven't followed an individual guide, but
> have taken bits from https://wiki.archlinux.org/ind
> ex.php/PCI_passthrough_via_OVMF and Alex Williamson's when relevant among
> others.
> 
> I have tried just about every possible solution I've found, but have come
> up short.
> 
> One thing I'll point out right away: in my VM, I put two virtual disks. On
> one I installed windows, on the other I installed Debian 9. When booting
> into the linux guest OS, with the proprietary nvidia driver, **everything
> works**. When I boot into windows: code 43. Based on that, I think I have
> most of my stuff correctly setup, and I am just at the receiving end of
> nvidia treachery. I hope I'm wrong about that though.

Nobody else is reporting new treachery.
 
> Since both cards are nvidia cards, I use vfio.conf to prevent the nvidia
> driver from loading on the guest. The guest GPU is in an IOMMU group that
> consists solely of it, its HDMI audio device, and the Host Bridge/PCI
> Bridge devices. Both the video and audio devices are forwarded to the guest.
> 
> What I have tried:
> 
>  1) Seemingly every permutation of hidden kvm, vendor_id settings, hyperv
> settings, both i440fx and q35, BIOS and OVMF,

My suggestion for GeForce remains 440fx + OVMF.

>  2) My current situation has my guest GPU in the first PCIe slot, with my
> host GPU in the third. I think the first 2 slots share an IOMMU group,
> presumably for SLI or some such. This puts my guest GPU as the boot card,
> which is what the system BIOS and bootloader comes up on. I read that may
> cause trouble, and so I've tried swapping the cards.  This I don't believe
> works because there is a boatload of other stuff in the IOMMU group of the
> third slot (Group 0), such as my host SATA and USB controllers, that I
> can't forward.

Look for a BIOS update for your system and/or BIOS settings to enable
ACS, AMD added ACS support in one of the AGESA releases for Ryzen7
systems.  Using the host boot graphics device is very likely
contributing to your problems.  I would at a very minimum swap cards in
order to get a ROM dump from your GPU and use that rather than one from
techpowerup.
 
>  3) Passing disable_vga in vfio.conf, doesn't seem to do anything
> 
>  4) Setting MSI interrupt signaling in windows for both the video and audio
> devices.
> 
> What I have not tried:
> 
>  1) Switching which GPU I use for host and guest. If I can't use the 1060
> the whole thing is pointless. It may tell me if the boot GPU issue is the
> cause of my woe, however.

Yes, it very well could be.

>  2) I have read sometimes booting with the virtual graphics device (Spice +
> QXL) causes problems. Trying that, I have not been able to boot up at all
> without it. Presence of -nographics on qemu commandline seems to make no
> difference. When I blast the display from the config, the qemu process
> starts up and sits at 0% and never does anything, and I can't later remote
> desktop or ssh into the guest.

If you can't get a BIOS/UEFI splash screen out of the assigned
graphics, it's likely not going to work in the OS either.  You're
pretty much guaranteed a Code 43 so long as the emulated graphics are
present.

>  3) I have not yet successfully gone through the process of personally
> downloading the ROM from the card and using it. I have instead grabbed a
> ROM file from techpowerup.com, and verified that it has both BIOS and EFI
> portions. I can passthrough in linux regardless whether I use the ROM or
> not, and in Windows it makes no difference, code 43 for both. When I try to
> download the ROM myself I get the Invalid PCI ROM header signature error,
> and an input/output error.

That's because you're trying to dump the ROM from the boot graphics...

Thanks,
Alex