<div dir="ltr"><div>Hello, I have been trying to setup a Windows 10 guest on my Linux host, passing through an nvidia 1060. I've been having a very difficult time, getting the dreaded code 43. I haven't followed an individual guide, but have taken bits from <a href="https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF" target="_blank">https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF</a> and Alex Williamson's when relevant among others. </div><div>I have tried just about every possible solution I've found, but have come up short. </div><div> </div><div>One thing I'll point out right away: in my VM, I put two virtual disks. On one I installed windows, on the other I installed Debian 9. When booting into the linux guest OS, with the proprietary nvidia driver, **everything works**. When I boot into windows: code 43. Based on that, I think I have most of my stuff correctly setup, and I am just at the receiving end of nvidia treachery. I hope I'm wrong about that though. Since both cards are nvidia cards, I use vfio.conf to prevent the nvidia driver from loading on the guest. The guest GPU is in an IOMMU group that consists solely of it, its HDMI audio device, and the Host Bridge/PCI Bridge devices. Both the video and audio devices are forwarded to the guest. </div><div>What I have tried: </div><div> 1) Seemingly every permutation of hidden kvm, vendor_id settings, hyperv settings, both i440fx and q35, BIOS and OVMF, </div><div> 2) My current situation has my guest GPU in the first PCIe slot, with my host GPU in the third. I think the first 2 slots share an IOMMU group, presumably for SLI or some such. This puts my guest GPU as the boot card, which is what the system BIOS and bootloader comes up on. I read that may cause trouble, and so I've tried swapping the cards. This I don't believe works because there is a boatload of other stuff in the IOMMU group of the third slot (Group 0), such as my host SATA and USB controllers, that I can't forward. 3) Passing disable_vga in vfio.conf, doesn't seem to do anything </div><div> </div><div> 4) Setting MSI interrupt signaling in windows for both the video and audio devices. </div><div> What I have not tried: </div><div> 1) Switching which GPU I use for host and guest. If I can't use the 1060 the whole thing is pointless. It may tell me if the boot GPU issue is the cause of my woe, however. </div><div> 2) I have read sometimes booting with the virtual graphics device (Spice + QXL) causes problems. Trying that, I have not been able to boot up at all without it. Presence of -nographics on qemu commandline seems to make no difference. When I blast the display from the config, the qemu process starts up and sits at 0% and never does anything, and I can't later remote desktop or ssh into the guest. </div><div> 3) I have not yet successfully gone through the process of personally downloading the ROM from the card and using it. I have instead grabbed a ROM file from <a href="http://techpowerup.com" target="_blank">techpowerup.com</a>, and verified that it has both BIOS and EFI portions. I can passthrough in linux regardless whether I use the ROM or not, and in Windows it makes no difference, code 43 for both. When I try to download the ROM myself I get the Invalid PCI ROM header signature error, and an input/output error. </div><div> 4) Giving up and buying an AMD card. </div><div>Any ideas what I'm missing? The oldest drivers from NVidia are from march 2017, most info I can find on this subject is from 2016 or earlier. Do the later drivers have additional checks that qemu is not able to fool at this time? Thanks for any help, and if you need any additional information I'll be happy to provide it. </div><div>System information: </div><div> * Linux 4.9.0-3-amd64 (Debian 9) * qemu-x86_64 version 2.8.1(Debian 1:2.8+dfsg-6+deb9u3) <div> * virsh -version: 3.0.0 </div><div> * Processor: AMD Ryzen 7 1700X </div><div> * Host GPU: NVidia 740, evga </div><div> * Guest GPU: Nvidia 1060, Asus </div> * Guest NVidia Drivers: 378.78 - the oldest I can get off the website. /etc/modprobe.d/vfio.conf : options vfio-pci ids=10de:1c02,10de:10f1 disable_vga=1 </div><div>lspci -nnv: (This is after vfio is setup, vfio-pci drivers are in use as expected) .... 21:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107 [GeForce GT 740] [10de:0fc8] (rev a1) (prog-if 00 [VGA controller]) Subsystem: eVga.com. Corp. GK107 [GeForce GT 740] [3842:3747] Flags: bus master, fast devsel, latency 0, IRQ 62 Memory at f4000000 (32-bit, non-prefetchable) [size=16M] Memory at e0000000 (64-bit, prefetchable) [size=256M] Memory at f0000000 (64-bit, prefetchable) [size=32M] I/O ports at d000 [size=128] [virtual] Expansion ROM at f5000000 [disabled] [size=512K] Capabilities: <access denied> Kernel driver in use: nvidia Kernel modules: nvidia 21:00.1 Audio device [0403]: NVIDIA Corporation GK107 HDMI Audio Controller [10de:0e1b] (rev a1) Subsystem: eVga.com. Corp. GK107 HDMI Audio Controller [3842:3747] Flags: bus master, fast devsel, latency 0, IRQ 43 Memory at f5080000 (32-bit, non-prefetchable) [size=16K] Capabilities: <access denied> Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel .... 28:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 3GB] [10de:1c02] (rev a1) (prog-if 00 [VGA controller]) Subsystem: ASUSTeK Computer Inc. GP106 [GeForce GTX 1060 3GB] [1043:85b1] Flags: bus master, fast devsel, latency 0, IRQ 63 Memory at f6000000 (32-bit, non-prefetchable) [size=16M] Memory at c0000000 (64-bit, prefetchable) [size=256M] Memory at d0000000 (64-bit, prefetchable) [size=32M] I/O ports at e000 [size=128] Expansion ROM at 000c0000 [disabled] [size=128K] Capabilities: <access denied> Kernel driver in use: vfio-pci Kernel modules: nvidia 28:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f1] (rev a1) Subsystem: ASUSTeK Computer Inc. Device [1043:85b1] Flags: bus master, fast devsel, latency 0, IRQ 65 Memory at f7080000 (32-bit, non-prefetchable) [size=16K] Capabilities: <access denied> Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel ....</div><div> </div><div>IOMMU groups: IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452] IOMMU Group 0 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453] IOMMU Group 0 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453] IOMMU Group 0 01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd Device [144d:a804] IOMMU Group 0 03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b9] (rev 02) IOMMU Group 0 03:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b5] (rev 02) IOMMU Group 0 03:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b0] (rev 02) IOMMU Group 0 1d:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02) IOMMU Group 0 1d:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02) IOMMU Group 0 1d:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02) IOMMU Group 0 1d:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02) IOMMU Group 0 21:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107 [GeForce GT 740] [10de:0fc8] (rev a1) IOMMU Group 0 21:00.1 Audio device [0403]: NVIDIA Corporation GK107 HDMI Audio Controller [10de:0e1b] (rev a1) IOMMU Group 0 25:00.0 USB controller [0c03]: ASMedia Technology Inc. Device [1b21:1343] IOMMU Group 0 26:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03) IOMMU Group 1 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452] IOMMU Group 2 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452] IOMMU Group 2 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453] IOMMU Group 2 28:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 3GB] [10de:1c02] (rev a1) IOMMU Group 2 28:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f1] (rev a1) IOMMU Group 3 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452] IOMMU Group 4 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452] IOMMU Group 4 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454] IOMMU Group 4 29:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:145a] IOMMU Group 4 29:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Device [1022:1456] IOMMU Group 4 29:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:145c] IOMMU Group 5 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452] IOMMU Group 5 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454] IOMMU Group 5 2a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:1455] IOMMU Group 5 2a:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51) IOMMU Group 5 2a:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Device [1022:1457] IOMMU Group 6 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59) IOMMU Group 6 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51) IOMMU Group 7 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1460] IOMMU Group 7 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1461] IOMMU Group 7 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1462] IOMMU Group 7 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1463] IOMMU Group 7 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1464] IOMMU Group 7 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1465] IOMMU Group 7 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1466] IOMMU Group 7 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1467] ROM I got from techpowerup: rom-parser /opt/Asus.GTX1060.3072.161104.rom Valid ROM signature found @a00h, PCIR offset 1a0h PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 1c02, class: 030000 PCIR: revision 0, vendor revision: 1 Valid ROM signature found @fa00h, PCIR offset 1ch PCIR: type 3 (EFI), vendor: 10de, device: 1c02, class: 030000 PCIR: revision 3, vendor revision: 0 EFI: Signature Valid, Subsystem: Boot, Machine: X64 Last image </div><div>libvirt config: <domain type='kvm'> <name>winefitry3</name> <uuid>69a4ea18-6a00-41f4-afbc-80fd42ad60fd</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>4</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.8'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader> <nvram>/var/lib/libvirt/qemu/nvram/winefitry3_VARS.fd</nvram> <bootmenu enable='yes'/> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='AMDMachine57'/> </hyperv> <kvm> <hidden state='on'/> </kvm> <vmport state='off'/> </features> <cpu mode='host-passthrough'/> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/bin/kvm</emulator> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/opt/KNOPPIX_V7.7.1DVD-2016-10-22-EN.iso'/> <target dev='sda' bus='sata'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='vmdk'/> <source file='/vmware/passthrough/winefi3.vmdk'/> <target dev='sdb' bus='sata'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='vmdk'/> <source file='/vmware/passthrough/linuxtest.vmdk'/> <target dev='sdc' bus='sata'/> <boot order='1'/> <address type='drive' controller='0' bus='0' target='0' unit='2'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <interface type='direct'> <mac address='52:54:00:2c:a2:8e'/> <source dev='enp38s0' mode='bridge'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='2'/> </channel> <input type='tablet' bus='usb'> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='spice' autoport='yes'> <listen type='address'/> </graphics> <sound model='ich6'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x28' slot='0x00' function='0x0'/> </source> <rom file='/opt/Asus.GTX1060.3072.161104.rom'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x28' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </hostdev> <redirdev bus='usb' type='spicevmc'> <address type='usb' bus='0' port='2'/> </redirdev> <redirdev bus='usb' type='spicevmc'> <address type='usb' bus='0' port='3'/> </redirdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </memballoon> </devices> </domain> </div><div>Thanks for reading this far! </div></div>