[vfio-users] NVidia code 43 with Winddows 10 guest - does NVidia have another trick up their sleeve?
Alex Kretzschmar
alexktz at gmail.com
Thu Jan 4 01:54:25 UTC 2018
I bet you've already tried this but looking through your xml file for the
guest I see a few inconsistencies when compared against the suggestions on
the Arch Wiki [here](https://wiki.archlinux.org/index.php/PCI_passthrough_
via_OVMF#.22Error_43:_Driver_failed_to_load.22_on_Nvidia_
GPUs_passed_to_Windows_VMs).
I found the important bits to the <hyperv> tags. Hope this helps.
Alex
On 3 January 2018 at 23:28, Robert Dahlstrom <robert.j.dahlstrom at gmail.com>
wrote:
> Hello, I have been trying to setup a Windows 10 guest on my Linux host,
> passing through an nvidia 1060. I've been having a very difficult time,
> getting the dreaded code 43. I haven't followed an individual guide, but
> have taken bits from https://wiki.archlinux.org/ind
> ex.php/PCI_passthrough_via_OVMF and Alex Williamson's when relevant among
> others.
>
> I have tried just about every possible solution I've found, but have come
> up short.
>
> One thing I'll point out right away: in my VM, I put two virtual disks. On
> one I installed windows, on the other I installed Debian 9. When booting
> into the linux guest OS, with the proprietary nvidia driver, **everything
> works**. When I boot into windows: code 43. Based on that, I think I have
> most of my stuff correctly setup, and I am just at the receiving end of
> nvidia treachery. I hope I'm wrong about that though.
>
> Since both cards are nvidia cards, I use vfio.conf to prevent the nvidia
> driver from loading on the guest. The guest GPU is in an IOMMU group that
> consists solely of it, its HDMI audio device, and the Host Bridge/PCI
> Bridge devices. Both the video and audio devices are forwarded to the guest.
>
> What I have tried:
>
> 1) Seemingly every permutation of hidden kvm, vendor_id settings, hyperv
> settings, both i440fx and q35, BIOS and OVMF,
>
> 2) My current situation has my guest GPU in the first PCIe slot, with my
> host GPU in the third. I think the first 2 slots share an IOMMU group,
> presumably for SLI or some such. This puts my guest GPU as the boot card,
> which is what the system BIOS and bootloader comes up on. I read that may
> cause trouble, and so I've tried swapping the cards. This I don't believe
> works because there is a boatload of other stuff in the IOMMU group of the
> third slot (Group 0), such as my host SATA and USB controllers, that I
> can't forward.
>
> 3) Passing disable_vga in vfio.conf, doesn't seem to do anything
>
> 4) Setting MSI interrupt signaling in windows for both the video and
> audio devices.
>
> What I have not tried:
>
> 1) Switching which GPU I use for host and guest. If I can't use the 1060
> the whole thing is pointless. It may tell me if the boot GPU issue is the
> cause of my woe, however.
>
> 2) I have read sometimes booting with the virtual graphics device (Spice
> + QXL) causes problems. Trying that, I have not been able to boot up at all
> without it. Presence of -nographics on qemu commandline seems to make no
> difference. When I blast the display from the config, the qemu process
> starts up and sits at 0% and never does anything, and I can't later remote
> desktop or ssh into the guest.
>
> 3) I have not yet successfully gone through the process of personally
> downloading the ROM from the card and using it. I have instead grabbed a
> ROM file from techpowerup.com, and verified that it has both BIOS and EFI
> portions. I can passthrough in linux regardless whether I use the ROM or
> not, and in Windows it makes no difference, code 43 for both. When I try to
> download the ROM myself I get the Invalid PCI ROM header signature error,
> and an input/output error.
>
> 4) Giving up and buying an AMD card.
>
> Any ideas what I'm missing? The oldest drivers from NVidia are from march
> 2017, most info I can find on this subject is from 2016 or earlier. Do the
> later drivers have additional checks that qemu is not able to fool at this
> time? Thanks for any help, and if you need any additional information I'll
> be happy to provide it.
>
> System information:
>
> * Linux 4.9.0-3-amd64 (Debian 9)
> * qemu-x86_64 version 2.8.1(Debian 1:2.8+dfsg-6+deb9u3)
> * virsh -version: 3.0.0
> * Processor: AMD Ryzen 7 1700X
> * Host GPU: NVidia 740, evga
> * Guest GPU: Nvidia 1060, Asus
> * Guest NVidia Drivers: 378.78 - the oldest I can get off the website.
>
> /etc/modprobe.d/vfio.conf :
> options vfio-pci ids=10de:1c02,10de:10f1 disable_vga=1
>
> lspci -nnv: (This is after vfio is setup, vfio-pci drivers are in use as
> expected)
>
> ....
> 21:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107
> [GeForce GT 740] [10de:0fc8] (rev a1) (prog-if 00 [VGA controller])
> Subsystem: eVga.com. Corp. GK107 [GeForce GT 740] [3842:3747]
> Flags: bus master, fast devsel, latency 0, IRQ 62
> Memory at f4000000 (32-bit, non-prefetchable) [size=16M]
> Memory at e0000000 (64-bit, prefetchable) [size=256M]
> Memory at f0000000 (64-bit, prefetchable) [size=32M]
> I/O ports at d000 [size=128]
> [virtual] Expansion ROM at f5000000 [disabled] [size=512K]
> Capabilities: <access denied>
> Kernel driver in use: nvidia
> Kernel modules: nvidia
>
> 21:00.1 Audio device [0403]: NVIDIA Corporation GK107 HDMI Audio
> Controller [10de:0e1b] (rev a1)
> Subsystem: eVga.com. Corp. GK107 HDMI Audio Controller [3842:3747]
> Flags: bus master, fast devsel, latency 0, IRQ 43
> Memory at f5080000 (32-bit, non-prefetchable) [size=16K]
> Capabilities: <access denied>
> Kernel driver in use: snd_hda_intel
> Kernel modules: snd_hda_intel
>
> ....
>
> 28:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106
> [GeForce GTX 1060 3GB] [10de:1c02] (rev a1) (prog-if 00 [VGA controller])
> Subsystem: ASUSTeK Computer Inc. GP106 [GeForce GTX 1060 3GB]
> [1043:85b1]
> Flags: bus master, fast devsel, latency 0, IRQ 63
> Memory at f6000000 (32-bit, non-prefetchable) [size=16M]
> Memory at c0000000 (64-bit, prefetchable) [size=256M]
> Memory at d0000000 (64-bit, prefetchable) [size=32M]
> I/O ports at e000 [size=128]
> Expansion ROM at 000c0000 [disabled] [size=128K]
> Capabilities: <access denied>
> Kernel driver in use: vfio-pci
> Kernel modules: nvidia
>
> 28:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f1] (rev a1)
> Subsystem: ASUSTeK Computer Inc. Device [1043:85b1]
> Flags: bus master, fast devsel, latency 0, IRQ 65
> Memory at f7080000 (32-bit, non-prefetchable) [size=16K]
> Capabilities: <access denied>
> Kernel driver in use: vfio-pci
> Kernel modules: snd_hda_intel
>
> ....
>
>
> IOMMU groups:
>
> IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc.
> [AMD] Device [1022:1452]
> IOMMU Group 0 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc.
> [AMD] Device [1022:1453]
> IOMMU Group 0 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc.
> [AMD] Device [1022:1453]
> IOMMU Group 0 01:00.0 Non-Volatile memory controller [0108]: Samsung
> Electronics Co Ltd Device [144d:a804]
> IOMMU Group 0 03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc.
> [AMD] Device [1022:43b9] (rev 02)
> IOMMU Group 0 03:00.1 SATA controller [0106]: Advanced Micro Devices, Inc.
> [AMD] Device [1022:43b5] (rev 02)
> IOMMU Group 0 03:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc.
> [AMD] Device [1022:43b0] (rev 02)
> IOMMU Group 0 1d:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc.
> [AMD] Device [1022:43b4] (rev 02)
> IOMMU Group 0 1d:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc.
> [AMD] Device [1022:43b4] (rev 02)
> IOMMU Group 0 1d:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc.
> [AMD] Device [1022:43b4] (rev 02)
> IOMMU Group 0 1d:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc.
> [AMD] Device [1022:43b4] (rev 02)
> IOMMU Group 0 21:00.0 VGA compatible controller [0300]: NVIDIA Corporation
> GK107 [GeForce GT 740] [10de:0fc8] (rev a1)
> IOMMU Group 0 21:00.1 Audio device [0403]: NVIDIA Corporation GK107 HDMI
> Audio Controller [10de:0e1b] (rev a1)
> IOMMU Group 0 25:00.0 USB controller [0c03]: ASMedia Technology Inc.
> Device [1b21:1343]
> IOMMU Group 0 26:00.0 Ethernet controller [0200]: Intel Corporation I211
> Gigabit Network Connection [8086:1539] (rev 03)
> IOMMU Group 1 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc.
> [AMD] Device [1022:1452]
> IOMMU Group 2 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc.
> [AMD] Device [1022:1452]
> IOMMU Group 2 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc.
> [AMD] Device [1022:1453]
> IOMMU Group 2 28:00.0 VGA compatible controller [0300]: NVIDIA Corporation
> GP106 [GeForce GTX 1060 3GB] [10de:1c02] (rev a1)
> IOMMU Group 2 28:00.1 Audio device [0403]: NVIDIA Corporation Device
> [10de:10f1] (rev a1)
> IOMMU Group 3 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc.
> [AMD] Device [1022:1452]
> IOMMU Group 4 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc.
> [AMD] Device [1022:1452]
> IOMMU Group 4 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc.
> [AMD] Device [1022:1454]
> IOMMU Group 4 29:00.0 Non-Essential Instrumentation [1300]: Advanced Micro
> Devices, Inc. [AMD] Device [1022:145a]
> IOMMU Group 4 29:00.2 Encryption controller [1080]: Advanced Micro
> Devices, Inc. [AMD] Device [1022:1456]
> IOMMU Group 4 29:00.3 USB controller [0c03]: Advanced Micro Devices, Inc.
> [AMD] Device [1022:145c]
> IOMMU Group 5 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc.
> [AMD] Device [1022:1452]
> IOMMU Group 5 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc.
> [AMD] Device [1022:1454]
> IOMMU Group 5 2a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro
> Devices, Inc. [AMD] Device [1022:1455]
> IOMMU Group 5 2a:00.2 SATA controller [0106]: Advanced Micro Devices, Inc.
> [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
> IOMMU Group 5 2a:00.3 Audio device [0403]: Advanced Micro Devices, Inc.
> [AMD] Device [1022:1457]
> IOMMU Group 6 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH
> SMBus Controller [1022:790b] (rev 59)
> IOMMU Group 6 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc.
> [AMD] FCH LPC Bridge [1022:790e] (rev 51)
> IOMMU Group 7 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc.
> [AMD] Device [1022:1460]
> IOMMU Group 7 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc.
> [AMD] Device [1022:1461]
> IOMMU Group 7 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc.
> [AMD] Device [1022:1462]
> IOMMU Group 7 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc.
> [AMD] Device [1022:1463]
> IOMMU Group 7 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc.
> [AMD] Device [1022:1464]
> IOMMU Group 7 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc.
> [AMD] Device [1022:1465]
> IOMMU Group 7 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc.
> [AMD] Device [1022:1466]
> IOMMU Group 7 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc.
> [AMD] Device [1022:1467]
>
> ROM I got from techpowerup:
>
> rom-parser /opt/Asus.GTX1060.3072.161104.rom
> Valid ROM signature found @a00h, PCIR offset 1a0h
> PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 1c02, class: 030000
> PCIR: revision 0, vendor revision: 1
> Valid ROM signature found @fa00h, PCIR offset 1ch
> PCIR: type 3 (EFI), vendor: 10de, device: 1c02, class: 030000
> PCIR: revision 3, vendor revision: 0
> EFI: Signature Valid, Subsystem: Boot, Machine: X64
> Last image
>
> libvirt config:
>
> <domain type='kvm'>
> <name>winefitry3</name>
> <uuid>69a4ea18-6a00-41f4-afbc-80fd42ad60fd</uuid>
> <memory unit='KiB'>4194304</memory>
> <currentMemory unit='KiB'>4194304</currentMemory>
> <vcpu placement='static'>4</vcpu>
> <os>
> <type arch='x86_64' machine='pc-i440fx-2.8'>hvm</type>
> <loader readonly='yes' type='pflash'>/usr/share/OVMF/
> OVMF_CODE.fd</loader>
> <nvram>/var/lib/libvirt/qemu/nvram/winefitry3_VARS.fd</nvram>
> <bootmenu enable='yes'/>
> </os>
> <features>
> <acpi/>
> <apic/>
> <hyperv>
> <relaxed state='on'/>
> <vapic state='on'/>
> <spinlocks state='on' retries='8191'/>
> <vendor_id state='on' value='AMDMachine57'/>
> </hyperv>
> <kvm>
> <hidden state='on'/>
> </kvm>
> <vmport state='off'/>
> </features>
> <cpu mode='host-passthrough'/>
> <clock offset='utc'>
> <timer name='rtc' tickpolicy='catchup'/>
> <timer name='pit' tickpolicy='delay'/>
> <timer name='hpet' present='no'/>
> </clock>
> <on_poweroff>destroy</on_poweroff>
> <on_reboot>restart</on_reboot>
> <on_crash>restart</on_crash>
> <pm>
> <suspend-to-mem enabled='no'/>
> <suspend-to-disk enabled='no'/>
> </pm>
> <devices>
> <emulator>/usr/bin/kvm</emulator>
> <disk type='file' device='cdrom'>
> <driver name='qemu' type='raw'/>
> <source file='/opt/KNOPPIX_V7.7.1DVD-2016-10-22-EN.iso'/>
> <target dev='sda' bus='sata'/>
> <readonly/>
> <address type='drive' controller='0' bus='0' target='0' unit='0'/>
> </disk>
> <disk type='file' device='disk'>
> <driver name='qemu' type='vmdk'/>
> <source file='/vmware/passthrough/winefi3.vmdk'/>
> <target dev='sdb' bus='sata'/>
> <address type='drive' controller='0' bus='0' target='0' unit='1'/>
> </disk>
> <disk type='file' device='disk'>
> <driver name='qemu' type='vmdk'/>
> <source file='/vmware/passthrough/linuxtest.vmdk'/>
> <target dev='sdc' bus='sata'/>
> <boot order='1'/>
> <address type='drive' controller='0' bus='0' target='0' unit='2'/>
> </disk>
> <controller type='usb' index='0' model='ich9-ehci1'>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
> function='0x7'/>
> </controller>
> <controller type='usb' index='0' model='ich9-uhci1'>
> <master startport='0'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
> function='0x0' multifunction='on'/>
> </controller>
> <controller type='usb' index='0' model='ich9-uhci2'>
> <master startport='2'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
> function='0x1'/>
> </controller>
> <controller type='usb' index='0' model='ich9-uhci3'>
> <master startport='4'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
> function='0x2'/>
> </controller>
> <controller type='pci' index='0' model='pci-root'/>
> <controller type='sata' index='0'>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> function='0x0'/>
> </controller>
> <controller type='virtio-serial' index='0'>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
> function='0x0'/>
> </controller>
> <interface type='direct'>
> <mac address='52:54:00:2c:a2:8e'/>
> <source dev='enp38s0' mode='bridge'/>
> <model type='virtio'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
> </interface>
> <serial type='pty'>
> <target port='0'/>
> </serial>
> <console type='pty'>
> <target type='serial' port='0'/>
> </console>
> <channel type='unix'>
> <target type='virtio' name='org.qemu.guest_agent.0'/>
> <address type='virtio-serial' controller='0' bus='0' port='1'/>
> </channel>
> <channel type='spicevmc'>
> <target type='virtio' name='com.redhat.spice.0'/>
> <address type='virtio-serial' controller='0' bus='0' port='2'/>
> </channel>
> <input type='tablet' bus='usb'>
> <address type='usb' bus='0' port='1'/>
> </input>
> <input type='mouse' bus='ps2'/>
> <input type='keyboard' bus='ps2'/>
> <graphics type='spice' autoport='yes'>
> <listen type='address'/>
> </graphics>
> <sound model='ich6'>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
> function='0x0'/>
> </sound>
> <video>
> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'
> primary='yes'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
> function='0x0'/>
> </video>
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0x28' slot='0x00' function='0x0'/>
> </source>
> <rom file='/opt/Asus.GTX1060.3072.161104.rom'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x08'
> function='0x0'/>
> </hostdev>
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0x28' slot='0x00' function='0x1'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a'
> function='0x0'/>
> </hostdev>
> <redirdev bus='usb' type='spicevmc'>
> <address type='usb' bus='0' port='2'/>
> </redirdev>
> <redirdev bus='usb' type='spicevmc'>
> <address type='usb' bus='0' port='3'/>
> </redirdev>
> <memballoon model='virtio'>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b'
> function='0x0'/>
> </memballoon>
> </devices>
> </domain>
>
> Thanks for reading this far!
>
> _______________________________________________
> vfio-users mailing list
> vfio-users at redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users
>
>
--
Alex Kretzschmar
07796 863 373
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20180104/f19aabb1/attachment.htm>
More information about the vfio-users
mailing list