[vfio-users] NVidia code 43 with Winddows 10 guest - does NVidia have another trick up their sleeve?
Robert Dahlstrom
robert.j.dahlstrom at gmail.com
Wed Jan 3 23:28:39 UTC 2018
Hello, I have been trying to setup a Windows 10 guest on my Linux host,
passing through an nvidia 1060. I've been having a very difficult time,
getting the dreaded code 43. I haven't followed an individual guide, but
have taken bits from https://wiki.archlinux.org/ind
ex.php/PCI_passthrough_via_OVMF and Alex Williamson's when relevant among
others.
I have tried just about every possible solution I've found, but have come
up short.
One thing I'll point out right away: in my VM, I put two virtual disks. On
one I installed windows, on the other I installed Debian 9. When booting
into the linux guest OS, with the proprietary nvidia driver, **everything
works**. When I boot into windows: code 43. Based on that, I think I have
most of my stuff correctly setup, and I am just at the receiving end of
nvidia treachery. I hope I'm wrong about that though.
Since both cards are nvidia cards, I use vfio.conf to prevent the nvidia
driver from loading on the guest. The guest GPU is in an IOMMU group that
consists solely of it, its HDMI audio device, and the Host Bridge/PCI
Bridge devices. Both the video and audio devices are forwarded to the guest.
What I have tried:
1) Seemingly every permutation of hidden kvm, vendor_id settings, hyperv
settings, both i440fx and q35, BIOS and OVMF,
2) My current situation has my guest GPU in the first PCIe slot, with my
host GPU in the third. I think the first 2 slots share an IOMMU group,
presumably for SLI or some such. This puts my guest GPU as the boot card,
which is what the system BIOS and bootloader comes up on. I read that may
cause trouble, and so I've tried swapping the cards. This I don't believe
works because there is a boatload of other stuff in the IOMMU group of the
third slot (Group 0), such as my host SATA and USB controllers, that I
can't forward.
3) Passing disable_vga in vfio.conf, doesn't seem to do anything
4) Setting MSI interrupt signaling in windows for both the video and audio
devices.
What I have not tried:
1) Switching which GPU I use for host and guest. If I can't use the 1060
the whole thing is pointless. It may tell me if the boot GPU issue is the
cause of my woe, however.
2) I have read sometimes booting with the virtual graphics device (Spice +
QXL) causes problems. Trying that, I have not been able to boot up at all
without it. Presence of -nographics on qemu commandline seems to make no
difference. When I blast the display from the config, the qemu process
starts up and sits at 0% and never does anything, and I can't later remote
desktop or ssh into the guest.
3) I have not yet successfully gone through the process of personally
downloading the ROM from the card and using it. I have instead grabbed a
ROM file from techpowerup.com, and verified that it has both BIOS and EFI
portions. I can passthrough in linux regardless whether I use the ROM or
not, and in Windows it makes no difference, code 43 for both. When I try to
download the ROM myself I get the Invalid PCI ROM header signature error,
and an input/output error.
4) Giving up and buying an AMD card.
Any ideas what I'm missing? The oldest drivers from NVidia are from march
2017, most info I can find on this subject is from 2016 or earlier. Do the
later drivers have additional checks that qemu is not able to fool at this
time? Thanks for any help, and if you need any additional information I'll
be happy to provide it.
System information:
* Linux 4.9.0-3-amd64 (Debian 9)
* qemu-x86_64 version 2.8.1(Debian 1:2.8+dfsg-6+deb9u3)
* virsh -version: 3.0.0
* Processor: AMD Ryzen 7 1700X
* Host GPU: NVidia 740, evga
* Guest GPU: Nvidia 1060, Asus
* Guest NVidia Drivers: 378.78 - the oldest I can get off the website.
/etc/modprobe.d/vfio.conf :
options vfio-pci ids=10de:1c02,10de:10f1 disable_vga=1
lspci -nnv: (This is after vfio is setup, vfio-pci drivers are in use as
expected)
....
21:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107 [GeForce
GT 740] [10de:0fc8] (rev a1) (prog-if 00 [VGA controller])
Subsystem: eVga.com. Corp. GK107 [GeForce GT 740] [3842:3747]
Flags: bus master, fast devsel, latency 0, IRQ 62
Memory at f4000000 (32-bit, non-prefetchable) [size=16M]
Memory at e0000000 (64-bit, prefetchable) [size=256M]
Memory at f0000000 (64-bit, prefetchable) [size=32M]
I/O ports at d000 [size=128]
[virtual] Expansion ROM at f5000000 [disabled] [size=512K]
Capabilities: <access denied>
Kernel driver in use: nvidia
Kernel modules: nvidia
21:00.1 Audio device [0403]: NVIDIA Corporation GK107 HDMI Audio Controller
[10de:0e1b] (rev a1)
Subsystem: eVga.com. Corp. GK107 HDMI Audio Controller [3842:3747]
Flags: bus master, fast devsel, latency 0, IRQ 43
Memory at f5080000 (32-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
....
28:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce
GTX 1060 3GB] [10de:1c02] (rev a1) (prog-if 00 [VGA controller])
Subsystem: ASUSTeK Computer Inc. GP106 [GeForce GTX 1060 3GB]
[1043:85b1]
Flags: bus master, fast devsel, latency 0, IRQ 63
Memory at f6000000 (32-bit, non-prefetchable) [size=16M]
Memory at c0000000 (64-bit, prefetchable) [size=256M]
Memory at d0000000 (64-bit, prefetchable) [size=32M]
I/O ports at e000 [size=128]
Expansion ROM at 000c0000 [disabled] [size=128K]
Capabilities: <access denied>
Kernel driver in use: vfio-pci
Kernel modules: nvidia
28:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f1] (rev a1)
Subsystem: ASUSTeK Computer Inc. Device [1043:85b1]
Flags: bus master, fast devsel, latency 0, IRQ 65
Memory at f7080000 (32-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
....
IOMMU groups:
IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc.
[AMD] Device [1022:1452]
IOMMU Group 0 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD]
Device [1022:1453]
IOMMU Group 0 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD]
Device [1022:1453]
IOMMU Group 0 01:00.0 Non-Volatile memory controller [0108]: Samsung
Electronics Co Ltd Device [144d:a804]
IOMMU Group 0 03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc.
[AMD] Device [1022:43b9] (rev 02)
IOMMU Group 0 03:00.1 SATA controller [0106]: Advanced Micro Devices, Inc.
[AMD] Device [1022:43b5] (rev 02)
IOMMU Group 0 03:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD]
Device [1022:43b0] (rev 02)
IOMMU Group 0 1d:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD]
Device [1022:43b4] (rev 02)
IOMMU Group 0 1d:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD]
Device [1022:43b4] (rev 02)
IOMMU Group 0 1d:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD]
Device [1022:43b4] (rev 02)
IOMMU Group 0 1d:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD]
Device [1022:43b4] (rev 02)
IOMMU Group 0 21:00.0 VGA compatible controller [0300]: NVIDIA Corporation
GK107 [GeForce GT 740] [10de:0fc8] (rev a1)
IOMMU Group 0 21:00.1 Audio device [0403]: NVIDIA Corporation GK107 HDMI
Audio Controller [10de:0e1b] (rev a1)
IOMMU Group 0 25:00.0 USB controller [0c03]: ASMedia Technology Inc. Device
[1b21:1343]
IOMMU Group 0 26:00.0 Ethernet controller [0200]: Intel Corporation I211
Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 1 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc.
[AMD] Device [1022:1452]
IOMMU Group 2 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc.
[AMD] Device [1022:1452]
IOMMU Group 2 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD]
Device [1022:1453]
IOMMU Group 2 28:00.0 VGA compatible controller [0300]: NVIDIA Corporation
GP106 [GeForce GTX 1060 3GB] [10de:1c02] (rev a1)
IOMMU Group 2 28:00.1 Audio device [0403]: NVIDIA Corporation Device
[10de:10f1] (rev a1)
IOMMU Group 3 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc.
[AMD] Device [1022:1452]
IOMMU Group 4 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc.
[AMD] Device [1022:1452]
IOMMU Group 4 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD]
Device [1022:1454]
IOMMU Group 4 29:00.0 Non-Essential Instrumentation [1300]: Advanced Micro
Devices, Inc. [AMD] Device [1022:145a]
IOMMU Group 4 29:00.2 Encryption controller [1080]: Advanced Micro Devices,
Inc. [AMD] Device [1022:1456]
IOMMU Group 4 29:00.3 USB controller [0c03]: Advanced Micro Devices, Inc.
[AMD] Device [1022:145c]
IOMMU Group 5 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc.
[AMD] Device [1022:1452]
IOMMU Group 5 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD]
Device [1022:1454]
IOMMU Group 5 2a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro
Devices, Inc. [AMD] Device [1022:1455]
IOMMU Group 5 2a:00.2 SATA controller [0106]: Advanced Micro Devices, Inc.
[AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 5 2a:00.3 Audio device [0403]: Advanced Micro Devices, Inc.
[AMD] Device [1022:1457]
IOMMU Group 6 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH
SMBus Controller [1022:790b] (rev 59)
IOMMU Group 6 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD]
FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 7 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc.
[AMD] Device [1022:1460]
IOMMU Group 7 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc.
[AMD] Device [1022:1461]
IOMMU Group 7 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc.
[AMD] Device [1022:1462]
IOMMU Group 7 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc.
[AMD] Device [1022:1463]
IOMMU Group 7 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc.
[AMD] Device [1022:1464]
IOMMU Group 7 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc.
[AMD] Device [1022:1465]
IOMMU Group 7 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc.
[AMD] Device [1022:1466]
IOMMU Group 7 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc.
[AMD] Device [1022:1467]
ROM I got from techpowerup:
rom-parser /opt/Asus.GTX1060.3072.161104.rom
Valid ROM signature found @a00h, PCIR offset 1a0h
PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 1c02, class: 030000
PCIR: revision 0, vendor revision: 1
Valid ROM signature found @fa00h, PCIR offset 1ch
PCIR: type 3 (EFI), vendor: 10de, device: 1c02, class: 030000
PCIR: revision 3, vendor revision: 0
EFI: Signature Valid, Subsystem: Boot, Machine: X64
Last image
libvirt config:
<domain type='kvm'>
<name>winefitry3</name>
<uuid>69a4ea18-6a00-41f4-afbc-80fd42ad60fd</uuid>
<memory unit='KiB'>4194304</memory>
<currentMemory unit='KiB'>4194304</currentMemory>
<vcpu placement='static'>4</vcpu>
<os>
<type arch='x86_64' machine='pc-i440fx-2.8'>hvm</type>
<loader readonly='yes' type='pflash'>/usr/share/OVMF/
OVMF_CODE.fd</loader>
<nvram>/var/lib/libvirt/qemu/nvram/winefitry3_VARS.fd</nvram>
<bootmenu enable='yes'/>
</os>
<features>
<acpi/>
<apic/>
<hyperv>
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='8191'/>
<vendor_id state='on' value='AMDMachine57'/>
</hyperv>
<kvm>
<hidden state='on'/>
</kvm>
<vmport state='off'/>
</features>
<cpu mode='host-passthrough'/>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/kvm</emulator>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<source file='/opt/KNOPPIX_V7.7.1DVD-2016-10-22-EN.iso'/>
<target dev='sda' bus='sata'/>
<readonly/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='vmdk'/>
<source file='/vmware/passthrough/winefi3.vmdk'/>
<target dev='sdb' bus='sata'/>
<address type='drive' controller='0' bus='0' target='0' unit='1'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='vmdk'/>
<source file='/vmware/passthrough/linuxtest.vmdk'/>
<target dev='sdc' bus='sata'/>
<boot order='1'/>
<address type='drive' controller='0' bus='0' target='0' unit='2'/>
</disk>
<controller type='usb' index='0' model='ich9-ehci1'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x7'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci1'>
<master startport='0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x0' multifunction='on'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci2'>
<master startport='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x1'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci3'>
<master startport='4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<controller type='sata' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x0'/>
</controller>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07'
function='0x0'/>
</controller>
<interface type='direct'>
<mac address='52:54:00:2c:a2:8e'/>
<source dev='enp38s0' mode='bridge'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
</interface>
<serial type='pty'>
<target port='0'/>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<channel type='unix'>
<target type='virtio' name='org.qemu.guest_agent.0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<channel type='spicevmc'>
<target type='virtio' name='com.redhat.spice.0'/>
<address type='virtio-serial' controller='0' bus='0' port='2'/>
</channel>
<input type='tablet' bus='usb'>
<address type='usb' bus='0' port='1'/>
</input>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='spice' autoport='yes'>
<listen type='address'/>
</graphics>
<sound model='ich6'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
</sound>
<video>
<model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'
primary='yes'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09'
function='0x0'/>
</video>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x28' slot='0x00' function='0x0'/>
</source>
<rom file='/opt/Asus.GTX1060.3072.161104.rom'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08'
function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x28' slot='0x00' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a'
function='0x0'/>
</hostdev>
<redirdev bus='usb' type='spicevmc'>
<address type='usb' bus='0' port='2'/>
</redirdev>
<redirdev bus='usb' type='spicevmc'>
<address type='usb' bus='0' port='3'/>
</redirdev>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0b'
function='0x0'/>
</memballoon>
</devices>
</domain>
Thanks for reading this far!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20180103/459a57e0/attachment.htm>
More information about the vfio-users
mailing list