[vfio-users] Windows 10 guest refuses to boot when passed my NVIDIA graphics card

Tue Sep 15 15:32:52 UTC 2015

Hello everyone,

I have been following the guides on the vfio blogspot for setting up a
Windows guest for gaming by passing it my NVIDIA GeForce GTX 980 graphics
card. I have an AMD Radeon R7 250 graphics card for my Linux host. I have
ran into a strange problem and have not managed to find a solution to it
online, so I decided to come here and see if anyone knows anything about it.

Here is the problem I am facing:

I installed the Windows guest (Windows 10 Pro) in a kvm virtual machine
made with virt-manager, using the default qxl/spice setup during the
installation, as instructed here
<http://vfio.blogspot.bg/2015/05/vfio-gpu-how-to-series-part-4-our-first.html>.
After the installation completed, I removed any extra virtual devices and
added the PCI host/vfio device for my NVIDIA graphics card. I also edited
the XML to hide the kvm virtualization, in preparation for installing the
NVIDIA drivers on Windows, when I boot with my actual graphics card. I have
made no attempt to configure hugepages yet.

However, when starting up the VM, Windows no longer booted successfully. I
saw the Tianocore splash on my physical monitor, followed by the Windows
logo. The Windows logo stayed frozen (without the spinning loading
indicator under it) for a few minutes, and then the spinning-dots loading
indicator appeared for a few seconds and the machine *reset*. It kept
resetting again and again, never getting past the Windows boot screen.

I would assume that my PCI passthrough worked successfully, since my
physical monitor turned on and I saw the video output from the virtual
machine on it. However, I can't figure out why Windows cannot finish
booting successfully. Any help with this issue would be greatly appreciated.

The same copy of Windows (or rather, installed from the same installation
ISO) works fine with my NVIDIA card on my actual hardware / when not in a
virtual machine, so I am sure this is not a hardware problem.

Here is some relevant information about my system:

/proc/cmdline:
... intel_iommu=on iommu=pt vfio-pci.ids=10de:13c0,10de:0fbb,8086:8d26
vfio.disable_vga=1 ...
(I omitted my rootfs and other irrelevant kernel options)
The PCI IDs listed in my kernel commandline are my GPU, its audio, and an
EHCI controller on my system for USB, in that order.

I did not need to use any "hacks" like the ACS override patch or the
enable_unsafe_interrupts option, since I did not experience the relevant
issues that they were made to fix. Each one of the PCI IDs above is in its
own IOMMU group. My system has no integrated graphics (Core i7 Extreme
5960X CPU).

These lines appeared in dmesg when the VM started:

[  582.886435] kvm: SMP vm created on host with unstable TSC; guest TSC
will not be reliable
[  584.200023] vfio-pci 0000:02:00.0: enabling device (0100 -> 0103)
[  584.200114] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x1e at 0x258
[  584.200120] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19 at 0x900
[  584.223041] vfio-pci 0000:00:1d.0: enabling device (0000 -> 0002)
[  584.324093] vfio_cap_init: 0000:00:1d.0 hiding cap 0xa
[  589.132151] kvm: zapping shadow pages for mmio generation wraparound
[  589.133051] kvm: zapping shadow pages for mmio generation wraparound

The libvirt domain XML of the virtual machine (yes, I called my VM
"nouveideo" = "nouveau" + "video" :) ):

<domain type='kvm'>
 <name>nouveideo</name>
 <uuid>849fe365-1dfe-44f6-9755-9b48eb325400</uuid>
 <memory unit='KiB'>12345344</memory>
 <currentMemory unit='KiB'>12345344</currentMemory>
 <memoryBacking>
   <nosharepages/>
 </memoryBacking>
 <vcpu placement='static'>16</vcpu>
 <cputune>
   <vcpupin vcpu='0' cpuset='0'/>
   <vcpupin vcpu='1' cpuset='1'/>
   <vcpupin vcpu='2' cpuset='2'/>
   <vcpupin vcpu='3' cpuset='3'/>
   <vcpupin vcpu='4' cpuset='4'/>
   <vcpupin vcpu='5' cpuset='5'/>
   <vcpupin vcpu='6' cpuset='6'/>
   <vcpupin vcpu='7' cpuset='7'/>
   <vcpupin vcpu='8' cpuset='8'/>
   <vcpupin vcpu='9' cpuset='9'/>
   <vcpupin vcpu='10' cpuset='10'/>
   <vcpupin vcpu='11' cpuset='11'/>
   <vcpupin vcpu='12' cpuset='12'/>
   <vcpupin vcpu='13' cpuset='13'/>
   <vcpupin vcpu='14' cpuset='14'/>
   <vcpupin vcpu='15' cpuset='15'/>
 </cputune>
 <os>
   <type arch='x86_64' machine='pc-i440fx-2.4'>hvm</type>
   <loader type='pflash'>/btrfs/VirtualMachines/OVMF_nouveideo.fd</loader>
 </os>
 <features>
   <acpi/>
   <apic/>
   <pae/>
   <kvm>
     <hidden state='on'/>
   </kvm>
   <vmport state='off'/>
 </features>
 <cpu mode='host-passthrough'>
   <topology sockets='1' cores='8' threads='2'/>
 </cpu>
 <clock offset='utc'>
   <timer name='rtc' tickpolicy='catchup'/>
   <timer name='pit' tickpolicy='delay'/>
   <timer name='hpet' present='no'/>
 </clock>
 <on_poweroff>destroy</on_poweroff>
 <on_reboot>restart</on_reboot>
 <on_crash>restart</on_crash>
 <pm>
   <suspend-to-mem enabled='no'/>
   <suspend-to-disk enabled='no'/>
 </pm>
 <devices>
   <emulator>/usr/bin/qemu-system-x86_64</emulator>
   <disk type='file' device='disk'>
     <driver name='qemu' type='raw'/>
     <source file='/btrfs/VirtualMachines/Win10NOUVEIDEO.img'/>
     <target dev='vda' bus='virtio'/>
     <boot order='1'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
function='0x0'/>
   </disk>
   <disk type='block' device='cdrom'>
     <driver name='qemu' type='raw'/>
     <target dev='hdb' bus='ide'/>
     <readonly/>
     <boot order='2'/>
     <address type='drive' controller='0' bus='0' target='0' unit='1'/>
   </disk>
   <disk type='file' device='cdrom'>
     <driver name='qemu' type='raw'/>
     <source file='/btrfs/VirtualMachines/virtio-win-0.1.109.iso'/>
     <target dev='hdc' bus='ide'/>
     <readonly/>
     <address type='drive' controller='0' bus='1' target='0' unit='0'/>
   </disk>
   <controller type='pci' index='0' model='pci-root'/>
   <controller type='ide' index='0'>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x1'/>
   </controller>
   <controller type='usb' index='0' model='nec-xhci'>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/>
   </controller>
   <interface type='bridge'>
     <mac address='52:54:00:b3:71:21'/>
     <source bridge='br0'/>
     <model type='virtio'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
   </interface>
   <sound model='ich6'>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
   </sound>
   <hostdev mode='subsystem' type='pci' managed='yes'>
     <source>
       <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
     </source>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x0a'
function='0x0' multifunction='on'/>
   </hostdev>
   <hostdev mode='subsystem' type='pci' managed='yes'>
     <source>
       <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
     </source>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x0a'
function='0x1'/>
   </hostdev>
   <hostdev mode='subsystem' type='pci' managed='yes'>
     <source>
       <address domain='0x0000' bus='0x00' slot='0x1d' function='0x0'/>
     </source>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
function='0x0'/>
   </hostdev>
   <memballoon model='virtio'>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x08'
function='0x0'/>
   </memballoon>
 </devices>
</domain>

Let me know if there is any other information I could provide which could
help identify and correct the problem.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20150915/9183333b/attachment.htm>