[vfio-users] VM not booting/no video output when PCI passthrough GPU is added

[Roel Van Gorp]

Hi guys,

After successfully setting up a W10 VM while passing through my GTX 980ti
on my old Arch linux install, I now have some difficulties setting it up on
a new Fedora installation using the exact same hardware configuration.

I'm using a fresh Fedora 23 installation, kernel 4.2.6-301, the
virt-preview repo for the latest virtualization packages and the firmware
repo at https://www.kraxel.org/repos/firmware.repo for the latest OVFM
firmware.

I mostly based myself on Alex Williamson's blog to set this up, resulting
in the following changes in the following files:

*/etc/sysconfig/grub:*

  GRUB_CMDLINE_LINUX="rd.lvm.lv=fedora/root rd.lvm.lv=fedora/swap rhgb
quiet intel_iommu=on rd.driver.pre=vfio-pci"

*/etc/modprobe.d/local.conf *

   options vfio-pci ids=10de:17c8,10de:0fb0
   options vfio-pci disable_vga=1

*/etc/dracut.conf.d/local.conf*

   add_drivers+="vfio vfio_iommu_type1 vfio_pci vfio_virqfd"


This seems to have successfully enabled the vfio-pci driver, as shown by
the output of lspci -nnk:

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce
GTX 980 Ti] [10de:17c8] (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3232]
Kernel driver in use: vfio-pci
Kernel modules: nouveau
01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fb0] (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3232]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel

*The VM:*

Now I first set up the VM without PCI passthrough using virt-manager and
everything worked as expected using a virtual display, allowing me to fully
install Windows 10 Pro. The problems started when removing the virtual
display and adding the PCI devices for the GTX 980 ti. There is no video
output, and looking at the CPU monitor in virt-manager there is a short CPU
spike after which it seems hangs at around 1-2%. Reattaching the virtual
display again shows the bios splash screen normally as expected, after
which it goes to black with just the small white prompt dash. Removing the
PCI devices from the configuration again allow the VM to boot normally
again. Here is the xml config for the VM:

<domain type='kvm'>
  <name>win10</name>
  <uuid>527f731e-0d9e-460a-a1ef-7a010e191c34</uuid>
  <memory unit='KiB'>2560000</memory>
  <currentMemory unit='KiB'>2560000</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.4'>hvm</type>
    <loader readonly='yes'
type='pflash'>/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
    <vmport state='off'/>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='2' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/win10.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <interface type='network'>
      <mac address='52:54:00:21:17:b8'/>
      <source network='default'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
    </interface>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08'
function='0x0'/>
    </memballoon>
  </devices>
</domain>


Any help would be greatly appreciated!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20151215/9270d942/attachment.htm>