Oh, and remove that "allow_unsafe_interrupts" snippet, we're telling people not to add it if they weren't told so in dmesg for two or more years but still. <div class="gmail_quote">On Sep 16, 2015 5:32 PM, "Blank Field" <<a href="mailto:ihatethisfield@gmail.com">ihatethisfield@gmail.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">You've gone the right way in running OVMF instead of VGA, as it is broken on your platform. Code 43: nvidia drivers have a check code that disables the card if it appears to be attached to a VM. Do you have hyper-v enlightenments turned off and kvm cpuid signature also turned off? Otherwise code 43 is just as bad as code 10: the driver fails to start the device for whatever reason it has. <div class="gmail_quote">On Sep 16, 2015 5:11 PM, "Mogliii" <<a href="mailto:mogliii@gmx.net" target="_blank">mogliii@gmx.net</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello, I have been trying to get vga passthrough working for quite some time now. I even ordered a new graphics card just for that purpose. I explain my setup and configuration below. My current problem: I can install Windows 8.1 with uefi (gpt partition layout) and using virt-manager. Then, I shutdown the VM, remove the spice and graphics, and add the pcie MSI GTX 750 Ti Gaming (it has a legacy and uefi bios; uefi bios selected by physical switch). When I then boot-up the VM, I see the OVMF bios on the physical screen attached to the GTX750 Ti (sometimes I get graphics glitches i.e. all white) and Windows 8.1 boots into 800x600. I see that a Nvidia GTX750Ti is recognized in the device manager. After installing driver (I tried 335.23, 340.52, 341.44 and most recent) the gpu is shown with yellow exclamation mark with error code 43 and I have to restart. After restart, I see again the bios and the boot splash, but then the screen goes into power save mode because it does not get any signal just when the login screen is supposed to appear. I can login via vnc and see that the gpu has error code 43. De-installing the driver and restarting I will go back to a 800x600 desktop on my physical screen. Any help with my setup would be immensely appreciated. I have another spare ASUS HD6570, but it does not have uefi bios. I am not sure if I would need i915 patch in that case? Does that patch only apply to Intel CPUs? Is the 00:02.2 pci device the problem? I would really appreciate any hint as I currently exhausted all options I see and I would like to avoid spending more money on video cards that I'm not sure I can even use. ========================== SETUP ========= [[[ 1. Physical setup Asus A88x-PRO AMD A10-7850K (on-board GPU used for linux host with radeon driver (fglrx not installed)) 12 GB memory MSI GTX 750 Ti Gaming (bios can be switched between legacy/uefi by hardware switch; using uefi) GPU is installed in middle pcie 16x slot (black, not golden) due to iommu grouping (see below) [[[ 2. OS UBuntu 15.04 64 bit, unpatched $ uname -a Linux host 3.19.0-28-generic #30-Ubuntu SMP Mon Aug 31 15:52:51 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux [[[ 3. Relevant packages (not sure if these are all) virt-manager 1:1.2.1-1~getdeb1 (get deb ppa) qemu 1:2.2+dfsg-5expubuntu9.4 libvirt-bin 1.2.12-0ubuntu14.2 qemu-kvm 1:2.2+dfsg-5expubuntu9.4 [[[ 4. Bios settings Set to boot from on board GPU. AMD-V and IOMMU activated. Legacy boot for ubuntu host [[[ 5. iommu grouping (iommu.sh output) ### Group 2 ### 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1424 00:02.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1425 01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750 Ti] (rev a2) 01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1) [[[ 6. pci identifier 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM107 [GeForce GTX 750 Ti] [10de:1380] (rev a2) Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3102] Kernel driver in use: pci-stub 01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbc] (rev a1) Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3102] Kernel driver in use: pci-stub [[[ 7. /proc/cmdline BOOT_IMAGE=/vmlinuz-3.19.0-28-generic root=/dev/mapper/main-root ro rootflags=subvol=@ quiet splash nomdmonddf nomdmonisw amd_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=10de:0fbc,10de:1380 vt.handoff=7 [[[ 8. /etc/initramfs-tools/modules pci_stub ids=10de:0fbc,10de:1380 [[[ 9. /etc/modules lp rtc pci_stub vfio vfio_iommu_type1 vfio_pci kvm kvm_amd [[[ 10. # dmseg | grep pci-stub [ 1.263269] pci-stub: add 10DE:0FBC sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 1.263287] pci-stub 0000:01:00.1: claimed by stub [ 1.263296] pci-stub: add 10DE:1380 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 1.263304] pci-stub 0000:01:00.0: claimed by stub [[[ 11. /usr/windows.sh (executed before running vm. Afterwards lspci -nnk shows "vfio-pci" as kernel driver in use for 01:00.0 and 01:00.1) #!/bin/bash configfile=/etc/vfio_pci1.cfg vfiobind() { dev="$1" vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id } modprobe vfio-pci cat $configfile | while read line;do echo $line | grep ^# >/dev/null 2>&1 && continue vfiobind $line done exit 0 [[[ 12. /etc/vfio_pci1.cfg 0000:01:00.0 0000:01:00.1 [[[ 13. OVMF-bios Got from sourceforge (edk2.git-ovmf-x64-0-20150910.b1206.g5abc2a7.noarch.rpm) and extracted. Then, made individual copy of OVMF-pure-efi.fd (md5 fbb13f2d4be2f9d72472628f885f5cb4 ) for VM. [[[ 14. Created VM in virt-manager using the following configuration (configuration shown after installation, i.e. no cdroms, but before attaching pci) <domain type='kvm'> <name>win81</name> <uuid>redacted</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>2</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-utopic'>hvm</type> <loader type='rom'>/usr/share/ovmf/OVMF-pure-efi_win81.fd</loader> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> <kvm> <hidden state='on'/> </kvm> </features> <cpu mode='host-passthrough'> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/main/Win7xen'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:ab:b6:10'/> <source bridge='br1'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </interface> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='spice' autoport='yes'/> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x0603'/> <product id='0x1605'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x05ac'/> <product id='0x1006'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x05ac'/> <product id='0x024f'/> </source> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> </domain> [[[ 15. After installation and installation of updates, shut down the VM [[[ 16. Attached PCI devices and removed video and display spice. Below is virsh configuration after attaching pcie passthrough. <domain type='kvm'> <name>win81</name> <uuid>redacted</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>2</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-utopic'>hvm</type> <loader type='rom'>/usr/share/ovmf/OVMF-pure-efi_win81.fd</loader> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> <kvm> <hidden state='on'/> </kvm> </features> <cpu mode='host-passthrough'> </cpu> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/main/Win7xen'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:ab:b6:10'/> <source bridge='br1'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </interface> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x0603'/> <product id='0x1605'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x05ac'/> <product id='0x1006'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x05ac'/> <product id='0x024f'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> </domain> [[[ 17. What else did I try - Different nvidia drivers (see top of email) - Attaching 00:02.2 as well, but cannot do vfio-pci assignement so failed # virsh nodedev-list --tree +- pci_0000_00_02_0 +- pci_0000_00_02_2 | | | +- pci_0000_01_00_0 | +- pci_0000_01_00_1 | +- pci_0000_00_03_0 # lspci -nnk (extract) 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1424] 00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1425] Kernel driver in use: pcieport - Reinstall the whole VM with seabios and using the GTX 750 Ti with legacy bios (physical switch on card). But no picture was shown on screen after attaching, not even the bios. - Tried to use q35 emulation, but not successful. It does not work with ovmf? I.e. I cannot use the qemu:cmd or ioh3420. _______________________________________________ vfio-users mailing list <a href="mailto:vfio-users@redhat.com" target="_blank">vfio-users@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/vfio-users" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/vfio-users</a> </blockquote></div> </blockquote></div>