[vfio-users] Initialization problem with pass-through video card
Alex Williamson
alex.williamson at redhat.com
Fri May 6 19:38:27 UTC 2016
On Fri, 6 May 2016 13:07:23 -0500
Sheldon Palmer <slf_palmer at hotmail.com> wrote:
> Ok so even though I have gone through Alex’s “step by step” (http://vfio.blogspot.com/)
> several times over, I think I am still missing something because each time I get
> the same result. That is: my pass-through video card doesn’t get initialized when
> I actually add it and boot the VM. There is just a quick flicker then a black
> screen. In fact, I don’t even think the OS boots because I can’t connect to it via
> VNC nor TeamViewer. The VM boots okay if I remove the video card.
>
>
>
> Specs:
>
> Processor: Intel Core i7-3770 Quad Core
>
> Host: Intel onboard video card with i915 driver / Gentoo
>
> Pass-through: ASUS GeForce GTX 980Ti STRIX / Windows 8.1 Pro
>
> qemu-2.5.1 / libvirt-1.3.1 / virt-manager-1.3.2 / kernel: linux-4.1.15
>
>
>
> Now I think I am a bit confused as it relates to what is actually needed
> these days for a set-up with the i915 driver being used on the host side. If I understand
> correctly, the patch is no longer needed if OVMF is being used correct?
>
>
>
> In addition, I noticed that I’m getting some DMA Write messages but I thought
> adding “vfio_iommu_type1.allow_unsafe_interrupts=1“ to modprobe.d would address
> it. Anyways, what am I overlooking here guys? Thanks is advance.
>
>
>
> Additional Info:# cat /proc/cmdline
> BOOT_IMAGE=/vmlinuz-4.1.15-gentoo-r1 root=UUID=b8357358-a71d-4250-83e5-5505e8ad5eba ro linux intel_iommu=on
>
> IOMMU group 1
> 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)
> 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX 980 Ti] [10de:17c8] (rev a1)
> 01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fb0] (rev a1)
>
> # cat /etc/modprobe.d/vfio.conf
> options vfio-pci ids=8086:0151,10de:17c8,10de:0fb0
Do NOT add the root port here, it needs to be bound to the default
driver which will be pcieport. If the bridge is not enabled we can't
talk to the card behind it. I'm not sure if this is actually taking
affect though since vfio-pci in your kernel shouldn't bind to it.
> options vfio-pci disable_vga=1
>
> # cat /etc/modprobe.d/kvm.conf
> # if vfio-pci was built as a module ( default on arch & ubuntu )
> options vfio_iommu_type1 allow_unsafe_interrupts=1
You don't need this, it has nothing to do with the DMA write error
you're seeing. The only time to use this is if the kernel specific
tells you to, and then only if you opt-in to the lesser isolation it
provides.
>
> # Some applications like Passmark Performance Test and SiSoftware Sandra crash the VM without this:
> options kvm ignore_msrs=1
>
> # dmesg | grep -i vfio
> [ 3.895580] VFIO - User Level meta-driver version: 0.3
> [ 3.920197] vfio_pci: add [8086:0151[ffff:ffff]] class 0x000000/00000000
> [ 3.930899] vfio_pci: add [10de:17c8[ffff:ffff]] class 0x000000/00000000
> [ 3.941936] vfio_pci: add [10de:0fb0[ffff:ffff]] class 0x000000/00000000
> [ 422.613903] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1e at 0x258
> [ 422.613919] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19 at 0x900
> [ 1953.837834] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1e at 0x258
> [ 1953.837843] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19 at 0x900
>
> # dmesg -T | grep -e DMAR -e IOMMU -e hardware
> [Thu May 5 21:40:04 2016] ACPI: DMAR 0x00000000C8972760 0000B8 (v01 INTEL SNB 00000001 INTL 00000001)
> [Thu May 5 21:40:04 2016] Intel-IOMMU: enabled
> [Thu May 5 21:40:04 2016] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
> [Thu May 5 21:40:04 2016] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
> [Thu May 5 21:40:04 2016] DMAR: No ATSR found
> [Thu May 5 21:40:04 2016] IOMMU: dmar0 using Queued invalidation
> [Thu May 5 21:40:04 2016] IOMMU: dmar1 using Queued invalidation
> [Thu May 5 21:40:04 2016] IOMMU: Setting RMRR:
> [Thu May 5 21:40:04 2016] IOMMU: Setting identity map for device 0000:00:02.0 [0xcb000000 - 0xcf1fffff]
> [Thu May 5 21:40:04 2016] IOMMU: Setting identity map for device 0000:00:14.0 [0xc88e5000 - 0xc88f6fff]
> [Thu May 5 21:40:04 2016] IOMMU: Setting identity map for device 0000:00:1a.0 [0xc88e5000 - 0xc88f6fff]
> [Thu May 5 21:40:04 2016] IOMMU: Setting identity map for device 0000:00:1d.0 [0xc88e5000 - 0xc88f6fff]
> [Thu May 5 21:40:04 2016] IOMMU: Prepare 0-16MiB unity mapping for LPC
> [Thu May 5 21:40:04 2016] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
> [Thu May 5 21:40:04 2016] [drm] DMAR active, disabling use of stolen memory
> [Thu May 5 21:40:05 2016] dmar: DMAR:[DMA Write] Request device [00:02.0] fault addr 29ff005000
> DMAR:[fault reason 05] PTE Write access is not set
> [Thu May 5 21:40:37 2016] dmar: DMAR:[DMA Write] Request device [00:02.0] fault addr 29ff005000
> DMAR:[fault reason 05] PTE Write access is not set
> [Thu May 5 21:40:37 2016] dmar: DMAR:[DMA Write] Request device [00:02.0] fault addr 29ff005000
> DMAR:[fault reason 05] PTE Write access is not set
> [Thu May 5 21:40:37 2016] dmar: DMAR:[DMA Write] Request device [00:02.0] fault addr 29ff005000
> DMAR:[fault reason 05] PTE Write access is not set
> [Thu May 5 21:40:37 2016] dmar: DMAR:[DMA Write] Request device [00:02.0] fault addr 29ff005000
> DMAR:[fault reason 05] PTE Write access is not set
> [Thu May 5 21:40:38 2016] dmar: DMAR:[DMA Write] Request device [00:02.0] fault addr 29ff005000
> DMAR:[fault reason 05] PTE Write access is not set
> [Thu May 5 21:40:38 2016] dmar: DMAR:[DMA Write] Request device [00:02.0] fault addr 29ff005000
> DMAR:[fault reason 05] PTE Write access is not set
> [Thu May 5 21:40:38 2016] dmar: DMAR:[DMA Write] Request device [00:02.0] fault addr 29ff005000
> DMAR:[fault reason 05] PTE Write access is not set
> [Thu May 5 21:40:38 2016] dmar: DMAR:[DMA Write] Request device [00:02.0] fault addr 29ff005000
> DMAR:[fault reason 05] PTE Write access is not set
> [Thu May 5 21:40:38 2016] dmar: DMAR:[DMA Write] Request device [00:02.0] fault addr 29ff005000
> DMAR:[fault reason 05] PTE Write access is not set
> [Thu May 5 21:40:38 2016] dmar: DMAR:[DMA Write] Request device [00:02.0] fault addr 29ff005000
> DMAR:[fault reason 05] PTE Write access is not set
> [Thu May 5 21:40:40 2016] dmar: DMAR:[DMA Write] Request device [00:02.0] fault addr 29ff005000
> DMAR:[fault reason 05] PTE Write access is not set
> [Thu May 5 21:40:40 2016] dmar: DMAR:[DMA Write] Request device [00:02.0] fault addr 29ff005000
> DMAR:[fault reason 05] PTE Write access is not set
> [Thu May 5 21:40:41 2016] dmar: DMAR:[DMA Write] Request device [00:02.0] fault addr 29ff005000
> DMAR:[fault reason 05] PTE Write access is not set
This is IGD going whacko, is this just a result of enabling the IOMMU
or does it have some relation to the VM starting? Have you tried
intel_iommu=on,igfx_off? It might be a bug in your host firmware not
enabling RMRR entries for stolen memory. iommu=pt may also resolve it.
> MY XML:
> <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
> <name>win8.1-pro</name>
> <uuid>39f1e987-667c-43ff-8641-e10f017d370c</uuid>
> <memory unit='KiB'>6291456</memory>
> <currentMemory unit='KiB'>6291456</currentMemory>
> <vcpu placement='static'>4</vcpu>
> <cputune>
> <vcpupin vcpu='0' cpuset='2'/>
> <vcpupin vcpu='1' cpuset='3'/>
> <vcpupin vcpu='2' cpuset='6'/>
> <vcpupin vcpu='3' cpuset='7'/>
> </cputune>
> <os>
> <type arch='x86_64' machine='pc-i440fx-2.5'>hvm</type>
> <loader readonly='yes' type='pflash'>/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
> <nvram>/var/lib/libvirt/qemu/nvram/win8.1-pro_VARS.fd</nvram>
> <bootmenu enable='yes'/>
> </os>
> <features>
> <acpi/>
> <apic/>
> <hyperv>
> <relaxed state='off'/>
> <vapic state='off'/>
> <spinlocks state='off'/>
> </hyperv>
> <kvm>
> <hidden state='on'/>
> </kvm>
> <vmport state='off'/>
> </features>
> <cpu mode='host-passthrough'>
> <topology sockets='1' cores='2' threads='2'/>
> </cpu>
> <clock offset='localtime'>
> <timer name='rtc' tickpolicy='catchup'/>
> <timer name='pit' tickpolicy='delay'/>
> <timer name='hpet' present='no'/>
> </clock>
> <on_poweroff>destroy</on_poweroff>
> <on_reboot>restart</on_reboot>
> <on_crash>restart</on_crash>
> <pm>
> <suspend-to-mem enabled='no'/>
> <suspend-to-disk enabled='no'/>
> </pm>
> <devices>
> <emulator>/usr/bin/qemu-system-x86_64</emulator>
> <disk type='file' device='disk'>
> <driver name='qemu' type='raw'/>
> <source file='/mnt/new_root/home/sheldon/VMs/win8.1-pro'/>
> <target dev='vda' bus='virtio'/>
> <boot order='1'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
> </disk>
> <controller type='usb' index='0' model='ich9-ehci1'>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/>
> </controller>
> <controller type='usb' index='0' model='ich9-uhci1'>
> <master startport='0'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/>
> </controller>
> <controller type='usb' index='0' model='ich9-uhci2'>
> <master startport='2'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
> </controller>
> <controller type='usb' index='0' model='ich9-uhci3'>
> <master startport='4'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/>
> </controller>
> <controller type='pci' index='0' model='pci-root'/>
> <interface type='direct'>
> <mac address='52:54:00:7e:e3:8a'/>
> <source dev='enp3s0' mode='bridge'/>
> <model type='virtio'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
> </interface>
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
> </hostdev>
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
> </hostdev>
> <memballoon model='virtio'>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
> </memballoon>
> </devices>
> <qemu:commandline>
> <qemu:arg value='-drive'/>
> <qemu:arg value='if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd'/>
> </qemu:commandline>
Ack, get rid of this.
> </domain>
>
My only suspicion is the misuse of that root port by trying to bind it
to vfio-pci. Undo that, if it still doesn't work send 'sudo lspci
-vvv' for 00:01.0, and 01:
More information about the vfio-users
mailing list