[vfio-users] Blank output on passed card. i915 host gtx1060 vm

Thu Apr 13 19:19:55 UTC 2017

On Wed, 12 Apr 2017 22:15:58 +0000
"zChris ." <simpson38 at hotmail.com> wrote:

> Hello,
> 
> 
> Ive been trying to get pci passthrough going for a couple of days now to no success. All i get when i start the vm is a blank screen on the vm and a console on the host.
> 
> 
> Ive been reading these guides: https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF and https://wiki.installgentoo.com/index.php/PCI_passthrough .
> 
> 
> My setup is as follow:
> 
> MSI B75M DGS R2 ( http://www.asrock.com/mb/Intel/B75M-DGS%20R2.0/ )
> 
> GTX 1060 not sure of vendor.
> 
> Im using 1 monitor, dvi connected to GTX1060 and VGA from built in vga output on the motherboard.
> 
> 
> dmesg | grep -i iommu gives:
> 
> 
> [    0.000000] DMAR: IOMMU enabled
> [    0.018085] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
> [    3.368817] iommu: Adding device 0000:00:00.0 to group 0
> [    3.368878] iommu: Adding device 0000:00:01.0 to group 1
> [    3.368935] iommu: Adding device 0000:00:02.0 to group 2
> [    3.368992] iommu: Adding device 0000:00:14.0 to group 3
> [    3.369050] iommu: Adding device 0000:00:16.0 to group 4
> [    3.369106] iommu: Adding device 0000:00:1a.0 to group 5
> [    3.369164] iommu: Adding device 0000:00:1b.0 to group 6
> [    3.369221] iommu: Adding device 0000:00:1c.0 to group 7
> [    3.369278] iommu: Adding device 0000:00:1c.5 to group 8
> [    3.369335] iommu: Adding device 0000:00:1d.0 to group 9
> [    3.369392] iommu: Adding device 0000:00:1e.0 to group 10
> [    3.369453] iommu: Adding device 0000:00:1f.0 to group 11
> [    3.369510] iommu: Adding device 0000:00:1f.2 to group 11
> [    3.369567] iommu: Adding device 0000:00:1f.3 to group 11
> [    3.369621] iommu: Adding device 0000:01:00.0 to group 1
> [    3.369676] iommu: Adding device 0000:01:00.1 to group 1
> [    3.369738] iommu: Adding device 0000:03:00.0 to group 12
> 
> dmesg | grep -i dmar gives:
> 
> [    0.000000] ACPI: DMAR 0x00000000BDEB2E90 0000B8 (v01 INTEL  SNB      00000001 INTL 00000001)
> [    0.000000] DMAR: IOMMU enabled
> [    0.017650] DMAR: Host address width 36
> [    0.017700] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
> [    0.017755] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
> [    0.017836] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
> [    0.017889] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
> [    0.017979] DMAR: RMRR base: 0x000000bdaba000 end: 0x000000bdae6fff
> [    0.018032] DMAR: RMRR base: 0x000000bf800000 end: 0x000000cf9fffff
> [    0.018085] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
> [    0.018138] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
> [    0.018414] DMAR-IR: Enabled IRQ remapping in xapic mode
> [    3.365966] DMAR: No ATSR found
> [    3.366226] DMAR: dmar0: Using Queued invalidation
> [    3.366280] DMAR: dmar1: Using Queued invalidation
> [    3.366640] DMAR: Setting RMRR:
> [    3.366715] DMAR: Setting identity map for device 0000:00:02.0 [0xbf800000 - 0xcf9fffff]
> [    3.367885] DMAR: Setting identity map for device 0000:00:14.0 [0xbdaba000 - 0xbdae6fff]
> [    3.367993] DMAR: Setting identity map for device 0000:00:1a.0 [0xbdaba000 - 0xbdae6fff]
> [    3.368100] DMAR: Setting identity map for device 0000:00:1d.0 [0xbdaba000 - 0xbdae6fff]
> [    3.368188] DMAR: Prepare 0-16MiB unity mapping for LPC
> [    3.368258] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
> [    3.368709] DMAR: Intel(R) Virtualization Technology for Directed I/O
> [   18.498352] [drm] DMAR active, disabling use of stolen memory
> [   18.521007] DMAR: DRHD: handling fault status reg 2
> [   18.521009] DMAR: [DMA Write] Request device [00:02.0] fault addr 7600000000 [fault reason 05] PTE Write access is not set
> [   18.627275] DMAR: DRHD: handling fault status reg 2
> [   18.627277] DMAR: [DMA Write] Request device [00:02.0] fault addr 7600000000 [fault reason 05] PTE Write access is not set
> [   19.753598] DMAR: DRHD: handling fault status reg 2
> [   19.753601] DMAR: [DMA Write] Request device [00:02.0] fault addr 7600000000 [fault reason 05] PTE Write access is not set

Are these in any way time correlated with launching the VM?  I suspect
not since they're happening within 20s of boot.  Almost humorous that
IGD announces it's disabling use of stolen memory then immediately
takes some faults.

> IOMMU groups is as follow:

Typical for Intel desktop processor [snip]

> And my group is IOMMU 1
> 
> lspci -k gives :
> 
> 01:00.0 VGA compatible controller: NVIDIA Corporation Device 1c03 (rev a1)
> Subsystem: ASUSTeK Computer Inc. Device 85ae
> Kernel driver in use: vfio-pci
> Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
> 01:00.1 Audio device: NVIDIA Corporation Device 10f1 (rev a1)
> Subsystem: ASUSTeK Computer Inc. Device 85ae
> Kernel driver in use: vfio-pci
> Kernel modules: snd_hda_intel
> 
> Which tells me that it got binded to vfio ok.
> 
> dmesg | grep -i vgaarb gives:
> 
> [    0.255440] vgaarb: setting as boot device: PCI:0000:00:02.0
> [    0.255492] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
> [    0.255575] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none
> [    0.255655] vgaarb: loaded
> [    0.255703] vgaarb: bridge control possible 0000:01:00.0
> [    0.255754] vgaarb: no bridge control possible 0000:00:02.0
> [    3.398850] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=io+mem:owns=none
> [   18.506506] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=io+mem:owns=none
> [   18.510607] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io+mem
> 
> 
> My Boot param is :
> BOOT_IMAGE=/kernel-genkernel-x86_64-4.9.16-gentoo root=UUID=b63be7e0-a4a3-476b-a43a-9cec033be22b ro video=efifb:off iommu=on intel_iommu=on vfio-pci.ids=10de:1c03,10de:10f1 i915.enable_hd_vgaarb=1

iommu=on doesn't do anything.

The i915 option only works if you've patched your kernel with that (the
patch isn't upstream) and it shouldn't be needed since you're not
using x-vga below.

vfio-pci.ids also only works if vfio-pci gets loaded before any
drivers that would otherwise claim those devices.  This usually
involves making sure that's the case as I've documented for a Fedora
example on vfio.blogspot.com.

> where i bind the GPU and HDMI AUDIO and enable vgaarbfor i915.
> 
> Now i try to start QEMU with:
> 
> qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off \
> -smp 2,sockets=1,cores=2,threads=1 \
> -device vfio-pci,host=01:00.0,multifunction=on \
> -device vfio-pci,host=01:00.1 \
> -drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd \
> -drive if=pflash,format=raw,file=/usr/share/OVMF/OVMF_VARS.fd \
> -vga none
> 
> when i switch over to the other input it just says no signal and swithes back.
> 
> And now i have no idea what to do.

You're trying to use OVMF, so aside from making sure that nouveau
doesn't get the card into a bad state and none of the other low level
framebuffers attach to it and steal resources, the other big question
is whether the ROM on the GPU supports UEFI.  With a 10-series card I'd
hope so, but it's probably worth dumping and testing.

http://vfio.blogspot.com/2014/08/does-my-graphics-card-rom-support-efi.html

Also, which is the primary graphics on the host system, the NVIDIA card
or IGD?  It should be IGD for the easiest chance of getting things
working.  Thanks,

Alex