[vfio-users] Fwd: failing IGD passthrough on apollo lake - BAR 2 error

Thu Jul 27 15:41:04 UTC 2017

On Thu, 27 Jul 2017 11:32:24 +0200
Geert Coulommier <g.coulommier at gmail.com> wrote:

> Hi,
> 
> so I've tried the 2 options you suggested:
> 
> 1) "look in /proc/iomem and identify the driver that's still claiming
> portions of IGD and disable it"
> 
> from /proc/iomem:
> 
> ...
> 80000000-cfffffff : PCI Bus 0000:00
>   80000000-8fffffff : 0000:00:02.0
>     80000000-808cffff : efifb
> ...
> 
> which is strange as to prevent this, the part "video=efifb:off" was added
> to grub:
> 
> GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on iommu=pt
> rd.driver.pre=vfio-pci video=vesafb:off,efifb:off"
> 
> Because I'm running the host on uefi, and to keeps things clean, I removed
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This is probably the next piece of the puzzle...

> the "vesafb:off"-part:
> GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on iommu=pt
> rd.driver.pre=vfio-pci video=efifb:off"
> 
> Unexpectedly, this seemed have an effect. Now from /proc/iomem (when not
> running the VM,full printout of /proc/iomem below in [1]):
> 
> ...
> 80000000-cfffffff : PCI Bus 0000:00
>   80000000-8fffffff : 0000:00:02.0
>   90000000-90ffffff : 0000:00:02.0
>   91000000-910fffff : 0000:00:0e.0
>   91100000-911fffff : PCI Bus 0000:03
>     91100000-911001ff : 0000:03:00.0
>       91100000-911001ff : ahci
> ...
> 
> when running the VM, it goes to:
> 
> ...
> 80000000-cfffffff : PCI Bus 0000:00
>   80000000-8fffffff : 0000:00:02.0
>     80000000-8fffffff : vfio-pci
>   90000000-90ffffff : 0000:00:02.0
>     90000000-90ffffff : vfio-pci
>   91000000-910fffff : 0000:00:0e.0
>     91000000-910fffff : vfio-pci
>   91100000-911fffff : PCI Bus 0000:03
>     91100000-911001ff : 0000:03:00.0
>       91100000-911001ff : ahci
>   91200000-912fffff : PCI Bus 0000:01
>     91200000-91203fff : 0000:01:00.0
>       91200000-91203fff : r8169
>     91204000-91204fff : 0000:01:00.0
>       91204000-91204fff : r8169
>   91300000-9130ffff : 0000:00:15.0
>     91300000-9130ffff : xhci-hcd
>   91310000-91313fff : 0000:00:0e.0
>     91310000-91313fff : vfio-pci
>   91314000-91315fff : 0000:00:12.0
>     91314000-91315fff : ahci
>   91316000-913160ff : 0000:00:1f.1
>   91317000-913177ff : 0000:00:12.0
>     91317000-913177ff : ahci
>   91318000-913180ff : 0000:00:12.0
>     91318000-913180ff : ahci
>   9131b000-9131bfff : 0000:00:0f.0
>     9131b000-9131bfff : mei_me
> ...
> 
> 
> and the dmesg log:
> 
> dmesg | grep -aiE '((DMAR)|(kvm)|(drm)|(Command line)|(iommu)|(vfio))'
> [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.12.3-041203-generic
> root=/dev/mapper/granada--vg-root ro quiet splash intel_iommu=on iommu=pt
> rd.driver.pre=vfio-pci video=efifb:off vt.handoff=7
> [    0.000000] ACPI: DMAR 0x000000006D9D0470 0000A8 (v01 INTEL  EDK2
> 00000003 BRXT 0100000D)
> [    0.000000] Kernel command line:
> BOOT_IMAGE=/boot/vmlinuz-4.12.3-041203-generic
> root=/dev/mapper/granada--vg-root ro quiet splash intel_iommu=on iommu=pt
> rd.driver.pre=vfio-pci video=efifb:off vt.handoff=7
> [    0.000000] DMAR: IOMMU enabled
> [    0.044107] DMAR: Host address width 39
> [    0.044109] DMAR: DRHD base: 0x000000fed64000 flags: 0x0
> [    0.044126] DMAR: dmar0: reg_base_addr fed64000 ver 1:0 cap
> 1c0000c40660462 ecap 7e3ff0505e
> [    0.044128] DMAR: DRHD base: 0x000000fed65000 flags: 0x1
> [    0.044139] DMAR: dmar1: reg_base_addr fed65000 ver 1:0 cap
> d2008c40660462 ecap f050da
> [    0.044142] DMAR: RMRR base: 0x0000006d5af000 end: 0x0000006d5cefff
> [    0.044145] DMAR: RMRR base: 0x0000006f800000 end: 0x0000007fffffff
> [    0.044148] DMAR-IR: IOAPIC id 1 under DRHD base  0xfed65000 IOMMU 1
> [    0.044150] DMAR-IR: HPET id 0 under DRHD base 0xfed65000
> [    0.044152] DMAR-IR: Queued invalidation will be enabled to support
> x2apic and Intr-remapping.
> [    0.046253] DMAR-IR: Enabled IRQ remapping in x2apic mode
> [    1.794596] DMAR: No ATSR found
> [    1.795685] DMAR: dmar0: Using Queued invalidation
> [    1.795694] DMAR: dmar1: Using Queued invalidation
> [    1.795872] DMAR: Hardware identity mapping for device 0000:00:00.0
> [    1.795882] DMAR: Hardware identity mapping for device 0000:00:02.0
> [    1.795886] DMAR: Hardware identity mapping for device 0000:00:0e.0
> [    1.795888] DMAR: Hardware identity mapping for device 0000:00:0f.0
> [    1.795890] DMAR: Hardware identity mapping for device 0000:00:12.0
> [    1.795892] DMAR: Hardware identity mapping for device 0000:00:13.0
> [    1.795895] DMAR: Hardware identity mapping for device 0000:00:13.1
> [    1.795897] DMAR: Hardware identity mapping for device 0000:00:13.2
> [    1.795899] DMAR: Hardware identity mapping for device 0000:00:13.3
> [    1.795902] DMAR: Hardware identity mapping for device 0000:00:15.0
> [    1.795904] DMAR: Hardware identity mapping for device 0000:00:1f.0
> [    1.795906] DMAR: Hardware identity mapping for device 0000:00:1f.1
> [    1.795911] DMAR: Hardware identity mapping for device 0000:01:00.0
> [    1.795916] DMAR: Hardware identity mapping for device 0000:03:00.0
> [    1.795917] DMAR: Setting RMRR:
> [    1.795920] DMAR: Ignoring identity map for HW passthrough device
> 0000:00:02.0 [0x6f800000 - 0x7fffffff]
> [    1.795922] DMAR: Ignoring identity map for HW passthrough device
> 0000:00:15.0 [0x6d5af000 - 0x6d5cefff]
> [    1.795924] DMAR: Prepare 0-16MiB unity mapping for LPC
> [    1.795926] DMAR: Ignoring identity map for HW passthrough device
> 0000:00:1f.0 [0x0 - 0xffffff]
> [    1.795954] DMAR: Intel(R) Virtualization Technology for Directed I/O
> [    1.796125] iommu: Adding device 0000:00:00.0 to group 0
> [    1.796140] iommu: Adding device 0000:00:02.0 to group 1
> [    1.796157] iommu: Adding device 0000:00:0e.0 to group 2
> [    1.796174] iommu: Adding device 0000:00:0f.0 to group 3
> [    1.796187] iommu: Adding device 0000:00:12.0 to group 4
> [    1.796229] iommu: Adding device 0000:00:13.0 to group 5
> [    1.796254] iommu: Adding device 0000:00:13.1 to group 5
> [    1.796271] iommu: Adding device 0000:00:13.2 to group 5
> [    1.796288] iommu: Adding device 0000:00:13.3 to group 5
> [    1.796317] iommu: Adding device 0000:00:15.0 to group 6
> [    1.796338] iommu: Adding device 0000:00:1f.0 to group 7
> [    1.796350] iommu: Adding device 0000:00:1f.1 to group 7
> [    1.796361] iommu: Adding device 0000:01:00.0 to group 5
> [    1.796371] iommu: Adding device 0000:03:00.0 to group 5
> [    2.512432] ata1.00: supports DRM functions and may not be fully
> accessible
> [    2.514160] ata1.00: supports DRM functions and may not be fully
> accessible
> [    3.124755] VFIO - User Level meta-driver version: 0.3
> [    3.137417] vfio-pci 0000:00:02.0: vgaarb: changed VGA decodes:
> olddecodes=io+mem,decodes=io+mem:owns=io+mem
> [    3.156196] vfio_pci: add [8086:5a85[ffff:ffff]] class 0x000000/00000000
> [    3.176202] vfio_pci: add [8086:5a98[ffff:ffff]] class 0x000000/00000000
> 
> with these entries added when running the VM:
> 
> [   49.439866] vfio_cap_init: 0000:00:0e.0 pci config conflict @0x80, was
> cap 0x9 now cap 0x10
> [   49.439869] vfio_cap_init: 0000:00:0e.0 pci config conflict @0x81, was
> cap 0x9 now cap 0x10
> [   49.439871] vfio_cap_init: 0000:00:0e.0 pci config conflict @0x82, was
> cap 0x9 now cap 0x10
> [   49.439873] vfio_cap_init: 0000:00:0e.0 pci config conflict @0x83, was
> cap 0x9 now cap 0x10
> [   49.442695] DMAR: DRHD: handling fault status reg 3
> [   49.442710] DMAR: [DMA Write] Request device [00:02.0] fault addr 0
> [fault reason 02] Present bit in context entry is clear
> [   49.567831] vfio_ecap_init: 0000:00:02.0 hiding ecap 0x1b at 0x100
> 
> Passthrough still doesn't work though, and the last two lines in the kernel
> log seem to hint at that. So from option one to option 2:

Nope, those are normal.

> 2) "don't blacklist i915, let the kernel boot with it, then do a 'virsh
> nodedev-detach pci_0000_00_02_0' at boot before starting the VM so that
> you're not binding it back to i915 after every instance of running the
> VM."
> 
> So I unblacklisted i915 an executed 'virsh nodedev-dettach
> pci_0000_00_02_0':
> 
> virsh nodedev-dettach pci_0000_00_02_0
> Device pci_0000_00_02_0 detached
> 
> Then ran the VM. Unfortunately, results are the same, as are the log
> entries in the kernel log (see above).
> 
> When running the same virsh 'nodedev-dettach pci_0000_00_02_0' command when
> running the VM, I get:
> 
> virsh nodedev-dettach pci_0000_00_02_0
> error: Failed to detach device pci_0000_00_02_0
> error: Requested operation is not valid: PCI device 0000:00:02.0 is in use
> by driver QEMU, domain ubuntu16.04_desktop
> 
> So it does seem to be attached to the VM correctly.
> 
> Maybe interesting observation: when the host boots, the screen shows grub
> and then goes black but stays powered on. When I launch the VM, the screen
> stays black but powers off.

I think either mechanism above is equally effective and the remaining
pieces is likely that there's no VGA BIOS to initialize the graphics
because the host is running in UEFI mode.  To solve this, burn some
sort of Linux live CD/DVD image and temporarily set the host BIOS to
boot into CSM/legacy mode to that live image.  Sometimes you'll be able
to select between a UEFI or legacy mode booting the CD or you might be
able to prioritize legacy over UEFI, it depends on the system.  Once
you've booted the image, dump the ROM for the IGD to a file and copy it
somewhere that you'll be able to get to it later.  Undo any settings
for booting the image and then look at my rom-fixer utility:

https://github.com/awilliam/rom-parser

Run that on the ROM and then add a <rom file='/path/to/vga.rom'/> to
the IGD hostdev entry in the xml.

> Finally, until now I had ignored the errors in the kernel log on the audio
> device (0000:00:0e.0) as I was focusing on the gpu. As requested, in [2]
> the output of 'sudo lspci -xxxxs 0000:00:0e.0'.
> 
Thanks, I'll take a look at this.

Alex