[vfio-users] IOMMU group is not a symlink - SR-IOV on 32Bit MMIO platform?

[Alex Williamson]

On Wed, 5 Jul 2017 02:33:52 -0400
"Taiidan at gmx.com" <Taiidan at gmx.com> wrote:

> error: internal error: Process exited prior to exec: libvirt:  error : 
> internal error: Invalid device 0000:09:00.1 iommu_group file 
> /sys/bus/pci/devices/0000:09:00.1/iommu_group is not a symlink
> 
> (intel 82576 quad port nic, assigning itself works fine)
> 
> I am trying to get sr-iov working on a platform with 32bit MMIO space 
> (is that even possible)
> 
> So I have added pci-realloc and pci-assign-buses to the kernel command 
> line, without that I get a "Not enough MMIO resources for SR-IOV" error 
> even when adding just one VF via sysfs.

This is not really a 32-bit vs 64-bit issue, it just means your BIOS
didn't allocate enough resources on the bus to enable SR-IOV.  The
82576 dual-port cards typically get away with working on non-SR-IOV
aware systems because the additional resources they need for SR-IOV
fits within the minimum bridge apertures anyway.  This is not true for
the quad-port card or various other SR-IOV devices.

pci-realloc should help with this, whether pci-assign-buses is
necessary would depend on the SR-IOV config of the device (82576
typically doesn't need an additional bus number).
 
> I also get an error about not enough space for BAR's in dmesg with or 
> without the cmdline additions but with the cmdline parms the vf bars 
> seem to allocate fine.
> 
> [    2.705747] pci 0000:07:00.0: VF(n) BAR0 space: [mem 
> 0x00000000-0x0001ffff 64bit] (contains BAR0 for 8 VFs)
> [    2.715887] pci 0000:07:00.0: VF(n) BAR3 space: [mem 
> 0x00000000-0x0001ffff 64bit] (contains BAR3 for 8 VFs)
> [    2.726342] pci 0000:07:00.1: VF(n) BAR0 space: [mem 
> 0x00000000-0x0001ffff 64bit] (contains BAR0 for 8 VFs)
> [    2.736477] pci 0000:07:00.1: VF(n) BAR3 space: [mem 
> 0x00000000-0x0001ffff 64bit] (contains BAR3 for 8 VFs)
> [    2.763896] pci 0000:09:00.0: VF(n) BAR0 space: [mem 
> 0x00000000-0x0001ffff 64bit] (contains BAR0 for 8 VFs)
> [    2.774036] pci 0000:09:00.0: VF(n) BAR3 space: [mem 
> 0x00000000-0x0001ffff 64bit] (contains BAR3 for 8 VFs)
> [    2.784487] pci 0000:09:00.1: VF(n) BAR0 space: [mem 
> 0x00000000-0x0001ffff 64bit] (contains BAR0 for 8 VFs)
> [    2.794621] pci 0000:09:00.1: VF(n) BAR3 space: [mem 
> 0x00000000-0x0001ffff 64bit] (contains BAR3 for 8 VFs)
> 
> I get this message and see the VF's in lspci but there isn't any IOMMU 
> group allocated.
> [  116.175246] igb 0000:07:00.0: 1 VFs allocated
> [  420.514477] igb 0000:09:00.1: 1 VFs allocated

Hmm, I don't understand why they wouldn't have an iommu group
associated with them.  The quad-port 82576 cards had some special kind
of brokenness about them though that I can't recall, perhaps something
about ARI.  Having no iommu group would imply that the devices don't
live downstream of an iommu.  Is this an Intel system?  The DMAR ACPI
table on Intel has path structures designed to take bus re-numbering
into account, but maybe you're not on Intel or maybe the BIOS has done
something particularly awful to negate this.

> Would disabling devices in the BIOS help?

Probably not.  Logs please.  dmesg, sudo lspci -vvv, /tmp/DMAR.dsl
after running:

# iasl -d -p /tmp/DMAR /sys/firmware/acpi/tables/DMAR

(assuming an Intel system).  Also:

# find  /sys/class/iommu/*/ -type l

And

# find /sys/kernel/iommu_groups/ -type l

Thanks,

Alex