[vfio-users] IOMMU group is not a symlink - SR-IOV on 32Bit MMIO platform?

[Andre Richter]

Alex Williamson <alex.williamson at redhat.com> schrieb am Mi., 5. Juli 2017
um 20:11 Uhr:

> On Wed, 5 Jul 2017 02:33:52 -0400
> "Taiidan at gmx.com" <Taiidan at gmx.com> wrote:
>
> > error: internal error: Process exited prior to exec: libvirt:  error :
> > internal error: Invalid device 0000:09:00.1 iommu_group file
> > /sys/bus/pci/devices/0000:09:00.1/iommu_group is not a symlink
> >
> > (intel 82576 quad port nic, assigning itself works fine)
> >
> > I am trying to get sr-iov working on a platform with 32bit MMIO space
> > (is that even possible)
> >
> > So I have added pci-realloc and pci-assign-buses to the kernel command
> > line, without that I get a "Not enough MMIO resources for SR-IOV" error
> > even when adding just one VF via sysfs.
>
> This is not really a 32-bit vs 64-bit issue, it just means your BIOS
> didn't allocate enough resources on the bus to enable SR-IOV.  The
> 82576 dual-port cards typically get away with working on non-SR-IOV
> aware systems because the additional resources they need for SR-IOV
> fits within the minimum bridge apertures anyway.  This is not true for
> the quad-port card or various other SR-IOV devices.
>
> pci-realloc should help with this, whether pci-assign-buses is
> necessary would depend on the SR-IOV config of the device (82576
> typically doesn't need an additional bus number).
>
> > I also get an error about not enough space for BAR's in dmesg with or
> > without the cmdline additions but with the cmdline parms the vf bars
> > seem to allocate fine.
> >
> > [    2.705747] pci 0000:07:00.0: VF(n) BAR0 space: [mem
> > 0x00000000-0x0001ffff 64bit] (contains BAR0 for 8 VFs)
> > [    2.715887] pci 0000:07:00.0: VF(n) BAR3 space: [mem
> > 0x00000000-0x0001ffff 64bit] (contains BAR3 for 8 VFs)
> > [    2.726342] pci 0000:07:00.1: VF(n) BAR0 space: [mem
> > 0x00000000-0x0001ffff 64bit] (contains BAR0 for 8 VFs)
> > [    2.736477] pci 0000:07:00.1: VF(n) BAR3 space: [mem
> > 0x00000000-0x0001ffff 64bit] (contains BAR3 for 8 VFs)
> > [    2.763896] pci 0000:09:00.0: VF(n) BAR0 space: [mem
> > 0x00000000-0x0001ffff 64bit] (contains BAR0 for 8 VFs)
> > [    2.774036] pci 0000:09:00.0: VF(n) BAR3 space: [mem
> > 0x00000000-0x0001ffff 64bit] (contains BAR3 for 8 VFs)
> > [    2.784487] pci 0000:09:00.1: VF(n) BAR0 space: [mem
> > 0x00000000-0x0001ffff 64bit] (contains BAR0 for 8 VFs)
> > [    2.794621] pci 0000:09:00.1: VF(n) BAR3 space: [mem
> > 0x00000000-0x0001ffff 64bit] (contains BAR3 for 8 VFs)
> >
> > I get this message and see the VF's in lspci but there isn't any IOMMU
> > group allocated.
> > [  116.175246] igb 0000:07:00.0: 1 VFs allocated
> > [  420.514477] igb 0000:09:00.1: 1 VFs allocated
>
> Hmm, I don't understand why they wouldn't have an iommu group
> associated with them.  The quad-port 82576 cards had some special kind
> of brokenness about them though that I can't recall, perhaps something
> about ARI.


I vaguely remember that the 82576 can work with both. If it discovers ARI
support, it uses ARI, if it is not there, it uses traditional bus numbers.


> Having no iommu group would imply that the devices don't
> live downstream of an iommu.  Is this an Intel system?  The DMAR ACPI
> table on Intel has path structures designed to take bus re-numbering
> into account, but maybe you're not on Intel or maybe the BIOS has done
> something particularly awful to negate this.
>
> > Would disabling devices in the BIOS help?
>
> Probably not.  Logs please.  dmesg, sudo lspci -vvv, /tmp/DMAR.dsl
> after running:
>
> # iasl -d -p /tmp/DMAR /sys/firmware/acpi/tables/DMAR
>
> (assuming an Intel system).  Also:
>
> # find  /sys/class/iommu/*/ -type l
>
> And
>
> # find /sys/kernel/iommu_groups/ -type l
>
> Thanks,
>
> Alex
>
> _______________________________________________
> vfio-users mailing list
> vfio-users at redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20170706/76483819/attachment.htm>