[vfio-users] IOMMU group is not a symlink - SR-IOV on 32Bit MMIO platform?

[Taiidan at gmx.com]

On 07/05/2017 02:11 PM, Alex Williamson wrote:

> On Wed, 5 Jul 2017 02:33:52 -0400
> "Taiidan at gmx.com" <Taiidan at gmx.com> wrote:
>
>> error: internal error: Process exited prior to exec: libvirt:  error :
>> internal error: Invalid device 0000:09:00.1 iommu_group file
>> /sys/bus/pci/devices/0000:09:00.1/iommu_group is not a symlink
>>
>> (intel 82576 quad port nic, assigning itself works fine)
>>
>> I am trying to get sr-iov working on a platform with 32bit MMIO space
>> (is that even possible)
>>
>> So I have added pci-realloc and pci-assign-buses to the kernel command
>> line, without that I get a "Not enough MMIO resources for SR-IOV" error
>> even when adding just one VF via sysfs.
> This is not really a 32-bit vs 64-bit issue, it just means your BIOS
> didn't allocate enough resources on the bus to enable SR-IOV.  The
> 82576 dual-port cards typically get away with working on non-SR-IOV
> aware systems because the additional resources they need for SR-IOV
> fits within the minimum bridge apertures anyway.  This is not true for
> the quad-port card or various other SR-IOV devices.
I was told that coreboot's 32bit MMIO space means that you have issues 
with too many devices.
Do you have any firmware implementation guides for SR-IOV that you can 
share? my chipset supports it and ARI.

Does this also apply to newer devices like the i350?
> pci-realloc should help with this, whether pci-assign-buses is
> necessary would depend on the SR-IOV config of the device (82576
> typically doesn't need an additional bus number).
I get an error saying invalid bus if I don't do assign buses.
> Hmm, I don't understand why they wouldn't have an iommu group
> associated with them.  The quad-port 82576 cards had some special kind
> of brokenness about them though that I can't recall, perhaps something
> about ARI.  Having no iommu group would imply that the devices don't
> live downstream of an iommu.  Is this an Intel system?  The DMAR ACPI
> table on Intel has path structures designed to take bus re-numbering
> into account, but maybe you're not on Intel or maybe the BIOS has done
> something particularly awful to negate this.
It is an AMD Opteron G34 Bulldozer, KGPE-D16 board running a libre 
coreboot.
I can forward other devices including GPU's and the cards PF's.

The PF's have a group, but the VF's are not assigned to any group at all.

Is ARI support advertised on the PCI-e chipset like it is advertised on 
the NIC? or is it just ARIFwd+ in DevCap?
>> Would disabling devices in the BIOS help?
> Probably not.  Logs please.  dmesg, sudo lspci -vvv, /tmp/DMAR.dsl
> after running:
>
> # iasl -d -p /tmp/DMAR /sys/firmware/acpi/tables/DMAR
>
> (assuming an Intel system).  Also:
>
> # find /sys/class/iommu/*/ -type l
>
> And
>
> # find /sys/kernel/iommu_groups/ -type l
>
> Thanks,
>
> Alex
Stuff attached! (in off list email as I don't know if the lists supports it)
Opteron AMD-Vi so no DMAR.

Thank you! it is always nice to see highly skilled people helping others.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20170706/6e534a51/attachment.htm>