[vfio-users] Only 8 devices functional on i7 system

[Alex Williamson]

On Sat, 20 Jun 2020 09:46:00 +0200
Kjeld Borch Egevang <kjeld.egevang at intel.com> wrote:

> Hi VFIO users,
> 
> I am working with a PCIe card that supports up to 255 VFs. In order to 
> get the SR-IOV stuff to work I added some of the latest patches to the 
> vfio-pci driver.
> 
> I have two servers.
> 
> The first one is a Centos 8 based system with many cores and a Xeon E5 
> CPU. I enable VFIO with:
> 
> insmod /lib/modules/$(uname -r)/kernel/drivers/vfio/vfio.ko.xz
> insmod /lib/modules/$(uname -r)/kernel/drivers/vfio/vfio_iommu_type1.ko.xz
> insmod /lib/modules/$(uname -r)/kernel/drivers/vfio/vfio_virqfd.ko.xz
> insmod /root/vfio-pci.ko enable_sriov=1 # The patched module
> echo "$vendor $device" > /sys/bus/pci/drivers/vfio-pci/new_id
> echo 15 > /sys/bus/pci/drivers/vfio-pci/*/sriov_numvfs
> 
> This works and I get a PF and 15 fully functional VFs.
> 
> My other server is also a Centos 8 based system but with an i7 CPU. The 
> motherboard is an Asrock Z370M Pro 4.
> 
> When I enable VFIO only the PF and the first 7 VFs are fully functional. 
> If I enter "lspci -s 01:00.1 -x" I get a nice dump of the config space 
> for VF0. But if I enter "lspci -s 01:01.0 -x" I get:
> 
> 00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> 10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> 20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> 30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> 
> My guess is that this is either due to a limitation in the PCI bridge or 
> a driver issue. What is causing this problem?

There might be an SR-IOV option in the BIOS that should be turned on,
but in general I'd expect consumer boards are pretty bad about
allocating bridge resources for SR-IOV.  This might be all that the
system can support under minimum apertures.  Add the following to the
kernel command line to have Linux do the SR-IOV allocation itself:

    pci=realloc,assign-busses

Thanks,

Alex