[vfio-users] Only 8 devices functional on i7 system

[Kjeld Borch Egevang]

Hi VFIO users,

I am working with a PCIe card that supports up to 255 VFs. In order to 
get the SR-IOV stuff to work I added some of the latest patches to the 
vfio-pci driver.

I have two servers.

The first one is a Centos 8 based system with many cores and a Xeon E5 
CPU. I enable VFIO with:

insmod /lib/modules/$(uname -r)/kernel/drivers/vfio/vfio.ko.xz
insmod /lib/modules/$(uname -r)/kernel/drivers/vfio/vfio_iommu_type1.ko.xz
insmod /lib/modules/$(uname -r)/kernel/drivers/vfio/vfio_virqfd.ko.xz
insmod /root/vfio-pci.ko enable_sriov=1 # The patched module
echo "$vendor $device" > /sys/bus/pci/drivers/vfio-pci/new_id
echo 15 > /sys/bus/pci/drivers/vfio-pci/*/sriov_numvfs

This works and I get a PF and 15 fully functional VFs.

My other server is also a Centos 8 based system but with an i7 CPU. The 
motherboard is an Asrock Z370M Pro 4.

When I enable VFIO only the PF and the first 7 VFs are fully functional. 
If I enter "lspci -s 01:00.1 -x" I get a nice dump of the config space 
for VF0. But if I enter "lspci -s 01:01.0 -x" I get:

00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

My guess is that this is either due to a limitation in the PCI bridge or 
a driver issue. What is causing this problem?


/Kjeld