[vfio-users] SRIOV & amdgpu

[Alex Williamson]

On Wed, 19 Oct 2016 16:03:59 +0200
Ward Poelmans <ward.poelmans at ugent.be> wrote:

> Hi,
> 
> Does anyone here have experience with using SRIOV on the new AMD FirePro
> S series? We are trying it using a FirePRO S7150 which should support up
> to 16 VF using SRIOV [1]. The hypervisor is running Fedora 25 beta with
> kernel 4.9rc1. We want to use qemu/kvm (instead of vmware).
> 
> If we look with lspci, we get:
> Capabilities: [330 v1] Single Root I/O Virtualization (SR-IOV)
>  IOVCap:	Migration-, Interrupt Message Number: 000
>  IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy+
>  IOVSta:	Migration-
>  Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Dependency
> Link: 00
>  VF offset: 256, stride: 1, Device ID: 692f
>  Supported Page Size: 00000553, System Page Size: 00000001
>  Region 0: Memory at 000003bee0000000 (64-bit, prefetchable)
>  Region 2: Memory at 000003bff0000000 (64-bit, prefetchable)
>  Region 5: Memory at 91c00000 (32-bit, non-prefetchable)
>  VF Migration: offset: 00000000, BIR: 0
> 
> So, it's there but not enabled...
> 
> I've looked at the AMDGPU module and I think it checks a table in the
> vbios to determine if it can do SRIOV. Upon testing it, the flag 'vBIOS
> is sr-iov ready' is set but the flag 'sr-iov is enabled on this GPU' is not.
> 
> Anybody any idea how to activate this? It works with vmware, so I should
> be possible to do it with qemu too.

Assuming they've implemented standard SR-IOV support in Linux, you
should find several sriov files in /sys/bus/pci/devices/$ADDRESS/,
namely srio_numvfs and sriov_totalvfs.  totalvfs should match the
SR-IOV capability and report 16, numvfs will report 0 and you can echo
values into numvfs to set the number of enabled VFs.  Note that due to
the way SR-IOV works, it is not possible change from one non-zero
number of VFs to another, for example adding one if VFs are already
enabled.  numvfs must be set to zero before any new non-zero value can
be set.  Let us know what you find, I haven't handled these cards
myself nor heard of users who have.  Thanks,

Alex