[vfio-users] Passthru problems with a Vega GPU

[Alex Williamson]

On Thu, 30 May 2019 19:24:03 +0100
James Courtier-Dutton <james.dutton at gmail.com> wrote:

> On Thu, 30 May 2019 at 18:54, James Courtier-Dutton <james.dutton at gmail.com>
> wrote:
> 
> > lspci -vvv on host:
> > 43:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI]
> > Vega 10 XL/XT [Radeon RX Vega 56/64] (rev c3) (prog-if 00 [VGA controller])
> >         Subsystem: ASUSTeK Computer Inc. Vega 10 XT [Radeon RX Vega 64]
> >         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> > ParErr- Stepping- SERR+ FastB2B- DisINTx+
> >         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> > <TAbort- <MAbort- >SERR- <PERR- INTx-
> >         Latency: 0, Cache Line Size: 64 bytes
> >         Interrupt: pin A routed to IRQ 107
> >         NUMA node: 1
> >         Region 0: Memory at 80000000 (64-bit, prefetchable) [size=256M]
> >         Region 2: Memory at 90000000 (64-bit, prefetchable) [size=2M]
> >         Region 4: I/O ports at 3000 [size=256]
> >         Region 5: Memory at 9fe00000 (32-bit, non-prefetchable) [size=512K]
> >         Expansion ROM at 9fe80000 [disabled] [size=128K]
> >
> > lspci -vvv on guest VM:
> > 07:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI]
> > Vega 10 XT [Radeon RX Vega 64] (rev c3) (prog-if 00 [VGA controller])
> > Subsystem: ASUSTeK Computer Inc. Vega 10 XL/XT [Radeon RX Vega 56/64]
> > Physical Slot: 0-6
> > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> > Stepping- SERR+ FastB2B- DisINTx+
> > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> > <MAbort- >SERR- <PERR- INTx-
> > Latency: 0, Cache Line Size: 64 bytes
> > Interrupt: pin A routed to IRQ 42
> > Region 0: Memory at 400000000 (64-bit, prefetchable) [size=8G]
> > Region 2: Memory at 300000000 (64-bit, prefetchable) [size=2M]
> > Region 4: I/O ports at d000 [size=256]
> > Region 5: Memory at fdc00000 (32-bit, non-prefetchable) [size=512K]
> > Expansion ROM at fdc80000 [disabled] [size=128K]
> >
> >
> > For "Region 0", why is it "size=8G" in guest, but "size=256M" on host
> >
> > When I try to use the card in the guest, I get errors like:
> > [  137.290586] amdgpu 0000:07:00.0: [gfxhub] no-retry page fault (src_id:0
> > ring:158 vmid:8 pasid:32769, for process  pid 0 thread  pid 0)
> > [  137.292141] amdgpu 0000:07:00.0:   in page starting at address
> > 0x00007f983d952000 from 27
> > [  137.293164] amdgpu 0000:07:00.0:
> > VM_L2_PROTECTION_FAULT_STATUS:0x0080093C
> > [  137.294065] Evicting PASID 32769 queues
> >
> > The same linux kernel 5.1.5 is on the host and the guest.
> >
> > Now, the GPU has 8GB of RAM, but I don't see how the guest memory window
> > can be larger than the host can do.
> >
> > Can anyone help?
> >
> > Kind Regards
> >
> > James
> >
> >
> >  
> Some more info.
> The guest, before the amdgpu driver is loaded:
> Region 0: Memory at d0000000 (64-bit, prefetchable) [size=256M]
> The guest, after the amdgpu driver is loaded:
> Region 0: Memory at 400000000 (64-bit, prefetchable) [size=8G]
> 
> So, the amdgpu driver is trying to expand the window to 8G, and it succeeds
> in the guest, and it re-programs the bridge and the gpu pci windows to
> match.
> But, it does not manage to do the same to the host PCI device.
> 
> Is there any test I can do to force a limit of 256M in the guest?

Hmm, resizable BAR support on the GPU?  (lspci -vvvxxxx)  If it's done
through the PCIe capability, it should be read-only.  Try giving the
QEMU patch below a try for QEMU to entirely hide the capability on the
device.  I'm confused how the guest driver could be activating such a
change, but the window through which we access the device doesn't
change dynamically from this, so it's not surprising that the device
wouldn't work correctly if the driver thinks it has direct access to
that memory.  I haven't personally seen devices with resizeable BARs to
implement the support.  If you load an unload the amdgpu driver on the
host, by chance will it also increase the BAR size and maybe leave it
increased?  Thanks,

Alex

diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
index 8cecb53d5cf6..ac9ef9323ef4 100644
--- a/hw/vfio/pci.c
+++ b/hw/vfio/pci.c
@@ -2118,6 +2118,7 @@ static void vfio_add_ext_cap(VFIOPCIDevice *vdev)
         case 0: /* kernel masked capability */
         case PCI_EXT_CAP_ID_SRIOV: /* Read-only VF BARs confuse OVMF */
         case PCI_EXT_CAP_ID_ARI: /* XXX Needs next function virtualization */
+        case PCI_EXT_CAP_ID_REBAR:
             trace_vfio_add_ext_cap_dropped(vdev->vbasedev.name, cap_id, next);
             break;
         default: