[vfio-users] PLX switch report a UR when EP tries to DMA to VM's memory

[Wuzongyong (Euler Dept)]

> > Hi,
> >
> > I notice a problem with a PCIe endpoint, which is behind a PLX switch,
> assigned to a VM by VFIO.
> > The problem is switch report a UR error when the EP tries to DMA to a
> memory zone inside VM's address space.
> > Assume that the DMA destination address is between in the VM's ram
> > address space, and unfortunately that address value in host's point of
> view just hit the PLX switch  upstream port BAR0 memory-mapped IO range.
> > In a result, the DMA will failed because SW think this memory request
> > is invalid if the destination address hit its UP's bar.
> > Is this a hardware bug or qemu/seabios doesn't maintain a proper address
> space for VM?
> 
> Upstream switch ports are generally single function devices and therefore
> governed by 6.12.1.3 (PCIe base spec rev 4.0, v1) which indicates an ACS
> capability must not be implemented.  We can therefore read into section
> 6.12.2 on interoperability which indicates the interaction between ACS and
> non-ACS components, including:
> 
>  * When ACS P2P Request Redirect, ACS P2P Completion Redirect, or both
>    are being used, certain components in the PCI Express hierarchy must
>    support ACS Upstream Forwarding (of Upstream redirected Requests).
>    Specifically:
>    ...
>    Between each ACS component where P2P TLP redirection is enabled and
>    its associated Root Port, any intermediate Switches must support ACS
>    Upstream Forwarding. Otherwise, how such Switches handle Upstream
>    redirected TLPs is undefined.
> 
> It's my interpretation therefore that in a configuration where the switch
> downstream ports supports ACS, the switch upstream port must implicitly
> support upstream forwarding, thus I would consider this a hardware issue.
> The alternative is that we need to poke holes in the VM address space to
> account for any possible conflict and assigned device hot-add becomes
> nearly a non-starter.  Thanks,
> 
> Alex

Thanks for your explanation.
Do you know are there other vendors' switches that wouldn't result in this problem?

Thanks,
Zongyong Wu