[vfio-users] Is there some method to merge 2 iommu group if I disable ACS

[Alex Williamson]

On Tue, 8 Aug 2017 06:58:40 +0000
"Wuzongyong (Euler Dept)" <cordius.wu at huawei.com> wrote:

> > On Mon, 7 Aug 2017 06:43:30 -0600
> > Alex Williamson <alex.l.williamson at gmail.com> wrote:
> >   
> > > On Aug 7, 2017 5:39 AM, "Wuzongyong (Euler Dept)"
> > > <cordius.wu at huawei.com>
> > > wrote:
> > >
> > > Hi,
> > >
> > >
> > >
> > > Assume that an endpoint device(called ep1) belongs to iommu group 1,
> > > and another endpoint device(called ep2) belongs to iommu group 2.
> > >
> > > Moreover, these two devices locate in different downstream ports of
> > > the same switch respectively. If I disable the ACS of these downstream
> > >
> > > ports,  we know ep1 and ep2 should locate in the same iommu group. So
> > > the question is if I can regenerate a iommu group to let ep1 and ep2
> > >
> > > locate in the same iommu group and let ep1 and ep2  can’t be assigned
> > > to two different VMs without  host rebooting?  
> > 
> > 
> > Wow, complete formatting fail from gmail on my phone.  Sorry for that.
> > Question is:
> >   
> > > Why would you want to be able to do this?  
> > 
> > Thanks,
> > 
> > Alex  
> 
> I want to do peer-to-peer communication between the two pci-e eps.

A significant problem here is the I/O virtual address (IOVA) space in
which the device operates when used with vfio.  Peer-to-peer will
operate that the physical address level while userspace drivers like
QEMU will operate with IOVAs.  The proper why to get both is for the
endpoint to support ATS and ACS on the downstream switch port to
support Direct Translation.  If these two things were both enabled,
then a transaction from the endpoint with the address translation field
set would be able to do direct p2p with another device while
non-translated transactions would go through the IOMMU.

Therefore, I don't think what you're about to try is going to work and
you're on your own for support.  That said, you can manipulate the ACS
control bits on the downstream switch ports with setpci to disable
ACS.  You might then be able to use the "remove" sysfs entry for the
PCI endpoint and the /sys/bus/pci/rescan entry to re-add the device,
which would hopefully detect the isolation issue and group the
endpoints together.  You might still run into problems that the
downstream ports are already in separate groups and this won't work
without also removing and rescanning the downstream port itself, but
this risks that ACS will be re-enabled on that port.  IOW, there are
some hacks you can try, but this is not an encouraged or supported
procedure and don't expect support for it.  Thanks,

Alex