[vfio-users] Nested VFIO with QEMU

[Samuel Ortiz]

On Tue, Nov 05, 2019 at 04:58:34PM -0700, Alex Williamson wrote:
> On Wed, 6 Nov 2019 00:29:52 +0100
> Samuel Ortiz <samuel.ortiz at intel.com> wrote:
> 
> > On Tue, Nov 05, 2019 at 01:21:48PM -0700, Alex Williamson wrote:
> > > On Fri, 18 Oct 2019 05:48:49 +0000
> > > "Boeuf, Sebastien" <sebastien.boeuf at intel.com> wrote:
> > >   
> > > > Hi folks,
> > > > 
> > > > I have been recently working with VFIO, and particularly trying to
> > > > achieve device passthrough through multiple layers of virtualization.
> > > > 
> > > > I wanted to assess QEMU's performances with nested VFIO, using the
> > > > emulated Intel IOMMU device. Unfortunately, I cannot make any of my
> > > > physical device work when I pass them through, attached to the emulated
> > > > Intel IOMMU. Using regular VFIO works properly, but as soon as I enable
> > > > the virtual IOMMU, the driver fails to probe (I tried on two different
> > > > machines with different types of NIC).
> > > > 
> > > > So I was wondering if someone was aware of any issue with using both
> > > > VFIO and virtual Intel IOMMU with QEMU? I'm sure I might be missing
> > > > something obvious but I couldn't find it so far.  
> > > 
> > > It's not something I test regularly, but I'm under the impression that
> > > nested device assignment does work.  When you say the driver fails to
> > > probe, which driver is that, the endpoint driver in the L2 guest or
> > > vfio-pci in the L1 guest?  Perhaps share your XML or command line?  
> > 
> > This is fixed now. Apparently the iommu device needs to be passed
> > _before_ the other devices on the command line. We managed to make it
> > work as expected.
> 
> Good news!
> 
> > Sebastien and Yi Liu figured this out but for some reasons the
> > thread moved to vfio-users-bounces at redhat.com.
> 
> Yes, I see some uncaught bounce notifications, it looks like Yi's
> initial reply was to vfio-users-bounces.  Yi, you might want to
> checkout your mailer configuration.  For posterity/follow-up, I'll
> paste the final message from the bounce notification below.  
Thanks for doing that.

Cheers,
Samuel.
---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris, 
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.