[vfio-users] Posted interrupts for nested virtualization?

[Alex Williamson]

On Fri, 18 Oct 2019 06:08:31 +0000
"Boeuf, Sebastien" <sebastien.boeuf at intel.com> wrote:

> Hi folks,
> 
> We have been recently implementing a nested VFIO solution for our Cloud
> Hypervisor VMM. Thanks to virtio-iommu, we can now pass a device
> through nested virtualization.
> 
> After some performances testing, we realized the device in L2 was
> slightly less performant than the same device in L1. The DMAR has
> nothing to do with it since it is programmed only at boot time with the
> entire guest RAM mapping for each device, but the problem comes from
> the interrupts. When using polling instead of interrupts, we can
> clearly see the device in L2 behaving very closely to what is expected
> in L1.
> 
> So basically, because the interrupt has to bounce through each layer of
> virtualization, the more layers the less performance we will get out of
> this device, right?

Yes

> We were wondering if there was any on going work to allow posted
> interrupts to be delivered directly to the last layer of virtualization
> where the device is actually used. I might be missing something but in
> theory, it seems feasible, the problem seems to be that VFIO does not
> update the IRT of the IOMMU, so I don't know how we could manually
> update it, the same way we can update the DMAR with VFIO_MAP_DMA ioctl
> for instance.
> And the second part is about knowing which interrupt needs to be
> updated in the IRT, which means the virtio-iommu device should be able
> to provide those kind of information.
> 
> It seems like there are some possibilities but I clearly don't have the
> whole picture in mind, and I'm interested in learning more about any
> existing work on this topic.

I agree with your assessment, it seems like it should be possible, but
I don't know of any work to support it.  Note that vfio doesn't
directly manipulate posted interrupts, it's done via the irqbypass
producer/consumer mechanism, so perhaps this largely needs to be done
in KVM.  I can't say I've spent many cycles thinking about it either
though.  Thanks,

Alex