[vfio-users] Posted interrupts for nested virtualization?

[Boeuf, Sebastien]

Hi folks,

We have been recently implementing a nested VFIO solution for our Cloud
Hypervisor VMM. Thanks to virtio-iommu, we can now pass a device
through nested virtualization.

After some performances testing, we realized the device in L2 was
slightly less performant than the same device in L1. The DMAR has
nothing to do with it since it is programmed only at boot time with the
entire guest RAM mapping for each device, but the problem comes from
the interrupts. When using polling instead of interrupts, we can
clearly see the device in L2 behaving very closely to what is expected
in L1.

So basically, because the interrupt has to bounce through each layer of
virtualization, the more layers the less performance we will get out of
this device, right?

We were wondering if there was any on going work to allow posted
interrupts to be delivered directly to the last layer of virtualization
where the device is actually used. I might be missing something but in
theory, it seems feasible, the problem seems to be that VFIO does not
update the IRT of the IOMMU, so I don't know how we could manually
update it, the same way we can update the DMAR with VFIO_MAP_DMA ioctl
for instance.
And the second part is about knowing which interrupt needs to be
updated in the IRT, which means the virtio-iommu device should be able
to provide those kind of information.

It seems like there are some possibilities but I clearly don't have the
whole picture in mind, and I'm interested in learning more about any
existing work on this topic.

Thanks,
Sebastien