[vfio-users] Cannot register eventfd with MSI/MSI-X interrupts

[Alex Williamson]

On Tue, 2 Aug 2016 10:21:30 +0000
chris thompson <the_cartographer at hotmail.com> wrote:

> Hi Alex,
>  
> Thanks for your patience, I noticed the count != 0 issue shortly after, but I get the same EINVAL error anyway when it is zero - but this is because the IRQ does not have a mapping yet. When I change the code to map it then unmap it then I get:
> IRQx - successful map then unmap
> MSI - unsuccessful map (and unsuccessful unmap)
> MSIX - successful map then unmap (despite the device still being in IRQx mode?)
> Error - successfully map, kernel driver crash on the unmap!

So the DATA_NONE, count=0 mechanism is not as ubiquitous as I'd like,
the PCI interrupts support it, the Error and Request interrupts do
not, and in fact Error doesn't like DATA_NONE at all.  The fix for this
is pretty easy, but this interrupt has had a torrid past, some of the
parameters were never sanitized, so we pretty much ignore them.  Start
and count parameters both fall into this camp.

However, I'm discovering we have more flexibility to actually fix it
correctly than we previously thought.  When DATA_NONE is not set, we do
a pretty good job of checking that start can only be 0 and count can
only be 0 or 1, but we don't sanitize at all when DATA_NONE is set.
However, based on the bug you found, I think we can conclude that
DATA_NONE is completely broken and therefore we can fix it correctly
and fully since it apparently has no users.

I'd need to see example code to understand the remainder of your test
results.
 
> I understand that the IRQx, MSI and MSI-X are different - mutually exclusive - IRQ channels, I hadn't realised that from the VFIO perspective they were mutually exclusive in terms of registering the eventfd. I was trying to bind them all so that it didn't matter how the device was programmed later, I would already have the callbacks bound.

This doesn't match the vfio API, there's no "store these eventfds for
future use, don't enable the interrupt now" interface.  When SET_IRQS
is called for a given index, that's an indication to enable, disable,
or in limited cases modify that parameters for that interrupt.
Userspace is responsible for managing their unused eventfds, not the
kernel.

> What I'm actually trying to do is use VFIO to do PCI(e) passthrough to an emulated (rather than virtualized) target platform, I'll just have to be smarter about registering/unregistering eventfd callbacks depending on how the "guest" kernel driver configured the device. Just now I'm just playing with some simple C code poking the registers and interrupts to get a feel for VFIO.

Emulated rather that virtualized, such as QEMU TCG vs KVM?  vfio-pci in
QEMU has no hard ties to KVM, it can already be used for emulated
platforms.  A sticky point though is that TCG emulation doesn't
guarantee atomic memory updates from the perspective of an independent
I/O device, so theoretically some forms of semaphores or
synchronization between an assigned device and an emulated guest won't
work like they do for a virtualized guest.
 
> If you want the whole test program for the sake of reproduction then I can send you it, but I expect you'll be able to put together an "unregister the VFIO_PCI_ERR_IRQ_INDEX" ioctl test in a couple of minutes anyway.

Yeah, reproducing the DATA_NONE issue was easy, but I'd need to see
code to really understand your results above.  Thanks,

Alex