[libvirt] [PATCH 00/10] VFIO fixes for PCI devices

[Alex Williamson]

On Wed, 2015-12-09 at 13:51 +0100, Andrea Bolognani wrote:
> On Wed, 2015-12-02 at 18:17 +0100, Andrea Bolognani wrote:
> > This series is my attempt at fixing
> > 
> >   https://bugzilla.redhat.com/show_bug.cgi?id=1272300
> > 
> [...]
> > 
> > The problem being solved is that, when using VFIO, IOMMU group
> > ownership can't be shared, eg. two devices that are in the
> > same IOMMU group can't be assigned to different guests, or to
> > the host and a guest. If that happens, the host will probably
> > crash.
> > 
> > The series deals with this issue by making sure safety
> > conditions are met before detaching devices from the host or
> > reattaching them to the host. In praticular, when we're asked
> > to reattach a device to the host but doing so would lead to
> > sharing IOMMU group ownership, we delay the operation until
> > we can guarantee this will not cause problems. As a nice side
> > effect of the changes we check for this when starting a guest
> > too, instead of assuming it will work and having QEMU error
> > out immediately afterwards.
> 
> Shivaprasad raised a concern on IRC, which I'm sharing here
> for wider discussion. I'm CC'ing Laine and Alex, hopefully
> they don't mind - let me know otherwise.
> 
> Assume we have a PCI device with two functions. With this
> series applied, when reattaching both functions to the host
> this would happen:
> 
>   f0 remove from guest
>   f1 remove from guest
>     f0 unbind from vfio-pci
>       f0 trigger host driver reprobe
>     f1 unbind from vfio-pci
>       f1 trigger host driver reprobe
> 
> Shivaprasad is concerned this is not actually safe, and the
> proper sequence would rather be:
> 
>   f0 remove from guest
>   f1 remove from guest
>     f0 unbind from vfio-pci
>     f1 unbind from vfio-pci
>       f0 trigger host driver reprobe
>       f1 trigger host driver reprobe
> 
> Doing so would AFAICT mean basically duplicating the delay
> logic this series adds to virHostdev into virPCI, to ensure
> that devices are unbound from vfio-pci only once the same
> operation has been requested for all devices in the IOMMU
> group, and reprobe is triggered only after all devices have
> been unbound from vfio-pci.
> 
> I was under the impression that what the current series
> does, eg. sharing devices in the same IOMMU group between
> the host driver and vfio-pci is safe as long as no guest is
> using them at the same time, and that devices could be
> safely "moved" between the host driver (eg. in use) and
> vfio-pci (eg. idle, waiting to be assigned to a guest) as
> many times as desired without ill consequences.
> 
> Is my understanding wrong? Do I need to rework the series
> so that unbinds and reprobes are always executed across the
> IOMMU group?
> 
> Any suggestion or pointers to relevant documentation will
> be very much appreciated.

Hi Andrea,

Your understanding is correct, so I think it just comes down to how sure
you are that the vfio group is idle/unused.  If there's any risk that a
device is still in use by QEMU, then we haven't solved the original
problem.  Unbinding isn't the only way to have good confidence of this
though.  You could track the QEMU pid, you could use fuser to make sure
the group is not in use, you could try to open the group yourself to
make sure it's not in use, and of course you can unbind as proposed in
the second option.  So long as you know the group is idle, somehow, it
shouldn't matter what order you unbind and reprobe.  Thanks,

Alex