[libvirt] VFIO device groups and libvirt

[Alex Williamson]

On Thu, 2013-05-30 at 12:29 -0400, Laine Stump wrote:
> VFIO device assignment has a concept of device "groups"; each group is a
> set of devices that must all be assigned to the same guest (alternately,
> unassigned devices can be attached to the vfio stub driver). This is to
> prevent cross-guest (or guest-host) contamination of devices that share
> DMA mapping or some such thing and thus can't be completely isolated
> from each other.
> 
> The vfio support currently in libvirt works with no problems if no other
> devices are in the same "group" as a device being assigned to a guest.
> If there are other devices in the group, it fails with a message noting
> that the other devices in the group must also be attached to the vfio
> stub driver (so that the host can't use them).
> 
> I am now adding code to libvirt that, when receiving a request to attach
> a device in a group to the vfio stub, will notice the other devices in
> the group and attach them to the stub as well; it will then allow
> attaching all of these devices to the guest, but not to any other guest.
> This will be nice for those who *want* this behavior, but I think that
> very often a person trying to assign a device that's in a group doesn't
> even realize that it's in a group, and if they knew it was in a group,
> they *wouldn't* want to lose all those devices just to assign one (once
> a single device in a group is assigned to the guest, none of the other
> devices in the group are usable by the host).
> 
> An example of this was encountered by Cole the other day. He had a
> non-SRIOV network card plugged into his system that he wanted to assign
> with VFIO, but it turned out that due to the hardware layout of his
> motherboard, this plugin network card was placed in a group with 2
> devices that were part of the motherboard chipset. When this occurs, a
> user might want to take one of three different courses:
> 
> 1) attach all those onboard devices to the vfio stub driver too (making
> them inaccessible to the host, but allowing the assignment of the target
> device via VFIO)
> 
> 2) use legacy KVM device assignment instead (which is less secure, but
> will allow assigning one of the devices while the rest remain available
> to the host)
> 
> 3) try plugging the card they want to assign into a different slot
> (which may remove the device from this group and place it in another,
> possibly empty, group)
> 
> It's difficult to say which the user will want, so I don't want the new
> "attach all devices in the group to the stub" code to be automatic.
> Instead, I think it should only take effect if some option is specified
> in the device's <driver> element. For example:
> 
> 
>    <hostdev managed='yes'>
>       <driver name='vfio' group='allow'/>
>       ...
>    </hostdev>
> 
> Without that xml option, the same error message logged now will continue
> to be logged. With that xml option, all the devices in the group will be
> attached to the vfio stub driver.
> 
> Likewise, in virNodeDeviceDetachFlags(), a VIR_NODE_DEVICE_DETACH_GROUP
> flag will have the following effect:
> 
>    Single device in group - no effect
>    Multiple devices, no flag - detach will fail
>    Multiple devices, flag set - all devices in the same group as
> specified device will be detached by one call.
> 
> Does this sound reasonable?

nit, vfio is not a stub driver, it's an actual driver.

vfio also whitelists some host drivers.  The two so far are pci-stub and
pcieport.  libvirt should not disconnect root ports from pcieport unless
the root port is being assigned to a guest (which isn't supported).
pci-stub can be used to satisfy the requirement that the device is
disconnected from the host, but doesn't allow direct access of the
device through vfio.

I'm also working on a kernel patch that will allow a user to specify on
the kernel command line devices and sets of devices to assume have ACS
support.  This should allow users to strategically change device
grouping if they want to opt-in to the risk of assigning devices in
topologies without ACS support.

The additional xml flag is unfortunate, but I can see why you want it.
Thanks,

Alex