[libvirt] Regarding PCI/PCIe Multifunction hotplug support in libvirt

[Laine Stump]

On 04/15/2016 05:15 AM, Shivaprasad bhat wrote:
> Hi All,
>
> I am exploring how to go about supporting multi-function 
> hot-plug/unplug support from libvirt now that Qemu has enabled it.
>
> The Qemu semantics is that, qemu queues the notification to guest 
> until function 0 is hot-
> plugged and on function 0 hot-plug, all the previously added devices 
> in the slot are
> notified to the guest.
> For hot-unplug - On x86, unplug of any device in the slot would unplug 
> all the functions of
> the slot. On PPC, unplug all non-zero functions first, then unplug the 
> function zero which
> triggers the unplug of all the functions in the slot.
>
> On Libvirt, I had a brief chat with Laine Stump on IRC and we "think" 
> the below semantics
> would be appropriate.
> 1) Change the virDomainAttachDeviceFlags() to recognize multiple 
> devices in the XML
> and the application should make one call to attach all the functions 
> for the slot at one
> time.
> 2) The libvirt qemu driver to translate this into multiple attach 
> device commands to qemu
> with the final operation being for function 0.

Some explanation of why I think the attach of all functions should be a 
single operation in the libvirt API: mainly because it is a single 
operation from the POV of the guest OS, and other hypervisors may 
present it to libvirt in that manner as well. It would be more difficult 
in such a case for libvirt to gather up the various attach calls to 
squirt them out to the hypervisor when the final function attach is 
requested. It would also lead to error recovery Hell if there was a 
failure in the final step (doing the actual work once the info for all 
the functions was saved up) - previous calls would have already returned 
success, and now they suddenly become a failure; how would you report 
that, and how would the management application recover?

(But of course I may be missing something stupid, so don't hesitate to 
protest/discredit/etc :-)

Note that putting the XML for multiple functions that will be in the 
same slot will allow management to let libvirt decide on the proper slot 
as well as automatically setting the "multifunction" flag on function 0 
(which should have always been done anyway. AFAIK, there isn't valid 
case for having multiple functions in a slot where function 0 *doesn't* 
have multifunction set).

> 3) The XML now needs to accept multiple devices and there is a need 
> for single parent
> element. I am thinking if we should have all devices to be enclosed in 
> <device></device>
> parent element. Note that this is only when all the devices should go 
> to the same slot. We
> probably should disallow user attempting to hot-plug to different 
> slots with this.
> 4) For the hot-unplug, the application to send all the devices the 
> same way enclosed in
> <device></devices>

"devices" in both cases, of course :-)


> and libvirt to go ahead with unplug only if all the devices are
> specified in the XML for the slot.
>
> Want to know if you foresee any problem with using <device></device> 
> semantics Or you
> have any different approach/suggestions. Any comments greatly appreciated.
>
> Thanks and Regards,
> Shivaprasad