[PATCH] qemu: add pcie-to-pci-bridge for q35 machines

[Peter Krempa]

On Fri, Nov 04, 2022 at 16:43:00 +0600, Oleg Vasilev wrote:
> Hotplugging PCI devices on pc-i440fx machines is supported without
> additional configuration. On q35, pcie-to-pci-bridge needs to be added
> prior to the machine startup in order to support hotplug [1].
> 
> The idea is to support the same workflow for creating VMs in q35, as was
> in pc. Namely, do not require additional configuration when hotplugging
> is needed. Otherwise, all libvirt clients need to be updated which there
> are a lot and they are maintained by different parties.
> 
> Instead, a pcie-to-pci-bridge better be created by default, so that PCI
> slots are readily available. Might be a good idea to make it configurable
> in the future.
> 
> Previously there was a pci-bridge present by default, but was removed in
>   d5fb8f4564 (qemu: don't add pci-bridge to Q35/arm domains unless it's needed, 2016-04-22)
> 
> [1]: https://libvirt.org/pci-hotplug.html
> 
> Signed-off-by: Oleg Vasilev <oleg.vasilev at virtuozzo.com>
> ---

[...]

>  73 files changed, 621 insertions(+), 361 deletions(-)
> 
> diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c
> index 9ef6c8bb64..52a393e07f 100644
> --- a/src/qemu/qemu_domain.c
> +++ b/src/qemu/qemu_domain.c
> @@ -3890,6 +3890,7 @@ qemuDomainDefAddDefaultDevices(virQEMUDriver *driver,
>      bool addImplicitSATA = false;
>      bool addPCIRoot = false;
>      bool addPCIeRoot = false;
> +    bool addPCIeToPCIBridge = false;
>      bool addDefaultMemballoon = true;
>      bool addDefaultUSBKBD = false;
>      bool addDefaultUSBMouse = false;
> @@ -3910,6 +3911,8 @@ qemuDomainDefAddDefaultDevices(virQEMUDriver *driver,
>          if (qemuDomainIsQ35(def)) {
>              addPCIeRoot = true;
>              addImplicitSATA = true;
> +            if (virQEMUCapsGet(qemuCaps, QEMU_CAPS_DEVICE_PCIE_PCI_BRIDGE))
> +                addPCIeToPCIBridge = true;
>  
>              /* Prefer adding a USB3 controller if supported, fall back
>               * to USB2 if there is no USB3 available, and if that's
> @@ -4046,11 +4049,7 @@ qemuDomainDefAddDefaultDevices(virQEMUDriver *driver,
>          }
>      }
>  
> -    /* When a machine has a pcie-root, make sure that there is always
> -     * a dmi-to-pci-bridge controller added as bus 1, and a pci-bridge
> -     * as bus 2, so that standard PCI devices can be connected
> -     *
> -     * NB: any machine that sets addPCIeRoot to true must also return
> +    /* NB: any machine that sets addPCIeRoot to true must also return
>       * true from the function qemuDomainSupportsPCI().
>       */
>      if (addPCIeRoot) {
> @@ -4069,6 +4068,16 @@ qemuDomainDefAddDefaultDevices(virQEMUDriver *driver,
>          }
>      }
>  
> +    /* Add a pcie-to-pci-bridge and pcie-root-port to plug it into. */
> +    if (addPCIeToPCIBridge) {
> +        if (virDomainDefMaybeAddController(def, VIR_DOMAIN_CONTROLLER_TYPE_PCI, 1,
> +                                       VIR_DOMAIN_CONTROLLER_MODEL_PCIE_ROOT_PORT) < 0)
> +            return -1;
> +        if (virDomainDefMaybeAddController(def, VIR_DOMAIN_CONTROLLER_TYPE_PCI, 2,
> +                                       VIR_DOMAIN_CONTROLLER_MODEL_PCIE_TO_PCI_BRIDGE) < 0)
> +            return -1;
> +    }

Doing this unconditionally breaks ABI of the VM, as new guest visible
devices appear without any change in configuration. What's worse this
implementation would also break migration compatibility from older
versions of libvirt as there isn't any form of explicit handling for it.

I unfortunately don't see any reasonable solution originating in
libvirt. The bridge will need to be added by users or higher level
management tools (e.g. virt-install) as libvirt explicitly wants to
support use cases when VMs are started without persisting the XML
definition (e.g. always re-define the XML before start or use the
'createXML' api).