[PATCH] qemu: add pcie-to-pci-bridge for q35 machines

[Laine Stump]

On 11/4/22 11:05 AM, Andrea Bolognani wrote:
> On Fri, Nov 04, 2022 at 08:23:52PM +0600, Oleg Vasilev wrote:
>> On 04.11.2022 20:19, Andrea Bolognani wrote:
>>> Additionally, after this change it would be impossible to create a
>>> q35 VM *without* the additional bridge. Most users of the q35 machine
>>> type are likely using PCIe devices exclusively, and such a change
>>> would negatively impact them.
>>>
>>> In the (hopefully rare) cases in which hotplugging traditional PCI
>>> devices is actually required, enabling that at the time when the
>>> domain is defined is already straightforward: just include the
>>> relevant controller in the XML.
>>>
>>> Unfortunately q35, unlike i440fx, doesn't support hotplugging of any
>>> kind out of the box, so it's up to the user / management application
>>> to ensure that the necessary controllers are added. This is not
>>> ideal, but there's no way around it, and it's still preferable to
>>> forcing unavoidable, potentially useless extra controllers on
>>> everybody.
>>
>> Apparently, there is a way to hotplug pcie-to-pci-bridge described in qemu
>> doc [1]. Do you think this approach with bus-reserve=1 would be acceptable
>> to support in libvirt by default?
>>
>> [1]: https://github.com/qemu/qemu/blob/master/docs/pcie_pci_bridge.txt#L25
> 
> I don't think so.
> 
> First of all, the document says that the feature is only supported by
> SeaBIOS, whereas most q35 guests are likely using UEFI firmware.
> 
> Moreover, we wouldn't be able to just set bus_reserve=1 for all
> pcie-root-ports, as that would result in a massive waste of PCI bus
> numbers. So we'd have to select a single pcie-root-port and make sure
> we don't hotplug anything but a pcie-to-pci-bridge in there. I can't
> imagine a way to implement this that's not going to be both
> incredibly complicated and extremely fragile.

Yes, my recollection of the description of how to *potentially* support 
hotplug of a pcie-to-pci-bridge into a running VM (as explained by 
Marcel, back when he was working on this stuff) is that 1) it would take 
a lot of work (and hand waving) in the guest OS as well as in qemu *and* 
in libvirt, and 2) it would be fragile, and likely not work in exactly 
the scenarios where people wanted it to work. In the end, the return on 
investment just seemed incredibly low.

> 
> At the end of the day, q35 just requires to plan the hotplug
> capabilities of the VM ahead of time, and this is true both for PCIe
> and traditional PCI hotplug. At the libvirt level, I don't really see
> a way around that.
> 
> Higher-level tools are of course free to offer a more opinionated out
> of the box experience. I believe both virt-manager and OpenStack nova
> allocate a few pcie-root-ports by default to ensure PCIe hotplug can
> work. Perhaps they'd be okay adding a pcie-to-pci-bridge by default
> too, making the experience for end users a less frustrating one.
> 

Don't ever let Marcel hear that! :-)

Seriously though, there is almost 0 reason to need a conventional PCI 
device under *any* circumstances, much less needing to hotplug one. If 
you pick the correct model of device, all your devices will be PCIe, and 
in that case having a pcie-to-pci-bridge hanging around in every single 
domain definition will just be wasting resources for no reason at all 
(not even a *bad* reason).