[PATCH] qemu: add pcie-to-pci-bridge for q35 machines

[Daniel P. Berrangé]

On Fri, Nov 04, 2022 at 03:01:38PM +0100, Denis V. Lunev wrote:
> On 11/4/22 14:41, Daniel P. Berrangé wrote:
> > On Fri, Nov 04, 2022 at 01:03:52PM +0100, Peter Krempa wrote:
> > > On Fri, Nov 04, 2022 at 16:43:00 +0600, Oleg Vasilev wrote:
> > > > Hotplugging PCI devices on pc-i440fx machines is supported without
> > > > additional configuration. On q35, pcie-to-pci-bridge needs to be added
> > > > prior to the machine startup in order to support hotplug [1].
> > > > 
> > > > The idea is to support the same workflow for creating VMs in q35, as was
> > > > in pc. Namely, do not require additional configuration when hotplugging
> > > > is needed. Otherwise, all libvirt clients need to be updated which there
> > > > are a lot and they are maintained by different parties.
> > > > 
> > > > Instead, a pcie-to-pci-bridge better be created by default, so that PCI
> > > > slots are readily available. Might be a good idea to make it configurable
> > > > in the future.
> > The goal of q35 is to provide a PCI-e topology though, and we want
> > devices to be exposed as PCI-e endpoints, with no legacy PCI usage
> > by default.
> > 
> > When a guest XML is provided for cold boot, libvirt will ensure
> > that all devices are placed onto the PCI-e bus via pcie-root-ports
> > 
> > With this proposal hotplugging a device will cause it to be placed
> > on a PCI bus instead, if there are no spare pcie-root-port available.
> > 
> > This behavioural difference between cold plug and hot plug is very
> > undesirable IMHO, as it is counter to our goal of avoiding legacy
> > PCI on PCI-e machines.
> 
> that is an interesting point which I have initially missed. I have
> checked our investigation log and found that we have
> potentially made a mistake with making compatibility
> check with e1000 device instead of virtio, which is
> our default. This could be the reason.
> 
> > 
> > If a mgmt app wants to enable this mis-matched behaviour they can
> > add a pcie-to-pci-bridge when initially cold booting the VM, but
> > we shouldn't add one by default IMHO. We really recommend that
> > apps just pre-populate 10-30 pcie-root-port devices when creating
> > a VM, so there is plenty of space for hotplugging devices later.
> 
> this potentially an option, but I think that these port should be
> created automatically by libvirt to avoid major client rewrites.
> Will the option to keep the amount of spare ports in /etc/libvirt/qemu.conf
> work for you?

Our general view is that qemu.conf should not cause changes that
affect the guest ABI.

For an given input XML, we want applications to have a predictable
guest ABI created. Influencing the guest ABI from qemu.conf settings
would mean applications have uncertain results for a given XML config,
so what works on one host may break on another host.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|