[libvirt] Analysis of the effect of adding PCIe root ports

[Laine Stump]

On 10/06/2016 11:31 AM, Daniel P. Berrange wrote:
> On Thu, Oct 06, 2016 at 12:58:51PM +0200, Andrea Bolognani wrote:
>> On Wed, 2016-10-05 at 18:36 +0100, Richard W.M. Jones wrote:
>>>>> (b) It would be nice to turn the whole thing off for people who don't
>>>>> care about / need hotplugging.
>>>>   
>>>> I had contemplated having an "availablePCIeSlots" (or something like
>>>> that) that was either an attribute of the config, or an option in
>>>> qemu.conf or libvirtd.conf. If we had such a setting, it could be
>>>> set to "0".
>> I remember some pushback when this was proposed. Maybe we
>> should just give up on the idea of providing spare
>> hotpluggable PCIe slots by default and ask the user to add
>> them explicitly after all.
>>
>>> Note that changes to libvirt conf files are not usable by libguestfs.
>>>   
>>> The setting would need to go into the XML, and please also make it
>>> possible to determine if $random version of libvirt supports the
>>> setting, either by a version check or something in capabilities.
>> Note that you can avoid using any PCIe root port at all by
>> assigning PCI addresses manually. It looks like the overhead
>> for the small (I'm assuming) number of devices a libguestfs
>> appliance will use is low enough that you will probably not
>> want to open that can of worm, though.
> For most apps the performance impact of the PCI enumeration
> is not a big deal. So having libvirt ensure there's enough
> available hotpluggable PCIe slots is reasonable, as long as
> we leave a get-out clause for libguestfs.
>
> This could be as simple as declaring that *if* we see one
> or more <controller type="pci"> in the input XML, then libvirt
> will honour those and not try to add new controllers to the
> guest.
>
> That way, by default libvirt will just "do the right thing"
> and auto-create a suitable number of controllers needed to
> boot the guest.
>
> Apps that want strict control though, can specify the
> <controllers> elements themselves.  Libvirt can still
> auto-assign device addresses onto these controllers.
> It simply wouldn't add any further controllers itself
> at that point. NB I'm talking cold-boot here. So libguestfs
> would specify <controller> itself to the minimal set it wants
> to optimize its boot performance.

That works for the initial definition of the domain, but as soon as 
you've saved it once, there will be controllers explicitly in the 
config, and since we don't have any way of differentiating between 
auto-added controllers and those specifically requested by the user, we 
have to assume they were explicitly added, so such a check is then 
meaningless because you will *always* have PCI controllers.

Say you create a domain definition with no controllers, you would get 
enough for the devices in the initial config, plus "N" more empty root 
ports. Let's say you then add 4 more devices (either hotplug or 
coldplug, doesn't matter). Those devices are placed on the existing 
unused pcie-root-ports. But now all your ports are full, and since you 
have PCI controllers in the config, libvirt is going to say "Ah, this 
user knows what they want to do, so I'm not going to add any extras! I'm 
so smart!". This would be especially maddening in the case of 
"coldplug", where libvirt could have easily added a new controller to 
accomodate the new device, but didn't.

Unless we don't care what happens after the initial definition (and then 
adding of "N" new devices), trying to behave properly purely based on 
whether or not there are any PCI controllers present in the config isn't 
going to work.

(BTW, isn't there something wrt aarch64 about "no pci controllers in 
config means use mmio for devices", or something like that? (Or maybe we 
were just *thinking* about that and didn't actually do it, I don't 
remember). Using the lack of PCI controllers in the config to imply 
"automatically add necessary + extra controllers" would directly 
conflict with that.)