[libvirt] Analysis of the effect of adding PCIe root ports

[Laine Stump]

On 10/06/2016 06:58 AM, Andrea Bolognani wrote:
> On Wed, 2016-10-05 at 18:36 +0100, Richard W.M. Jones wrote:
>>>> (b) It would be nice to turn the whole thing off for people who don't
>>>> care about / need hotplugging.
>>>   
>>> I had contemplated having an "availablePCIeSlots" (or something like
>>> that) that was either an attribute of the config, or an option in
>>> qemu.conf or libvirtd.conf. If we had such a setting, it could be
>>> set to "0".
> I remember some pushback when this was proposed. Maybe we
> should just give up on the idea of providing spare
> hotpluggable PCIe slots by default and ask the user to add
> them explicitly after all.


We need to have *something* new in the config related to this, otherwise 
it will be impossible to add new "available for hotplug" ports at the 
same time as new unaddressed endpoint devices are added (unless an extra 
port is added for each endpoint device as well, and of course that is 
assuming that the user knows about things like auto-added memory balloon 
devices, and which devices are going to be PCIe and which will be legacy 
PCI, etc). If the "available" ports aren't somehow marked, the endpoint 
devices will end up being placed on them.


>
>> Note that changes to libvirt conf files are not usable by libguestfs.
>>   
>> The setting would need to go into the XML, and please also make it
>> possible to determine if $random version of libvirt supports the
>> setting, either by a version check or something in capabilities.
> Note that you can avoid using any PCIe root port at all by
> assigning PCI addresses manually. It looks like the overhead
> for the small (I'm assuming) number of devices a libguestfs
> appliance will use is low enough that you will probably not
> want to open that can of worm, though.

I looked in an example domain config Rich pastebin-ed in IRC yesterday. 
These are the PCI devices he has:

* <controller type='scsi' model='virtio-scsi'> - currently put on a 
pci-bridge, but will end up on a root-port after my patches

* <controller type='usb' model='blah'> x 4 - a set of USB2 controllers. 
This will turn into a single USB3 controller on a root-port after my 
patches. Alternately, since it seems you don't use it, you could 
eliminate it with:

      <controller type='usb' model='none'/>

* <controller type='sata'> - this is fixed on 00:1f.2, and there's 
nothing we can do to get rid of it or even change its address

* <controller type='pci' model='dmi-to-pci-bridge'> - the need for this 
will be eliminated by my patches

* <controller type='pci' model='pci-bridge'> - also eliminated

* <controller type='virtio-serial'> - currently on the pci-bridge, but 
will move to a root-port after my patches

* <memballoon model='virtio'> - Rich, do you actually use this device? 
Seems doubtful. If not, then add

     <memballoon model='none'/>

  to your config to prevent it.

After getting rid of the controllers that will be eliminated by my 
patches, and the "default" devices that you can get rid of with manual 
intervention (model='none'), you'll be left with the following devices 
on pcie-root-ports which *could* be manually moved to pcie-root (thus 
eliminating one pcie-root-port from the domain):

1) virtio-scsi controller
2) virtio-serial controller

and nothing else. Manually address those two to be on bus 0 (pcie-root), 
and (with my patches) you've reduced your PCI device+controller count 
from the current 10 down to 3 (including the sata controller).