[libvirt] Analysis of the effect of adding PCIe root ports

[Laine Stump]

On 10/05/2016 11:50 AM, Richard W.M. Jones wrote:
> I was asked to look at the impact on boot times of adding (empty) PCIe
> root ports to the device model.  The proposal from Laine is to add a
> few of these to every guest to allow hotplugging.
>
> Last time I looked into this I found that probing any (legacy) PCI
> device is expensive because of the inefficient way that qemu emulates
> accesses to PCI config space, requiring IIRC 2 or 4 VMEXITs to access
> every word.  (PCI slots which are not occupied are basically free, the
> problem is PCI devices).  At that time I did not look at Q35/PCIe at all.
>
> We generally aim for boot times under 600-700ms.  Probing PCI devices
> takes a significant fraction of this time.
>
> The detailed analysis is attached.  It comes from a program called
> 'boot-analysis'
> (https://github.com/libguestfs/libguestfs/tree/master/utils/boot-analysis).
> It is best viewed using 'less -r' so that you can see the colours.
>
> The summary table is:
>
>                 Number of ports       0       1      2      3     4
>
> bios:overhead                        874     884    875    878   935
> kernel:entry                         159     163    165    174   491
> kernel:initcalls-before-userspace    1065    1090   1110   1147  1263
> /init:udev-overhead                  173     187    185    193   301
> insmod virtio_pci                    43      41     41     41    74
>
>                 TOTAL                       + 51   + 62   + 119 + 750
>
>                       (All times are in milliseconds.)
>
> A few observations:
>
> (1) For #ports <= 3, each extra port adds around 30-40ms to the boot
> time, which is consistent with what I saw when I measured legacy PCI.

Thanks for doing this (and especially on such short notice!). This is 
interesting info. I'm wondering now if the discontinuity is due to going 
over a threshold in the number of pcie-root-ports, the number of total 
PCI controllers, or the total number of devices. This is significant 
because a couple of the patches in the "Use more PCIe less legacy PCI" 
patchset I sent a two weeks ago will eliminate both the 
dmi-to-pci-bridge and the pci-bridge in the default config (as long as 
there are no legacy PCI devices). If the "jump" is due to an increase in 
total devices, or total PCI controllers, then getting rid of those two 
will change the number of root-ports that can be added prior to the jump.

Even if this is the case, it seems like we still want to have as little 
fat as possible.


>
> (2) There is a sudden, unexplained and reproducible discontinuity when
> going from 3 to 4 ports.  (Because libguestfs uses other devices, this
> might not actually be 3 to 4 PCIe devices, this is spare ports after
> all the others added for libguestfs devices.)
>
> (3) "kernel:entry" covers a lot of initialization that happens before
> the kernel prints its version string.  This is moderately affected by
> adding PCIe ports until we hit the discontinuity.
>
> Based on this analysis I conclude:
>
> (a) Unless we can explain the discontinuity, avoid adding more than
> 3 root ports.
>
> (b) It would be nice to turn the whole thing off for people who don't
> care about / need hotplugging.

I had contemplated having an "availablePCIeSlots" (or something like 
that) that was either an attribute of the config, or an option in 
qemu.conf or libvirtd.conf. If we had such a setting, it could be set to 
"0".

>
> (c) We could make PCI probing a lot faster by having the kernel handle
> PCI config space.