[libvirt] Analysis of the effect of adding PCIe root ports

Richard W.M. Jones rjones at redhat.com
Wed Oct 5 15:50:44 UTC 2016


I was asked to look at the impact on boot times of adding (empty) PCIe
root ports to the device model.  The proposal from Laine is to add a
few of these to every guest to allow hotplugging.

Last time I looked into this I found that probing any (legacy) PCI
device is expensive because of the inefficient way that qemu emulates
accesses to PCI config space, requiring IIRC 2 or 4 VMEXITs to access
every word.  (PCI slots which are not occupied are basically free, the
problem is PCI devices).  At that time I did not look at Q35/PCIe at all.

We generally aim for boot times under 600-700ms.  Probing PCI devices
takes a significant fraction of this time.

The detailed analysis is attached.  It comes from a program called
'boot-analysis'
(https://github.com/libguestfs/libguestfs/tree/master/utils/boot-analysis).
It is best viewed using 'less -r' so that you can see the colours.

The summary table is:

               Number of ports       0       1      2      3     4

bios:overhead                        874     884    875    878   935
kernel:entry                         159     163    165    174   491
kernel:initcalls-before-userspace    1065    1090   1110   1147  1263
/init:udev-overhead                  173     187    185    193   301
insmod virtio_pci                    43      41     41     41    74

               TOTAL                       + 51   + 62   + 119 + 750

                     (All times are in milliseconds.)

A few observations:

(1) For #ports <= 3, each extra port adds around 30-40ms to the boot
time, which is consistent with what I saw when I measured legacy PCI.

(2) There is a sudden, unexplained and reproducible discontinuity when
going from 3 to 4 ports.  (Because libguestfs uses other devices, this
might not actually be 3 to 4 PCIe devices, this is spare ports after
all the others added for libguestfs devices.)

(3) "kernel:entry" covers a lot of initialization that happens before
the kernel prints its version string.  This is moderately affected by
adding PCIe ports until we hit the discontinuity.

Based on this analysis I conclude:

(a) Unless we can explain the discontinuity, avoid adding more than
3 root ports.

(b) It would be nice to turn the whole thing off for people who don't
care about / need hotplugging.

(c) We could make PCI probing a lot faster by having the kernel handle
PCI config space.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pci.tar.xz
Type: application/x-xz
Size: 43988 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20161005/c0ae8c5d/attachment-0001.xz>


More information about the libvir-list mailing list