Adding support for multiple PCI domains (multiple root controllers)

[Richard Hansen]

I would like to add support for multiple PCI domains to libvirt.  My main motivation is bug #360 [1], though it would also make pSeries machine configurations more intuitive (they can have multiple pci-root controllers).

Currently, every PCI controller is in PCI domain 0, and the controller's index number equals its bus ID.  To support multiple domains, my plan is to map domain:bus ID pairs to/from controller index numbers as follows:

   * A root controller with index N establishes domain N, and provides bus 0 in that domain.
   * A non-root controller with index N, and plugged in to a PCI controller in domain D, provides bus N in domain D.

Furthermore, each non-root controller must have a higher index number than the controller it plugs in to, so the controller with index 0 is guaranteed to be a root controller.  This means that in the common case of only a single domain, the behavior would be the same as it is now (domain == 0, controller index == bus ID).

Example domain:bus pairs:

   * 0000:00 would identify the bus provided by the root controller with index 0 (as it does now).
   * 000f:1b would identify the bus provided by the non-root controller with index 27 (0x1b), and that controller is plugged into domain 15 (0xf).

Implications of this scheme:

   * Bus IDs never match any domain IDs (exception: bus 0 in domain 0).
   * Bus IDs never appear in more than one domain (exception: bus 0 appears in every domain).
   * There can be gaps between domain IDs.
   * There can be gaps between bus IDs in a domain.
   * Each bus's ID is greater than its domain's ID (except for bus 0 in each domain).

The gaps between IDs means that the current PCI address set buses array [2] would have to be replaced with a different data structure.  I would probably use nested GHashTables, where the first dimension is the domain and the second dimension is the bus.

The default names generated for QEMU would continue to be "pci.N" (or "pcie.N") where N is the PCI controller's index number.

The PCI specification says that bus IDs are 8 bits, but libvirt bus IDs are independent of the bus IDs seen by the guest so this limitation doesn't need to exist in libvirt configs.  As long as each domain has no more than 256 buses, it should be possible to support arbitrarily large bus IDs.  (I doubt this will ever be an issue in practice, but I thought it would be worth mentioning.)

Thoughts?

Thanks,
Richard

[1] https://gitlab.com/libvirt/libvirt/-/issues/360
[2] https://gitlab.com/libvirt/libvirt/-/blob/v8.7.0/src/conf/domain_addr.h#L126-127
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20220915/38798cbd/attachment-0001.sig>