[libvirt] new libvirt "pci" controller type and pcie/q35 (was Re: [PATCH 4/7] add pci-bridge controller type)

Thu Apr 11 11:23:05 UTC 2013

On Thu, Apr 11, 2013 at 07:03:56AM -0400, Laine Stump wrote:
> On 04/10/2013 05:26 AM, Daniel P. Berrange wrote:
> > On Tue, Apr 09, 2013 at 04:06:06PM -0400, Laine Stump wrote:
> >> On 04/09/2013 04:58 AM, Daniel P. Berrange wrote:
> >>> On Mon, Apr 08, 2013 at 03:32:07PM -0400, Laine Stump wrote:
> >>> Actually I do wonder if we should reprent a PCI root as two
> >>> <controller> elements, one representing the actual PCI root
> >>> device, and the other representing the host bridge that is
> >>> built-in.
> >>>
> >>> Also we should use the actual model names, not 'pci-root' or
> >>> 'pcie-root' but rather i440FX for "pc" machine type, and whatever
> >>> the q35 model name is.
> >>>
> >>>  - One PCI root with built-in PCI bus (ie todays' setup)
> >>>
> >>>    <controller type="pci-root" index="0">
> >>>      <model name="i440FX"/>
> >>>    </controller>
> >>>    <controller type="pci" index="0"> <!-- Host bridge -->
> >>>      <address type='pci' domain='0' bus='0' slot='0''/>
> >> Isn't this saying that the bridge connects to itself? (since bus 0 is
> >> this bus)
> >>
> >> I understand (again, possibly wrongly) that the builtin PCI bus connects
> >> to the chipset using its own slot 0 (that's why it's reserved), but
> >> that's its address on itself. How is this bridge associated with the
> >> pci-root?
> >>
> >> Ah, I *think* I see it - the domain attribute of the pci controller is
> >> matched to the index of the pci-root controller, correct? But there's
> >> still something strange about the <address> of the pci controller being
> >> self-referential.
> > Yes, the index of the pci-root matches the 'domain' of <address>
> 
> 
> Okay, then the way that libvirt differentiates between a pci bridge that
> is connected to the root, and one that is connected to a slot of another
> bridge is 1) the "bus" attribute of the bridge's <address> matches the
> "index" attribute of the bridge itself, and 2) "slot" is always 0. Correct?
> 
> (The corollary of this is that if slot == 0 and bus != index, or bus ==
> index and slot != 0, it is a configuration error).
> 
> I'm still unclear on the usefulness of the pci-root controller though -
> all the necessary information is contained in the pci controller, except
> for the type of root. But in the case of pcie root, I think you're not
> allowed to connect a standard bridge to it, only a "dmi-to-pci-bridge"
> (i82801b11-bridge)

Yes you can connect a pci bridge to pcie-root.
It's represented as a root complex integrated device.

> >>>    </controller>
> >>>    <interface type='direct'>
> >>>       ...
> >>>      <address type='pci' domain='0' bus='0' slot='3'/>
> >>>    </controller>
> >>>
> >>>  - One PCI root with built-in PCI bus and extra PCI bridge
> >>>
> >>>    <controller type="pci-root" index="0">
> >>>      <model name="i440FX"/>
> >>>    </controller>
> >>>    <controller type="pci" index="0"> <!-- Host bridge -->
> >>>      <address type='pci' domain='0' bus='0' slot='0'/>
> >>>    </controller>
> >>>    <controller type="pci" index="1"> <!-- Additional bridge -->
> >>>      <address type='pci' domain='0' bus='0' slot='1'/>
> >>>    </controller>
> >>>    <interface type='direct'>
> >>>       ...
> >>>      <address type='pci' domain='0' bus='1' slot='3'/>
> >>>    </controller>
> >>>
> >>>  - One PCI root with built-in PCI bus, PCI-E bus and and extra PCI bridge
> >>>    (ie possible q35 setup)
> >> Why would a q35 machine have an i440FX pci-root?
> > It shouldn't, that's a typo
> >
> >>>    <controller type="pci-root" index="0">
> >>>      <model name="i440FX"/>
> >>>    </controller>
> >>>    <controller type="pci" index="0"> <!-- Host bridge -->
> >>>      <address type='pci' domain='0' bus='0' slot='0'/>
> >>>    </controller>
> >>>    <controller type="pci" index="1"> <!-- Additional bridge -->
> >>>      <address type='pci' domain='0' bus='0' slot='1'/>
> >>>    </controller>
> >>>    <controller type="pci" index="1"> <!-- Additional bridge -->
> >>>      <address type='pci' domain='0' bus='0' slot='1'/>
> >>>    </controller>
> >> I think you did a cut-paste here and intended to change something, but
> >> didn't - those two bridges are identical.
> > Yep, the slot should be 2 in the second one
> >
> >>>    <interface type='direct'>
> >>>       ...
> >>>      <address type='pci' domain='0' bus='1' slot='3'/>
> >>>    </controller>
> >>>
> >>> So if we later allowed for mutiple PCI roots, then we'd have something
> >>> like
> >>>
> >>>    <controller type="pci-root" index="0">
> >>>      <model name="i440FX"/>
> >>>    </controller>
> >>>    <controller type="pci-root" index="1">
> >>>      <model name="i440FX"/>
> >>>    </controller>
> >>>    <controller type="pci" index="0"> <!-- Host bridge 1 -->
> >>>      <address type='pci' domain='0' bus='0' slot='0''/>
> >>>    </controller>
> >>>    <controller type="pci" index="0"> <!-- Host bridge 2 -->
> >>>      <address type='pci' domain='1' bus='0' slot='0''/>
> >>>    </controller>
> >>>    <interface type='direct'> <!-- NIC on host bridge 2 -->
> >>>       ...
> >>>      <address type='pci' domain='1' bus='0' slot='3'/>
> >>>    </controller>
> >>>
> >>>
> >>> NB this means that 'index' values can be reused against the
> >>> <controller>, provided they are setup on different pci-roots.
> >>>
> >>>> (also note that it might happen that the bus number in libvirt's config
> >>>> will correspond to the bus numbering that shows up in the guest OS, but
> >>>> that will just be a happy coincidence)
> >>>>
> >>>> Does this make sense?
> >>> Yep, I think we're fairly close.
> >> What about the other types of pci controllers that are used by PCIe? We
> >> should make sure they fit in this model before we settle on it.
> > What do they do ?
> 
> 
> Although I've asked that question, and had it answered, several times
> now, without a tabular list, I'm still unable to get the answer to that
> question clear in my mind :-/
> 
> Here's the controllers that aw has talked about (in a recent response on
> a different sub-thread):
> 
> 
> On 04/05/2013 03:26 PM, Alex Williamson wrote:
> > For PCIe, we create new buses for root ports (ioh3420), upstream switch
> > ports (xio3130-upstream), downstream switch ports (xio3130-downstream),
> > and the dmi-to-pci bridge (i82801b11-bridge).  For PCI, PCI-to-PCI
> > bridges create new buses (pci-bridge and dec-21154).
> 
> 
> Alex: what do each of these connect to, and what can be connected to them?
> 
> root-port (ioh3420)

this is connected to the root bus.
any express device connects here (at most one device
but it can be multifunction)

> upstream-switch-port (xio3130-upstream)

this connects to a root port or downstream port,
as any express device. only downstream ports connect here.

> downstream-switch-port (xio3130-downstream)

this connects to the upstream port.
any express device connects here (at most one device
but it can be multifunction)

> dmi-to-pci-bridge (i82801b11-bridge)

this connects to the root bus.
any pci (non express) device connects here

> 
> We already know/understand this one: pci-bridge; is this identical in
> behavior/function? dec-21154

Yes, only dec pretends it's an intel bridge device, pci-bridge
tells guest it's a generic device.
Does not seem to matter with any guest I tried.

> 
> > One of my goals is to move us away from emulation of specific chips and
> > create more devices like pci-bridge that adhere to the standard, but
> > don't try to emulate a specific device.  Then we might have "root-port",
> > "pcie-upstream-switch-port", "pcie-downstream-switch-port", and
> > "dmi-to-pci-bridge" (none of these names have been discussed).  