[libvirt] new libvirt "pci" controller type and pcie/q35 (was Re: [PATCH 4/7] add pci-bridge controller type)

[Laine Stump]

On 04/10/2013 05:26 AM, Daniel P. Berrange wrote:
> On Tue, Apr 09, 2013 at 04:06:06PM -0400, Laine Stump wrote:
>> On 04/09/2013 04:58 AM, Daniel P. Berrange wrote:
>>> On Mon, Apr 08, 2013 at 03:32:07PM -0400, Laine Stump wrote:
>>> Actually I do wonder if we should reprent a PCI root as two
>>> <controller> elements, one representing the actual PCI root
>>> device, and the other representing the host bridge that is
>>> built-in.
>>>
>>> Also we should use the actual model names, not 'pci-root' or
>>> 'pcie-root' but rather i440FX for "pc" machine type, and whatever
>>> the q35 model name is.
>>>
>>>  - One PCI root with built-in PCI bus (ie todays' setup)
>>>
>>>    <controller type="pci-root" index="0">
>>>      <model name="i440FX"/>
>>>    </controller>
>>>    <controller type="pci" index="0"> <!-- Host bridge -->
>>>      <address type='pci' domain='0' bus='0' slot='0''/>
>> Isn't this saying that the bridge connects to itself? (since bus 0 is
>> this bus)
>>
>> I understand (again, possibly wrongly) that the builtin PCI bus connects
>> to the chipset using its own slot 0 (that's why it's reserved), but
>> that's its address on itself. How is this bridge associated with the
>> pci-root?
>>
>> Ah, I *think* I see it - the domain attribute of the pci controller is
>> matched to the index of the pci-root controller, correct? But there's
>> still something strange about the <address> of the pci controller being
>> self-referential.
> Yes, the index of the pci-root matches the 'domain' of <address>


Okay, then the way that libvirt differentiates between a pci bridge that
is connected to the root, and one that is connected to a slot of another
bridge is 1) the "bus" attribute of the bridge's <address> matches the
"index" attribute of the bridge itself, and 2) "slot" is always 0. Correct?

(The corollary of this is that if slot == 0 and bus != index, or bus ==
index and slot != 0, it is a configuration error).

I'm still unclear on the usefulness of the pci-root controller though -
all the necessary information is contained in the pci controller, except
for the type of root. But in the case of pcie root, I think you're not
allowed to connect a standard bridge to it, only a "dmi-to-pci-bridge"
(i82801b11-bridge)


>>>    </controller>
>>>    <interface type='direct'>
>>>       ...
>>>      <address type='pci' domain='0' bus='0' slot='3'/>
>>>    </controller>
>>>
>>>  - One PCI root with built-in PCI bus and extra PCI bridge
>>>
>>>    <controller type="pci-root" index="0">
>>>      <model name="i440FX"/>
>>>    </controller>
>>>    <controller type="pci" index="0"> <!-- Host bridge -->
>>>      <address type='pci' domain='0' bus='0' slot='0'/>
>>>    </controller>
>>>    <controller type="pci" index="1"> <!-- Additional bridge -->
>>>      <address type='pci' domain='0' bus='0' slot='1'/>
>>>    </controller>
>>>    <interface type='direct'>
>>>       ...
>>>      <address type='pci' domain='0' bus='1' slot='3'/>
>>>    </controller>
>>>
>>>  - One PCI root with built-in PCI bus, PCI-E bus and and extra PCI bridge
>>>    (ie possible q35 setup)
>> Why would a q35 machine have an i440FX pci-root?
> It shouldn't, that's a typo
>
>>>    <controller type="pci-root" index="0">
>>>      <model name="i440FX"/>
>>>    </controller>
>>>    <controller type="pci" index="0"> <!-- Host bridge -->
>>>      <address type='pci' domain='0' bus='0' slot='0'/>
>>>    </controller>
>>>    <controller type="pci" index="1"> <!-- Additional bridge -->
>>>      <address type='pci' domain='0' bus='0' slot='1'/>
>>>    </controller>
>>>    <controller type="pci" index="1"> <!-- Additional bridge -->
>>>      <address type='pci' domain='0' bus='0' slot='1'/>
>>>    </controller>
>> I think you did a cut-paste here and intended to change something, but
>> didn't - those two bridges are identical.
> Yep, the slot should be 2 in the second one
>
>>>    <interface type='direct'>
>>>       ...
>>>      <address type='pci' domain='0' bus='1' slot='3'/>
>>>    </controller>
>>>
>>> So if we later allowed for mutiple PCI roots, then we'd have something
>>> like
>>>
>>>    <controller type="pci-root" index="0">
>>>      <model name="i440FX"/>
>>>    </controller>
>>>    <controller type="pci-root" index="1">
>>>      <model name="i440FX"/>
>>>    </controller>
>>>    <controller type="pci" index="0"> <!-- Host bridge 1 -->
>>>      <address type='pci' domain='0' bus='0' slot='0''/>
>>>    </controller>
>>>    <controller type="pci" index="0"> <!-- Host bridge 2 -->
>>>      <address type='pci' domain='1' bus='0' slot='0''/>
>>>    </controller>
>>>    <interface type='direct'> <!-- NIC on host bridge 2 -->
>>>       ...
>>>      <address type='pci' domain='1' bus='0' slot='3'/>
>>>    </controller>
>>>
>>>
>>> NB this means that 'index' values can be reused against the
>>> <controller>, provided they are setup on different pci-roots.
>>>
>>>> (also note that it might happen that the bus number in libvirt's config
>>>> will correspond to the bus numbering that shows up in the guest OS, but
>>>> that will just be a happy coincidence)
>>>>
>>>> Does this make sense?
>>> Yep, I think we're fairly close.
>> What about the other types of pci controllers that are used by PCIe? We
>> should make sure they fit in this model before we settle on it.
> What do they do ?


Although I've asked that question, and had it answered, several times
now, without a tabular list, I'm still unable to get the answer to that
question clear in my mind :-/

Here's the controllers that aw has talked about (in a recent response on
a different sub-thread):


On 04/05/2013 03:26 PM, Alex Williamson wrote:
> For PCIe, we create new buses for root ports (ioh3420), upstream switch
> ports (xio3130-upstream), downstream switch ports (xio3130-downstream),
> and the dmi-to-pci bridge (i82801b11-bridge).  For PCI, PCI-to-PCI
> bridges create new buses (pci-bridge and dec-21154).


Alex: what do each of these connect to, and what can be connected to them?

root-port (ioh3420)
upstream-switch-port (xio3130-upstream)
downstream-switch-port (xio3130-downstream)
dmi-to-pci-bridge (i82801b11-bridge)


We already know/understand this one: pci-bridge; is this identical in
behavior/function? dec-21154


> One of my goals is to move us away from emulation of specific chips and
> create more devices like pci-bridge that adhere to the standard, but
> don't try to emulate a specific device.  Then we might have "root-port",
> "pcie-upstream-switch-port", "pcie-downstream-switch-port", and
> "dmi-to-pci-bridge" (none of these names have been discussed).