[libvirt] new libvirt "pci" controller type and pcie/q35 (was Re: [PATCH 4/7] add pci-bridge controller type)

[Laine Stump]

On 04/09/2013 04:58 AM, Daniel P. Berrange wrote:
> On Mon, Apr 08, 2013 at 03:32:07PM -0400, Laine Stump wrote:
>> On 04/08/2013 12:48 PM, Daniel P. Berrange wrote:
>>> On Mon, Apr 08, 2013 at 12:37:49PM -0400, Laine Stump wrote:
>>>> I think we're starting to get closer to the concrete problem that's
>>>> bothering me. As I understand it (and again - "what I understand" has
>>>> repeatedly been shown to be incorrect in this thread :-):
>>>>
>>>> * Ihere are multiple different types of devices that provide a bus with
>>>> 1 or more "slots" that PCI devices (e.g., the virtio-net-pci device, the
>>>> e1000 network device, etc) can be plugged into.
>>>>
>>>> * In the config for those devices, there is a required (auto-generated
>>>> if not explicitly provided) <address> element that indicates what
>>>> controller that device is plugged into e.g.:
>>>>
>>>>     <interface type='direct'>
>>>>       ...
>>>>       <address type='pci' domain='0' bus='0' slot='3' function='0'/>
>>>>       ...
>>>>     </interface>
>>>>
>>>> * domain is always hardcoded to 0, and in the past bus was also always
>>>> hardcoded to 0 because until now there has only been a single place
>>>> where PCI devices could be connected - the builtin pci.0 bus, which is a
>>>> part of the basic "pc" (and some others) virtual machine and provides 32
>>>> slots.
>>>>
>>>> * Now we are adding the ability to define new PCI buses, for now just a
>>>> single kind - a pci-bridge controller, which itself must connect to an
>>>> existing PCI slot, and provides 32 new PCI slots. But in the future
>>>> there will be more different types of controllers that provide one or
>>>> more PCI slots where PCI devices/controllers can be plugged in.
>>>>
>>>> * In these patches adding support for pci-bridge, we are making the
>>>> assumption that there is a 1:1 correspondence between the "index='n'"
>>>> attribute of the pci-bridge controller and the "bus='n'" attribute of
>>>> the <address> element in devices that will be plugged into that
>>>> controller. So for example if we have:
>>>>
>>>>
>>>>    <controller type='pci-bridge' index='1'>
>>>>       <address type='pci' domain='0' bus='0' slot='10' function='0'/>
>>>>    </controller>
>>>>
>>>> and then change the <interface> definition above to say "bus='1'", that
>>>> interface device will plug into this new bus at slot 3.
>>>>
>>>> * So let's assume that we add a new controller called "dmi-to-pci-bridge":
>>>>
>>>>   <controller type='dmi-to-pci-bridge' index='0'/>
>>>>
>>>> Ignoring for now the question of what address we give in the definition
>>>> of *this* device (which is itself problematic - do we need a new "pcie"
>>>> address type?), if some device is then defined with
>>>>
>>>>
>>>>    <address type='pci bus='0' .../>
>>>>
>>>> How do we differentiate between that meaning "the pci-ptp controller
>>>> that is index='0'" and "the pci-bridge controller that is index='0'"? Do
>>>> we need to expand our <address> element further? If, as I think you
>>>> suggest, we have multiple different kinds of controllers that provide
>>>> PCI slots, each with its own namespace, the current pci address element
>>>> is inadequate to unambiguously describe where a pci device should be
>>>> plugged in.
>>> Hmm yes, you're right - as long as we only have   <adress type='pci'>
>>> then all <controller> elements should use  type='pci' too, and we should
>>> just distinguish based on the model name of the controller. So ignore
>>> my previous suggestion to have 'pci-bridge' and 'pci-root' types, we
>>> can only use  type='pci' on <controller> elements.
>> Okay, so that means we preserve the correlation between
>>
>>    <controller type='pci' index='1'>
>>
>> and
>>
>>    <address type='pci' bus='1' ..../>
>>
>>
>> Should the <controller> device use, e.g. <model type='pci-bridge'/> for
>> the model, as is done for <interface> devices? One notable difference is
>> that in the case of <interface> (with the exception of "<model
>> type='virtio'/>"), the model isn't used for anything except passing
>> directly through to qemu (and very recently validating against a list of
>> known interface models), while in the case of controllers with
>> type='pci', different models will have different rules about what they
>> can connect to and what can connect to them, and they will affect what
>> is valid in other devices.
>>
>> An example on a "pc" machinetype that has the builtin PCI bus, one extra
>> pci-pci bridge, and an interface device plugged into slot 3 of the
>> pci-bridge:
>>
>>    <controller type='pci' index='0'>
>>      <model type='pci-root'/> <!-- builtin pci bus -->
>>    </controller>
>>    <controller type='pci' index='1'>
>>      <model type='pci-bridge'/>
>>    </controller>
>>    <interface type='direct'>
>>      ...
>>      <address type='pci' bus='1' slot='3'/>
>>    </controller>
>>
>> And for a q35 machinetype that has the root pcie, an i82801b11-bridge
>> connected to slot 1e of that, a pci bridge connected to slot 1 of the
>> i82801b11-bridge, and an interface plugged into slot 3 of the pci-bridge:
>>
>>    <controller type='pci' index='0'>
>>      <model type='pcie-root'/>
>>    </controller>
>>    <controller type='pci' index='1'>
>>      <model type='i82801b11-bridge'/> <!-- [*] -->
>>      <address type='pci' bus='0' slot='0x1e'/>
>>    </controller>
>>    <controller type='pci' index='2'>
>>      <model type='pci-bridge'/>
>>      <address type='pci' bus='1' slot='1'/>
>>    </controller>
>>    <interface type='direct'>
>>      ...
>>      <address type='pci' bus='2' slot='3'/>
>>    </controller>
>>
>> (note that controllers with model='(pci|pcie)-root' will not have any
>> <address> element, because they exist in the basic machine so we don't
>> need to connect them to anywhere.)
> Actually I do wonder if we should reprent a PCI root as two
> <controller> elements, one representing the actual PCI root
> device, and the other representing the host bridge that is
> built-in.
>
> Also we should use the actual model names, not 'pci-root' or
> 'pcie-root' but rather i440FX for "pc" machine type, and whatever
> the q35 model name is.
>
>  - One PCI root with built-in PCI bus (ie todays' setup)
>
>    <controller type="pci-root" index="0">
>      <model name="i440FX"/>
>    </controller>
>    <controller type="pci" index="0"> <!-- Host bridge -->
>      <address type='pci' domain='0' bus='0' slot='0''/>

Isn't this saying that the bridge connects to itself? (since bus 0 is
this bus)

I understand (again, possibly wrongly) that the builtin PCI bus connects
to the chipset using its own slot 0 (that's why it's reserved), but
that's its address on itself. How is this bridge associated with the
pci-root?

Ah, I *think* I see it - the domain attribute of the pci controller is
matched to the index of the pci-root controller, correct? But there's
still something strange about the <address> of the pci controller being
self-referential.


>    </controller>
>    <interface type='direct'>
>       ...
>      <address type='pci' domain='0' bus='0' slot='3'/>
>    </controller>
>
>  - One PCI root with built-in PCI bus and extra PCI bridge
>
>    <controller type="pci-root" index="0">
>      <model name="i440FX"/>
>    </controller>
>    <controller type="pci" index="0"> <!-- Host bridge -->
>      <address type='pci' domain='0' bus='0' slot='0'/>
>    </controller>
>    <controller type="pci" index="1"> <!-- Additional bridge -->
>      <address type='pci' domain='0' bus='0' slot='1'/>
>    </controller>
>    <interface type='direct'>
>       ...
>      <address type='pci' domain='0' bus='1' slot='3'/>
>    </controller>
>
>  - One PCI root with built-in PCI bus, PCI-E bus and and extra PCI bridge
>    (ie possible q35 setup)

Why would a q35 machine have an i440FX pci-root?

>
>    <controller type="pci-root" index="0">
>      <model name="i440FX"/>
>    </controller>
>    <controller type="pci" index="0"> <!-- Host bridge -->
>      <address type='pci' domain='0' bus='0' slot='0'/>
>    </controller>
>    <controller type="pci" index="1"> <!-- Additional bridge -->
>      <address type='pci' domain='0' bus='0' slot='1'/>
>    </controller>
>    <controller type="pci" index="1"> <!-- Additional bridge -->
>      <address type='pci' domain='0' bus='0' slot='1'/>
>    </controller>

I think you did a cut-paste here and intended to change something, but
didn't - those two bridges are identical.

>    <interface type='direct'>
>       ...
>      <address type='pci' domain='0' bus='1' slot='3'/>
>    </controller>
>
> So if we later allowed for mutiple PCI roots, then we'd have something
> like
>
>    <controller type="pci-root" index="0">
>      <model name="i440FX"/>
>    </controller>
>    <controller type="pci-root" index="1">
>      <model name="i440FX"/>
>    </controller>
>    <controller type="pci" index="0"> <!-- Host bridge 1 -->
>      <address type='pci' domain='0' bus='0' slot='0''/>
>    </controller>
>    <controller type="pci" index="0"> <!-- Host bridge 2 -->
>      <address type='pci' domain='1' bus='0' slot='0''/>
>    </controller>
>    <interface type='direct'> <!-- NIC on host bridge 2 -->
>       ...
>      <address type='pci' domain='1' bus='0' slot='3'/>
>    </controller>
>
>
> NB this means that 'index' values can be reused against the
> <controller>, provided they are setup on different pci-roots.
>
>> (also note that it might happen that the bus number in libvirt's config
>> will correspond to the bus numbering that shows up in the guest OS, but
>> that will just be a happy coincidence)
>>
>> Does this make sense?
> Yep, I think we're fairly close.

What about the other types of pci controllers that are used by PCIe? We
should make sure they fit in this model before we settle on it.