[libvirt] new libvirt "pci" controller type and pcie/q35 (was Re: [PATCH 4/7] add pci-bridge controller type)

[Laine Stump]

On 04/08/2013 03:37 PM, Michael S. Tsirkin wrote:
> On Mon, Apr 08, 2013 at 03:32:07PM -0400, Laine Stump wrote:
>> On 04/08/2013 12:48 PM, Daniel P. Berrange wrote:
>>> On Mon, Apr 08, 2013 at 12:37:49PM -0400, Laine Stump wrote:
>>>> I think we're starting to get closer to the concrete problem that's
>>>> bothering me. As I understand it (and again - "what I understand" has
>>>> repeatedly been shown to be incorrect in this thread :-):
>>>>
>>>> * Ihere are multiple different types of devices that provide a bus with
>>>> 1 or more "slots" that PCI devices (e.g., the virtio-net-pci device, the
>>>> e1000 network device, etc) can be plugged into.
>>>>
>>>> * In the config for those devices, there is a required (auto-generated
>>>> if not explicitly provided) <address> element that indicates what
>>>> controller that device is plugged into e.g.:
>>>>
>>>>     <interface type='direct'>
>>>>       ...
>>>>       <address type='pci' domain='0' bus='0' slot='3' function='0'/>
>>>>       ...
>>>>     </interface>
>>>>
>>>> * domain is always hardcoded to 0, and in the past bus was also always
>>>> hardcoded to 0 because until now there has only been a single place
>>>> where PCI devices could be connected - the builtin pci.0 bus, which is a
>>>> part of the basic "pc" (and some others) virtual machine and provides 32
>>>> slots.
>>>>
>>>> * Now we are adding the ability to define new PCI buses, for now just a
>>>> single kind - a pci-bridge controller, which itself must connect to an
>>>> existing PCI slot, and provides 32 new PCI slots. But in the future
>>>> there will be more different types of controllers that provide one or
>>>> more PCI slots where PCI devices/controllers can be plugged in.
>>>>
>>>> * In these patches adding support for pci-bridge, we are making the
>>>> assumption that there is a 1:1 correspondence between the "index='n'"
>>>> attribute of the pci-bridge controller and the "bus='n'" attribute of
>>>> the <address> element in devices that will be plugged into that
>>>> controller. So for example if we have:
>>>>
>>>>
>>>>    <controller type='pci-bridge' index='1'>
>>>>       <address type='pci' domain='0' bus='0' slot='10' function='0'/>
>>>>    </controller>
>>>>
>>>> and then change the <interface> definition above to say "bus='1'", that
>>>> interface device will plug into this new bus at slot 3.
>>>>
>>>> * So let's assume that we add a new controller called "dmi-to-pci-bridge":
>>>>
>>>>   <controller type='dmi-to-pci-bridge' index='0'/>
>>>>
>>>> Ignoring for now the question of what address we give in the definition
>>>> of *this* device (which is itself problematic - do we need a new "pcie"
>>>> address type?), if some device is then defined with
>>>>
>>>>
>>>>    <address type='pci bus='0' .../>
>>>>
>>>> How do we differentiate between that meaning "the pci-ptp controller
>>>> that is index='0'" and "the pci-bridge controller that is index='0'"? Do
>>>> we need to expand our <address> element further? If, as I think you
>>>> suggest, we have multiple different kinds of controllers that provide
>>>> PCI slots, each with its own namespace, the current pci address element
>>>> is inadequate to unambiguously describe where a pci device should be
>>>> plugged in.
>>> Hmm yes, you're right - as long as we only have   <adress type='pci'>
>>> then all <controller> elements should use  type='pci' too, and we should
>>> just distinguish based on the model name of the controller. So ignore
>>> my previous suggestion to have 'pci-bridge' and 'pci-root' types, we
>>> can only use  type='pci' on <controller> elements.
>> Okay, so that means we preserve the correlation between
>>
>>    <controller type='pci' index='1'>
>>
>> and
>>
>>    <address type='pci' bus='1' ..../>
>>
>>
>> Should the <controller> device use, e.g. <model type='pci-bridge'/> for
>> the model, as is done for <interface> devices? One notable difference is
>> that in the case of <interface> (with the exception of "<model
>> type='virtio'/>"), the model isn't used for anything except passing
>> directly through to qemu (and very recently validating against a list of
>> known interface models), while in the case of controllers with
>> type='pci', different models will have different rules about what they
>> can connect to and what can connect to them, and they will affect what
>> is valid in other devices.
>>
>> An example on a "pc" machinetype that has the builtin PCI bus, one extra
>> pci-pci bridge, and an interface device plugged into slot 3 of the
>> pci-bridge:
>>
>>    <controller type='pci' index='0'>
>>      <model type='pci-root'/> <!-- builtin pci bus -->
>>    </controller>
>>    <controller type='pci' index='1'>
>>      <model type='pci-bridge'/>
>>    </controller>
>>    <interface type='direct'>
>>      ...
>>      <address type='pci' bus='1' slot='3'/>
>>    </controller>
>>
>> And for a q35 machinetype that has the root pcie, an i82801b11-bridge
>> connected to slot 1e of that, a pci bridge connected to slot 1 of the
>> i82801b11-bridge, and an interface plugged into slot 3 of the pci-bridge:
>>
>>    <controller type='pci' index='0'>
>>      <model type='pcie-root'/>
>>    </controller>
>>    <controller type='pci' index='1'>
>>      <model type='i82801b11-bridge'/> <!-- [*] -->
>>      <address type='pci' bus='0' slot='0x1e'/>
>>    </controller>
>>    <controller type='pci' index='2'>
>>      <model type='pci-bridge'/>
>>      <address type='pci' bus='1' slot='1'/>
>>    </controller>
>>    <interface type='direct'>
>>      ...
>>      <address type='pci' bus='2' slot='3'/>
>>    </controller>
>>
>> (note that controllers with model='(pci|pcie)-root' will not have any
>> <address> element, because they exist in the basic machine so we don't
>> need to connect them to anywhere.)
>>
>> (also note that it might happen that the bus number in libvirt's config
>> will correspond to the bus numbering that shows up in the guest OS, but
>> that will just be a happy coincidence)
>>
>> Does this make sense?
> Confused. So why are you using bus numbers at all?

I'm kind of wondering that too :-) That's why I suggested naming each
controller that provided a connection point/bus and using the name
instead of a number.

I think initially the pci address element was given a bus number because
it made the config in libvirt look similar to the view of the devices
from the guest (and also that's what is necessary for a PCI address on
the host side, e.g. when doing PCI passthrough); maybe now we're only
considering keeping it for backwards compatibility and consistency with
the hostside address element? Or does using numbers fit better with the
paradigms for other hypervisors? (doesn't seem like that should be a
problem, because the name can be completely internal to libvirt, and
translated into what each different hypervisor needs to have).