[libvirt] new libvirt "pci" controller type and pcie/q35 (was Re: [PATCH 4/7] add pci-bridge controller type)
Laine Stump
laine at laine.org
Fri Apr 12 15:46:15 UTC 2013
On 04/11/2013 07:23 AM, Michael S. Tsirkin wrote:
> On Thu, Apr 11, 2013 at 07:03:56AM -0400, Laine Stump wrote:
>> On 04/10/2013 05:26 AM, Daniel P. Berrange wrote:
>>> On Tue, Apr 09, 2013 at 04:06:06PM -0400, Laine Stump wrote:
>>>> On 04/09/2013 04:58 AM, Daniel P. Berrange wrote:
>>>>> On Mon, Apr 08, 2013 at 03:32:07PM -0400, Laine Stump wrote:
>>>>> Actually I do wonder if we should reprent a PCI root as two
>>>>> <controller> elements, one representing the actual PCI root
>>>>> device, and the other representing the host bridge that is
>>>>> built-in.
>>>>>
>>>>> Also we should use the actual model names, not 'pci-root' or
>>>>> 'pcie-root' but rather i440FX for "pc" machine type, and whatever
>>>>> the q35 model name is.
>>>>>
>>>>> - One PCI root with built-in PCI bus (ie todays' setup)
>>>>>
>>>>> <controller type="pci-root" index="0">
>>>>> <model name="i440FX"/>
>>>>> </controller>
>>>>> <controller type="pci" index="0"> <!-- Host bridge -->
>>>>> <address type='pci' domain='0' bus='0' slot='0''/>
>>>> Isn't this saying that the bridge connects to itself? (since bus 0 is
>>>> this bus)
>>>>
>>>> I understand (again, possibly wrongly) that the builtin PCI bus connects
>>>> to the chipset using its own slot 0 (that's why it's reserved), but
>>>> that's its address on itself. How is this bridge associated with the
>>>> pci-root?
>>>>
>>>> Ah, I *think* I see it - the domain attribute of the pci controller is
>>>> matched to the index of the pci-root controller, correct? But there's
>>>> still something strange about the <address> of the pci controller being
>>>> self-referential.
>>> Yes, the index of the pci-root matches the 'domain' of <address>
>>
>> Okay, then the way that libvirt differentiates between a pci bridge that
>> is connected to the root, and one that is connected to a slot of another
>> bridge is 1) the "bus" attribute of the bridge's <address> matches the
>> "index" attribute of the bridge itself, and 2) "slot" is always 0. Correct?
>>
>> (The corollary of this is that if slot == 0 and bus != index, or bus ==
>> index and slot != 0, it is a configuration error).
>>
>> I'm still unclear on the usefulness of the pci-root controller though -
>> all the necessary information is contained in the pci controller, except
>> for the type of root. But in the case of pcie root, I think you're not
>> allowed to connect a standard bridge to it, only a "dmi-to-pci-bridge"
>> (i82801b11-bridge)
> Yes you can connect a pci bridge to pcie-root.
> It's represented as a root complex integrated device.
ARGHH!! Just when I think I'm starting to understand *something* about
these devices...
(later edit: after some coaching on IRC, I *think* I've got a bit better
handle on it.)
>>>>> </controller>
>>>>> <interface type='direct'>
>>>>> ...
>>>>> <address type='pci' domain='0' bus='0' slot='3'/>
>>>>> </controller>
>>>>>
>>>>> - One PCI root with built-in PCI bus and extra PCI bridge
>>>>>
>>>>> <controller type="pci-root" index="0">
>>>>> <model name="i440FX"/>
>>>>> </controller>
>>>>> <controller type="pci" index="0"> <!-- Host bridge -->
>>>>> <address type='pci' domain='0' bus='0' slot='0'/>
>>>>> </controller>
>>>>> <controller type="pci" index="1"> <!-- Additional bridge -->
>>>>> <address type='pci' domain='0' bus='0' slot='1'/>
>>>>> </controller>
>>>>> <interface type='direct'>
>>>>> ...
>>>>> <address type='pci' domain='0' bus='1' slot='3'/>
>>>>> </controller>
>>>>>
>>>>> - One PCI root with built-in PCI bus, PCI-E bus and and extra PCI bridge
>>>>> (ie possible q35 setup)
>>>> Why would a q35 machine have an i440FX pci-root?
>>> It shouldn't, that's a typo
>>>
>>>>> <controller type="pci-root" index="0">
>>>>> <model name="i440FX"/>
>>>>> </controller>
>>>>> <controller type="pci" index="0"> <!-- Host bridge -->
>>>>> <address type='pci' domain='0' bus='0' slot='0'/>
>>>>> </controller>
>>>>> <controller type="pci" index="1"> <!-- Additional bridge -->
>>>>> <address type='pci' domain='0' bus='0' slot='1'/>
>>>>> </controller>
>>>>> <controller type="pci" index="1"> <!-- Additional bridge -->
>>>>> <address type='pci' domain='0' bus='0' slot='1'/>
>>>>> </controller>
>>>> I think you did a cut-paste here and intended to change something, but
>>>> didn't - those two bridges are identical.
>>> Yep, the slot should be 2 in the second one
>>>
>>>>> <interface type='direct'>
>>>>> ...
>>>>> <address type='pci' domain='0' bus='1' slot='3'/>
>>>>> </controller>
>>>>>
>>>>> So if we later allowed for mutiple PCI roots, then we'd have something
>>>>> like
>>>>>
>>>>> <controller type="pci-root" index="0">
>>>>> <model name="i440FX"/>
>>>>> </controller>
>>>>> <controller type="pci-root" index="1">
>>>>> <model name="i440FX"/>
>>>>> </controller>
>>>>> <controller type="pci" index="0"> <!-- Host bridge 1 -->
>>>>> <address type='pci' domain='0' bus='0' slot='0''/>
>>>>> </controller>
>>>>> <controller type="pci" index="0"> <!-- Host bridge 2 -->
>>>>> <address type='pci' domain='1' bus='0' slot='0''/>
>>>>> </controller>
There is a problem here - within a given controller type, we will now
have the possibility of multiple controllers with the same index - the
differentiating attribute will be in the <address> subelement, which
could create some awkwardness. Maybe instead this should be handled with
a different model of pci controller, and we can add a "domain" attribute
at the toplevel rather than specifying an <address>?
>>>>> <interface type='direct'> <!-- NIC on host bridge 2 -->
>>>>> ...
>>>>> <address type='pci' domain='1' bus='0' slot='3'/>
>>>>> </controller>
>>>>>
>>>>>
>>>>> NB this means that 'index' values can be reused against the
>>>>> <controller>, provided they are setup on different pci-roots.
>>>>>
>>>>>> (also note that it might happen that the bus number in libvirt's config
>>>>>> will correspond to the bus numbering that shows up in the guest OS, but
>>>>>> that will just be a happy coincidence)
>>>>>>
>>>>>> Does this make sense?
>>>>> Yep, I think we're fairly close.
>>>> What about the other types of pci controllers that are used by PCIe? We
>>>> should make sure they fit in this model before we settle on it.
>>> What do they do ?
(The descriptions of different models below tell what each of these
other devices does; in short, they're all just some sort of electronic
Lego to help connect PCI and PCIe devices into a tree).
Okay, I'll make yet another attempt at understanding these devices, and
suggesting how they can all be described in the XML. I'm thinking that
*all* of the express hubs, switch ports, bridges, etc can be described
in xml in the manner above, i.e.:
<controller type='pci' index='n'>
<model type='xxx'/>
</controller>
and that the method for connecting a device to any of them would be by
specifying:
<address type='pci' domain='n' bus='n' slot='n' function='n'/>
Any limitations about which devices/controllers can connect to which
controllers, and how many devices can connect to any particular
controller will be derived from the <model type='xxx'/>. (And, as we've
said before, although qemu doesn't assign each of these controllers a
numeric bus id, and although we can make no guarantee that the bus id we
use for a particular controller is what will be used by the guest
BIOS/OS, it's still a convenient notation and works well with other
hypervisors as well as qemu. I'll also note that when I run lspci on an
X58-based machine I have here, *all* of the relationships between all
the devices listed below are described with simple bus:slot.function
numbers.)
Here is a list of the pci controller model types and their restrictions
(thanks to mst and aw for repeating these over and over to me; I'm sure
I still have made mistakes, but at least it's getting closer).
<controller type='pci-root'>
============================
Upstream: nothing
Downstream: only a single pci-root-bus (implied)
qemu commandline: nothing (it's implied in the q35 machinetype)
Explanation:
Each machine will have a different controller called "pci-root" as
outlined above by Daniel. Two types of pci-root will be supported:
i440FX and q35. If a pci-root is not spelled out in the config, one will
be auto-added (depending on machinetype).
An i440FX pci-root has an implicitly added pci-bridge at 0:0:0.0 (and
any bridge that has an address of slot='0' on its own bus is, by
definition, connected to a pci-root controller - the two are matched by
setting "domain" in the address of the pci-bridge to "index" of the
pci-root). This bridge can only have PCI devices added.
A q35 pci-root also implies a different kind of pci-bridge device - one
that can only have PCIe devices/controllers attached, but is otherwise
identical to the pci-bridge added for i440FX. This bus will be called
"root-bus" (Note that there are generally followed conventions for what
can be connected to which slot on this bus, and we will probably follow
those conventions when building a machine, *but* we will not hardcode
this convention into libvirt; each q35 machine will be an empty slate)
<controller type='pci'>
=======================
This will be used for *all* of the following controller devices
supported by qemu:
<model type='pcie-root-bus'/> (implicit/integrated)
----------------------------
Upstream: connect to pci-root controller *only*
Downstream: 32 slots, PCIe devices only, no hotplug.
qemu commandline: nothing (implicit in the q35-* machinetype)
This controller is the bus described above that connects to a q35's
pci-root, and provides places for PCIe devices to connect. Examples are
root-ports, dmi-to-pci-bridges sata controllers, integrated
sound/usb/ethernet devices (do any of those that can be connected to the
pcie-root-bus exist yet?).
There is only one of these controllers, and it will *always* be
index='0', and will always have the following address:
<address type='pci' domain='0' bus='0' slot='0' function='0'/>
<model type='root-port'/> (ioh3420)
-------------------------
Upstream: PCIe, connect to pcie-root-bus *only* (?)
Downstream: 1 slot, PCIe devices only (?)
qemu commandline: -device ioh3420,...
These can only connect to the "pcie-root-bus" of of a q35 (implying that
this bus will need to have a different model name than the simple
"pci-bridge"
<model type='dmi-to-pci-bridge'/> (i82801b11-bridge)
---------------------------------
(btw, what does "dmi" mean?)
Upstream: pcie-root-bus *only*
Downstream: 32 slots, any PCI device, no hotplug (?)
qemu commandline: -device i82801b11-bridge,...
<model type='upstream-switch-port'/> (x3130-upstream)
------------------------------------
Upstream: PCIe, connect to pcie-root-bus, root-port, or
downstream-switch-port (?)
Downstream: 32 slots, connect *only* to downstream-switch-port
qemu-commandline: -device x3130-upstream
This is the upper side of a switch that can multiplex multiple devices
onto a single port. It's only useful when one or more downstream switch
ports are connected to it.
<model type='downstream-switch-port'/> (xio3130-downstream)
--------------------------------------
Upstream: connect *only* to upstream-switch-port
Downstream: 1 slot, any PCIe device
qemu commandline: -device xio3130-downstream
You can connect one or more of these to an upstream-switch-port in order
to effectively plug multiple devices into a single PCIe port.
<model type='pci-bridge'/> (pci-bridge)
--------------------------
Upstream: PCI, connect to 1) pci-root, 2) dmi-to-pci-bridge, 3)
another pci-bridge
Downstream: any PCI device, 32 slots
qemu commandline: -device pci-bridge,...
This differs from dmi-to-pci-bridge in that its upstream connection is
PCI rather than PCIe (so it will work on an i440FX system, which has no
root PCIe bus) and that hotplug is supported. In general, if a guest
will have any PCI devices, one of these controllers should be added, and
===============================================================
Comment: I'm not quite convinced that we really need the separate
"pci-root" device. Since 1) every pci-root will *always* have either a
pcie-root-bus or a pci-bridge connected to it, 2) the pci-root-bus will
only ever be connected to the pci-root, and 3) the pci-bridge that
connects to it will need special handling within the pci-bridge case
anyway, why not:
1) eliminate the separate pci-root controller type
2) within <controller type='pci'>, a new <model type='pci-root-bus'/>
will be added.
3) a pcie-root-bus will automatically be added for q35 machinetypes, and
pci-root-bus for any machinetype that supports a PCI bus (e.g. "pc-*")
4) model type='pci-root-bus' will behave like pci-bridge, except that it
will be an implicit device (nothing on qemu commandline) and it won't
need an <address> element (neither will pcie-root-bus).
5) to support multiple domains, we can simply add a "domain" attribute
to the toplevel of controller.
More information about the libvir-list
mailing list