[libvirt] new libvirt "pci" controller type and pcie/q35 (was Re: [PATCH 4/7] add pci-bridge controller type)
Laine Stump
laine at laine.org
Mon Apr 15 20:09:13 UTC 2013
On 04/15/2013 06:29 AM, Daniel P. Berrange wrote:
> On Fri, Apr 12, 2013 at 11:46:15AM -0400, Laine Stump wrote:
>> On 04/11/2013 07:23 AM, Michael S. Tsirkin wrote:
>>> On Thu, Apr 11, 2013 at 07:03:56AM -0400, Laine Stump wrote:
>>>> On 04/10/2013 05:26 AM, Daniel P. Berrange wrote:
>>>>>>> So if we later allowed for mutiple PCI roots, then we'd have something
>>>>>>> like
>>>>>>>
>>>>>>> <controller type="pci-root" index="0">
>>>>>>> <model name="i440FX"/>
>>>>>>> </controller>
>>>>>>> <controller type="pci-root" index="1">
>>>>>>> <model name="i440FX"/>
>>>>>>> </controller>
>>>>>>> <controller type="pci" index="0"> <!-- Host bridge 1 -->
>>>>>>> <address type='pci' domain='0' bus='0' slot='0''/>
>>>>>>> </controller>
>>>>>>> <controller type="pci" index="0"> <!-- Host bridge 2 -->
>>>>>>> <address type='pci' domain='1' bus='0' slot='0''/>
>>>>>>> </controller>
>>
>> There is a problem here - within a given controller type, we will now
>> have the possibility of multiple controllers with the same index - the
>> differentiating attribute will be in the <address> subelement, which
>> could create some awkwardness. Maybe instead this should be handled with
>> a different model of pci controller, and we can add a "domain" attribute
>> at the toplevel rather than specifying an <address>?
> IIUC there is a limit on the number of PCI buses you can create per
> domain, due to fixed size of PCI addresses. Google suggests to me
> the limit is 256. So for domain 1, we could just start index at
> 256, and domain 2 at 512, etc
Okay. Whether we choose that method, or a separate domain attribute, I'm
satisfied that we'll be able to find a way to solve it when the time
comes (and it hasn't yet), so we can ignore that problem for now.
>
>
>> Comment: I'm not quite convinced that we really need the separate
>> "pci-root" device. Since 1) every pci-root will *always* have either a
>> pcie-root-bus or a pci-bridge connected to it, 2) the pci-root-bus will
>> only ever be connected to the pci-root, and 3) the pci-bridge that
>> connects to it will need special handling within the pci-bridge case
>> anyway, why not:
>>
>> 1) eliminate the separate pci-root controller type
> Ok, lets leave it out - we can always add it later if desired.
Okay.
>
>> 2) within <controller type='pci'>, a new <model type='pci-root-bus'/>
>> will be added.
>>
>> 3) a pcie-root-bus will automatically be added for q35 machinetypes, and
>> pci-root-bus for any machinetype that supports a PCI bus (e.g. "pc-*")
>>
>> 4) model type='pci-root-bus' will behave like pci-bridge, except that it
>> will be an implicit device (nothing on qemu commandline) and it won't
>> need an <address> element (neither will pcie-root-bus).
>>
>> 5) to support multiple domains, we can simply add a "domain" attribute
>> to the toplevel of controller.
> Or use index numbers modulo 256 to identify domain numbers.
Right. One or the other. But we can defer that discussion.
> One note on q35 - we need to make sure whatever we do in terms of creating
> default <controller>s in the XML 'just works' for applications. eg if they
> define a guest using <type machine="q35">hvm</type>, and then add a
> <interface>, it should do the right thing wrt PCI addressing/connectivity.
> We must not require applications to manually add <controller> elements
> for q35 for things to work. Adding <controller>s must purely be an opt-in
> for apps which have the detailed knowledge rquired & need full control
> over bus layout.
Yep. What I see happening is that the place where we currently add
default controllers will, in the future, automatically add this for
machinetype pc* and rhel-*:
<controller type='pci'> <!-- implied index='0' -->
<model type='pci-root'/>
</controller>
and for machinetype q35* it will add (something like):
<controller type='pci'> <!-- index='0' -->
<model type='pcie-root'/>
</controller>
<controller type='pci'> <!-- index='1' -->
<model type='dmi-to-pci-bridge'/>
<address type='pci' bus='0' slot='0x1e'/>
</controller>
<controller type='pci'> <!-- index='2' -->
<model type='pci-bridge'>
<address type='pci' bus='1' slot='1'/>
</controller>
The slot-auto-reserve code will look through all pci controllers and
only auto-reserve slots on controllers appropriate for the given device
- controller 0 is already inappropriate for PCI devices, and we can mark
the dmi-to-pci-bridge type as being inappropriate for auto-reserve
(since, if I recall correctly, I was told that you can't hotplug devices
on that bus). So, all new PCI devices in the config will get addresses
with bus='2'.
Of course this means that it will not be possible to switch an existing
domain config from pc to q35 simply by changing the machinetype - the
bus number in the address of all devices will need to be changed from 0
to 2. But this is another case of "opt in", and already requires editing
the domain config anyway. If someone creates a brand new q35 machine
though, all PCI devices will get added with bus='whatever is the bus
number of the first pci-root or pci-bridge controller' (in this case, '2').
So, here are the proposed pci controller types cleaned up an
re-summarized, followed by an example.
<controller type='pci'>
=======================
This will be used for *all* of the following PCI controller devices
supported by qemu:
<model type='pci-root'/> (implicit/integrated)
------------------------
Upstream: implicit connection to the host
Downstream: 32 slots (slot 0 reserved), PCI devices only
qemu commandline: nothing (implicit in the pc-* etc. machinetypes)
This controller represents a pc* (or rhel-*) machine's integrated PCI
bus (pci.0) and provides places for PCI devices to connect (including
the "pci-bridge" type of PCI controller).
There is only one of these controllers, and it will *always* be
index='0', and will have no <address> element.
<model type='pcie-root'/> (implicit/integrated)
-------------------------
Upstream: implicit connection to the host
Downstream: 32 slots (slot 0 reserved), PCIe devices only, no hotplug.
qemu commandline: nothing (implicit in the q35-* machinetype)
This controller represents a q35's PCI "root complex", and provides
places for PCIe devices to connect. Examples are root-ports,
dmi-to-pci-bridges sata controllers, integrated sound/usb/ethernet
devices (do any of those integrated devices that can be connected to
the pcie-root-bus exist yet?).
There is only one of these controllers, and it will *always* be
index='0', and will have no <address> element.
<model type='root-port'/> (ioh3420)
-------------------------
Upstream: PCIe, connect to pcie-root-bus *only* (?)
Downstream: 1 slot, PCIe devices only (?)
qemu commandline: -device ioh3420,...
These can only connect to the "pcie-root" of a q35. Any PCIe
devices can connect to it, including an upstream-switch-port.
<model type='upstream-switch-port'/> (x3130-upstream)
------------------------------------
Upstream: PCIe, connect to pcie-root-bus, root-port, or
downstream-switch-port (?)
Downstream: 32 slots, connect *only* to downstream-switch-port
qemu-commandline: -device x3130-upstream
This is the upper side of a switch that can multiplex multiple devices
onto a single port. It's only useful when one or more downstream switch
ports are connected to it.
<model type='downstream-switch-port'/> (xio3130-downstream)
--------------------------------------
Upstream: connect *only* to upstream-switch-port
Downstream: 1 slot, any PCIe device
qemu commandline: -device xio3130-downstream
You can connect one or more of these to an upstream-switch-port in order
to effectively plug multiple devices into a single PCIe port.
<model type='dmi-to-pci-bridge'/> (i82801b11-bridge)
---------------------------------
(btw, what does "dmi" mean?)
Upstream: pcie-root *only*
Downstream: 32 slots, any PCI device (including "pci-bridge"),
no hotplug (?)
qemu commandline: -device i82801b11-bridge,...
This is the gateway to the world of standard old PCI.
<model type='pci-bridge'/> (pci-bridge)
--------------------------
Upstream: PCI, connect to 1) pci-root, 2) dmi-to-pci-bridge
3) another pci-bridge
Downstream: any PCI device, 32 slots
qemu commandline: -device pci-bridge,...
This differs from dmi-to-pci-bridge in that its upstream connection is
PCI rather than PCIe (so it will work on an i440FX system, which has a
pci-root rather than pcie-root) and that hotplug is supported. In
general, if a guest will have any PCI devices, one of these
controllers should be added, and the PCI devices connected to it
rather than to the dmi-to-pci-bridge.
************************************
(For q35, we *may* decide to always auto-add a dmi-to-pci-bridge at
00:1E.0, and a pci-bridge on slot 1 of the dmi-to-pci-bridge. This
will allow a continuation of the tradition of simply adding new
devices to the config without worrying about where they connect.)
============================================================================
Just to make sure this config model will work, here is the XML to
replicate the layout (only the ones involved in the PCI tree, along with
3 ethernet devices as examples) of the X58 hardware I have sitting under
my desk (I've attached lspci and virsh nodedev-list --tree output from
that machine):
<controller type='pci' index='0'>
<model type='pcie-root'/>
</controller>
<controller type='pci' index='1'>
<model type='root-port'/>
<address type='pci' bus='0' slot='1'/>
</controller>
( there is a scsi controller connected to bus='1')
<controller type='pci' index='2'>
<model type='root-port'/>
<address type='pci' bus='0' slot='3'/>
</controller>
(the VGA controller is connected to bus='2')
<controller type='pci' index='3'>
<model type='root-port'/>
<address type='pci' bus='0' slot='7'/>
</controller>
(PCIe SRIOV network card (in external PCIe slot) connected to bus='3')
<controller type='pci' index='4'>
<model type='root-port'/>
<address type='pci' bus='0' slot='0x1c' function='0'/>
</controller>
(unused PCIe slot available on bus='4')
<!-- pcie-root (0:1c.4) -> root-port (5:0.0) -> onboard ethernet ->
<controller type='pci' index='5'>
<model type='root-port'/>
<address type='pci' bus='0' slot='0x1c' function='4'/>
</controller>
<interface type='blah'>
...
<mac address='00:27:13:53:db:76'/>
<address type='pci' bus='5' slot='0' function='0'/>
</interface>
<!-- more complicated connection to 2nd systemboard ethernet -->
<!-- pcie-root ->(0:1c:5)root-port -> (6:0.0)upstream-switch-port
-> (7:3.0)downstream-switch-port -> (9:0.0)ethernet -->
<controller type='pci' index='6'>
<model type='root-port'/>
<address type='pci' bus='0' slot='0x1c' function='5'/>
</controller>
<controller type='pci' index='7'>
<model type='upstream-switch-port'/>
<address type='pci' bus='6' slot='0' function='0'/>
</controller>
<controller type='pci' index='8'>
<model type='downstream-switch-port'/>
<address type='pci' bus='7' slot='2' function='0'/>
</controller>
<controller type='pci' index='9'>
<model type='downstream-switch-port'/>
<address type='pci' bus='7' slot='3' function='0'/>
</controller>
<interface type='blah'>
...
<mac address='00:27:13:53:db:77'/>
<address type='pci' bus='9' slot='0' function='0'/>
</interface>
<!-- old-fashioned PCI ethernet in an external PCI slot -->
<controller type='pci' index='0x0a'>
<model type='dmi-to-pci-bridge'/>
<address type='pci' bus='0x1e' slot='0' function='0'/>
</controller>
<interface type='blah'>
...
<mac address='00:03:47:7b:63:e6'/>
<address type='pci' bus='0x0a' slot='0x0e' function='0'/>
</interface>
So I think this will all work. Does anyone see any problems?
If not, then we can draw it all back to the *current* patchset - support
for multiple PCI buses using the pci-bridge device. For *that*, we only
need to implement the follow bits of the above:
1) There will be a new <controller type='pci'> device, with a <model
type='xyz'/> subelement. Initially we will support types "pci-root" and
"pci-bridge" (all the other types discussed above can be added later).
pci-root will have *no <address>* element (and will generate nothing on
the qemu commandline, but will create a 32 slot "bus='0'" to plug PCI
devices into). pci-bridge will have an <address> element, will generate
a -device option on the qemu commandline, and will also create a 32 slot
"bus='n'" to plug PCI devices into.
2) for machinetypes that have a PCI bus, the config should have this
controller auto-added:
<controller type='pci'>
<model type='pci-root'/>
</controller>
This will make bus='0' available (but add nothing to the qemu
commandline). Any attempt to add a PCI device when there is no bus
available should be an error.
3) The way to add more buses will be to add a controller like this:
<controller type='pci'>
<model type='pci-bridge'/>
</controller>
4) When <controller type='usb'> was added, resulting in auto-generated
devices, that caused problems when migrating from a host with newer
libvirt to one with older libvirt. We need to make sure we don't suffer
the same problem this time. See the following two BZes for details
(unless you have a better memory than me! :-):
https://bugzilla.redhat.com/show_bug.cgi?id=815503
https://bugzilla.redhat.com/show_bug.cgi?id=856864
(and note how danpb eerily prophesied the current pending situation :-)
I think everything else about Jan's/Liguang's pci-bridge patches can remain.
-------------- next part --------------
00:00.0 Host bridge: Intel Corporation 5520 I/O Hub to ESI Port (rev 22)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 22)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 22)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 22)
00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 22)
00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 22)
00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 22)
00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.1 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.2 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.3 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.4 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.5 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.6 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.7 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:1a.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
00:1a.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
00:1a.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
00:1b.0 Audio device: Intel Corporation 82801JI (ICH10 Family) HD Audio Controller
00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1
00:1c.4 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 5
00:1c.5 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 6
00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.3 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIB (ICH10) LPC Interface Controller
00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller
00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
01:00.0 RAID bus controller: Marvell Technology Group Ltd. MV64460/64461/64462 System Controller, Revision B (rev 01)
02:00.0 VGA compatible controller: NVIDIA Corporation G86 [Quadro NVS 290] (rev a1)
03:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
03:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5755 Gigabit Ethernet PCI Express (rev 02)
06:00.0 PCI bridge: Integrated Device Technology, Inc. PES3T3 PCI Express Switch (rev 0e)
07:02.0 PCI bridge: Integrated Device Technology, Inc. PES3T3 PCI Express Switch (rev 0e)
07:03.0 PCI bridge: Integrated Device Technology, Inc. PES3T3 PCI Express Switch (rev 0e)
09:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5754 Gigabit Ethernet PCI Express (rev 02)
0a:0a.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
0a:0e.0 Ethernet controller: Intel Corporation 82557/8/9/0/1 Ethernet Pro 100 (rev 08)
-------------- next part --------------
virsh nodedev-list --tree
computer
|
+- net_eth0_43_00_27_13_53_db_76
+- net_lo_00_00_00_00_00_00
+- net_macvtap0_52_54_00_25_58_d5
+- net_virbr0_nic_52_54_00_97_57_57
+- net_virbr2_nic_52_54_00_e7_d4_cf
+- net_vnet0_fe_54_00_d6_49_25
+- pci_0000_00_00_0
+- pci_0000_00_01_0
| |
| +- pci_0000_01_00_0
| |
| +- scsi_host0
| |
| +- scsi_target0_0_0
| |
| +- scsi_0_0_0_0
| |
| +- block_sda_ST31500341AS_9VS3WLVQ
|
+- pci_0000_00_03_0
| |
| +- pci_0000_02_00_0
|
+- pci_0000_00_07_0
| |
| +- pci_0000_03_00_0
| | |
| | +- net_eth20_00_1b_21_3a_7d_50
| |
| +- pci_0000_03_00_1
| | |
| | +- net_eth19_00_1b_21_3a_7d_51
| |
| +- pci_0000_03_10_0
| | |
| | +- net_eth5_42_bd_62_63_2b_0f
| |
| +- pci_0000_03_10_1
| +- pci_0000_03_10_2
| | |
| | +- net_eth18_02_e2_63_03_20_cd
| |
| +- pci_0000_03_10_3
| | |
| | +- net_eth12_06_d1_6b_ea_3d_7c
| |
| +- pci_0000_03_10_4
| | |
| | +- net_eth6_de_4e_b3_6a_ad_86
| |
| +- pci_0000_03_10_5
| | |
| | +- net_eth13_e6_69_dd_db_2d_e2
| |
| +- pci_0000_03_10_6
| | |
| | +- net_eth7_86_fa_45_44_ea_94
| |
| +- pci_0000_03_10_7
| | |
| | +- net_eth14_e6_2b_55_1d_41_3c
| |
| +- pci_0000_03_11_0
| | |
| | +- net_eth8_3e_1d_bf_8b_21_f9
| |
| +- pci_0000_03_11_1
| | |
| | +- net_eth15_7e_75_c8_fc_95_94
| |
| +- pci_0000_03_11_2
| | |
| | +- net_eth9_f2_55_ae_dc_c5_bb
| |
| +- pci_0000_03_11_3
| | |
| | +- net_eth16_9e_86_ec_f9_04_79
| |
| +- pci_0000_03_11_4
| | |
| | +- net_eth10_7a_0a_d3_d8_58_ba
| |
| +- pci_0000_03_11_5
| |
| +- net_eth17_0a_b7_28_86_0f_c9
|
+- pci_0000_00_14_0
+- pci_0000_00_14_1
+- pci_0000_00_14_2
+- pci_0000_00_16_0
+- pci_0000_00_16_1
+- pci_0000_00_16_2
+- pci_0000_00_16_3
+- pci_0000_00_16_4
+- pci_0000_00_16_5
+- pci_0000_00_16_6
+- pci_0000_00_16_7
+- pci_0000_00_1a_0
| |
| +- usb_usb3
| |
| +- usb_3_0_1_0
|
+- pci_0000_00_1a_1
| |
| +- usb_usb4
| |
| +- usb_4_0_1_0
|
+- pci_0000_00_1a_7
| |
| +- usb_usb1
| |
| +- usb_1_0_1_0
|
+- pci_0000_00_1b_0
+- pci_0000_00_1c_0
+- pci_0000_00_1c_4
| |
| +- pci_0000_05_00_0
| |
| +- net_eth0_00_27_13_53_db_76
|
+- pci_0000_00_1c_5
| |
| +- pci_0000_06_00_0
| |
| +- pci_0000_07_02_0
| +- pci_0000_07_03_0
| |
| +- pci_0000_09_00_0
| |
| +- net_eth1_00_27_13_53_db_77
|
+- pci_0000_00_1d_0
| |
| +- usb_usb5
| |
| +- usb_5_0_1_0
|
+- pci_0000_00_1d_1
| |
| +- usb_usb6
| |
| +- usb_6_0_1_0
|
+- pci_0000_00_1d_2
| |
| +- usb_usb7
| |
| +- usb_7_0_1_0
| +- usb_7_1
| | |
| | +- usb_7_1_1_0
| |
| +- usb_7_2
| |
| +- usb_7_2_1_0
|
+- pci_0000_00_1d_3
| |
| +- usb_usb8
| |
| +- usb_8_0_1_0
|
+- pci_0000_00_1d_7
| |
| +- usb_usb2
| |
| +- usb_2_0_1_0
|
+- pci_0000_00_1e_0
| |
| +- pci_0000_0a_0a_0
| +- pci_0000_0a_0e_0
| |
| +- net_eth21_00_03_47_7b_63_e6
|
+- pci_0000_00_1f_0
+- pci_0000_00_1f_2
| |
| +- scsi_host1
| | |
| | +- scsi_target1_0_0
| | |
| | +- scsi_1_0_0_0
| | |
| | +- block_sr0
| |
| +- scsi_host2
| +- scsi_host3
| +- scsi_host4
| +- scsi_host5
| +- scsi_host6
|
+- pci_0000_00_1f_3
More information about the libvir-list
mailing list