[libvirt-users] pcie-expander-bus doesn't support pcie-pci-bridge and pcie-switch-upstream-port

[Marcel Apfelbaum]

Hi Laine,

On 10/17/18 8:29 PM, Laine Stump wrote:
> On 10/17/2018 08:56 AM, Andrea Bolognani wrote:
>> On Wed, 2018-10-17 at 10:50 +0800, Han Han wrote:
>>> In libvirt, I found pcie-expander-bus controller doesn't support pcie-to-pci-bridge and pcie-switch-upstream-port.
>> [...]
>>> # virsh -k0 -K0 define /tmp/c.xml
>> Aside: the -k and -K virsh options are documented as
>>
>>    -k | --keepalive-interval=NUM
>>                        keepalive interval in seconds, 0 for disable
>>    -K | --keepalive-count=NUM
>>                        number of possible missed keepalive messages
>>
>> So -k0 disables keepalive entirely, making -K0 unnecessary :)
>>
>>> error: Failed to define domain from /tmp/c.xml
>>> error: XML error: The device at PCI address 0000:01:00.0 cannot be plugged into the PCI controller with index='1'. It requires a controller that accepts a pcie-to-pci-bridge.
>> [...]
>>> In function virDomainPCIAddressBusSetModel, I find pcie-expander-bus only supports pcie-root-port and dmi-to-pci-bridge
>>>     353     case VIR_DOMAIN_CONTROLLER_MODEL_PCIE_EXPANDER_BUS:
>>>     354     ┆   /* 32 slots, no hotplug, only accepts pcie-root-port or
>>>     355     ┆   ┆* dmi-to-pci-bridge
>>>     356     ┆   ┆*/
>>>     357     ┆   bus->flags = (VIR_PCI_CONNECT_TYPE_PCIE_ROOT_PORT |
>>>     358     ┆   ┆   ┆   ┆   ┆ VIR_PCI_CONNECT_TYPE_DMI_TO_PCI_BRIDGE);
>>>     359     ┆   bus->minSlot = 0;
>>>     360     ┆   bus->maxSlot = VIR_PCI_ADDRESS_SLOT_LAST;
>>>     361     ┆   break;
>> The above is consistent with pcie-root itself, which also doesn't
>> support plugging a pcie-to-pci-bridge directly into it but requires
>> a pcie-root <- pcie-root-port <- pcie-to-pci-bridge topology.
>>
>> Now, I don't quite recall *why* that is the case - perhaps Laine
>> does? - but I'm sure we had very compelling reasons O:-)
> Umm, because Marcel told us that was the proper behavior ? Right, Marcel?

Yes it is.
The pcie-pxb expander does not support "embedded" devices, any devices
should be plugged into pcie ports.

Even if QEMU cmd line allows embedded devices, it is not a supported 
configuration.
Zooming it into the specific case of the pcie-pci bridge, this is a PCI 
Express device,
and PCI Express devices may miss-behave if plugged as Integrated Devices.

Thanks,
Marcel

>
>>> But it works in qemu:
>>> # /usr/libexec/qemu-kvm -machine q35 -m 1024 -device pxb-pcie,bus_nr=250,id=pci.1,bus=pcie.0,addr=0x4 -device pcie-pci-bridge,id=pci.250,bus=pci.1,addr=0x0 -device pcie-pci-bridge,id=pci.251,bus=pci.1,addr=0x1 -spice port=5902,addr=0.0.0.0,disable-ticketing /var/lib/libvirt/images/q35.qcow2
>> It should be noted that QEMU in general allows users to make pretty
>> unwise device placement choices without so much of a warning, so I
>> would not take the above as proof the pcie-to-pci-bridge should be
>> allowed to plug into pcie-expander-bus (or pcie-root) directly :)
>>
>