how hotplug devices onto pcie-to-pci-bridge

Tue Nov 8 03:05:46 UTC 2022

On 11/7/22 9:08 PM, longguang.yue wrote:
>>But in general it's correct that this isn't anything libvirt can easily fix.
>>
>>Rather than trying to make hotplug to a pcie-to-pci-bridge work, the 
>>*proper* fix is to just allow the virtio devices to be hotplugged to 
>>pcie-root-ports, which is what libvirt would do by default. Then they 
>>will be recognized as virtio-1.0 devices and use a mode that doesn't 
>>require any memory to be mapped for them.
>>
> Do I have to add as many pcie-root-ports as posibble in advance?

As many as you'll need, yes. When you're defining a new domain, just add 
up the number of PCI devices you'll initially have, add the number of 
devices you might need to hotplug, and then put in that number of 
controller lines like this:

     <controller type='pci' model='pcie-root-port'/>

(into the <devices> section). libvirt will take care of assigning all 
these ports to a place on the root bus, giving each an index, and 
assigning each of the existing devices to one of these root ports - you 
won't need to figure out any pci addresses yourself.

> 
> 
> 
> 
> At 2022-11-08 02:53:55, "Laine Stump" <laine at laine.org> wrote:
>>On 11/6/22 3:28 AM, longguang.yue wrote:
>>> Hi, Laine Stump and Oleg Vasilev:
>>> 1.  Laine, what is 'conventional PCI device'? in my case they are all 
>>> virtio device.
>>
>>A "conventional PCI" device is one that is designed to work with the 
>>original PCI spec, i.e. before PCI Express. Among other differences, 
>>they lack a PCIe capabilities section in their config data.
>>
>>All virtio devices are PCI Express devices or conventional PCI, 
>>depending on the type of slot they are plugged into (that's my 
>>understanding of it anyway). Most other devices are always just one or 
>>the other.
>>
>>On real hardware, a PCIe device could never be plugged into a 
>>conventional PCI slot, nor could a conventional PCI device be plugged 
>>into a PCIe slot. QEMU allows this though, and just kind of glosses over 
>>the rough spots. It can lead to problems though, e.g. the PCIe extended 
>>capabilities space is not visible and so things like bus width/speed are 
>>not communicated properly to the guest OS (and can't be changed).
>>
>>> 2. Followed your guide,succeed in hotplug.  but there are lots of 
>>> problems after hotplug. I think problems are more related to qemu.
>>>    2.1) disk devices(qcow2,virio) hotpluged onto pci-bridge can not 
>>> alloc bar resource.
>>
>>When you plug a virtio device into a conventional PCI slot, it uses the 
>>older (pre-virtio-1.0) method of host<->device communication, which 
>>memory-maps a region. However, if a pci-bridge has no devices plugged 
>>into it when the guest is started, then the BIOS won't reserve space for 
>>this memory-mapped region (or something like that - I never retain 
>>details). Because it's using the older conventional PCI virtio that 
>>requires this region to be mapped for proper operation, it fails.
>>
>>>    2.2) disk devices(qcow2,virio) are staticly pluged to pci-bridge can 
>>> not be found by guest os.
>>>           those devices are on pci bus, but can not be recognized by 
>>> guest os, no block device are probed.
>>
>>No idea about that.
>>
>>But in general it's correct that this isn't anything libvirt can easily fix.
>>
>>Rather than trying to make hotplug to a pcie-to-pci-bridge work, the 
>>*proper* fix is to just allow the virtio devices to be hotplugged to 
>>pcie-root-ports, which is what libvirt would do by default. Then they 
>>will be recognized as virtio-1.0 devices and use a mode that doesn't 
>>require any memory to be mapped for them.
>>
>>
>>> qemu qemu-6.2.0
>>> -----hotplug error message--------
>>> [ 1067.577384] pci 0000:06:02.0: [1af4:1001] type 00 class 0x010000
>>> [ 1067.578312] pci 0000:06:02.0: reg 0x10: [io 0x0000-0x007f]
>>> [ 1067.578794] pci 0000:06:02.0: reg 0x14: [mem 0x00000000-0x00000fff]
>>> [ 1067.579419] pci 0000:06:02.0: reg 0x20: [mem 0x00000000-0x00003fff 
>>> 64bit pref]
>>> [ 1067.581592] pci 0000:06:02.0: BAR 4: no space for [mem size 
>>> 0x00004000 64bit pref]
>>> [ 1067.581942] pci 0000:06:02.0: BAR 4: failed to assign [mem size 
>>> 0x00004000 64bit pref]
>>> [ 1067.582264] pci 0000:06:02.0: BAR 1: no space for [mem size 0x00001000]
>>> [ 1067.582625] pci 0000:06:02.0: BAR 1: failed to assign [mem size 
>>> 0x00001000]
>>> [ 1067.582942] pci 0000:06:02.0: BAR 0: no space for [io size 0x0080]
>>> [ 1067.583253] pci 0000:06:02.0: BAR 0: failed to assign [io size 0x0080]
>>> [ 1067.584019] PCI Interrupt Link [GSIE] enabled at IRQ 20
>>> [ 1067.584749] virtio-pci 0000:06:02.0: virtio_pci: leaving for legacy 
>>> driver
>>> [ 1067.585296] virtio-pci: probe of 0000:06:02.0 failed with error -12
>>> 
>>> -------staticly plug------------
>>> [ 0.680299] pci 0000:06:02.0: [1af4:1001] type 00 class 0x010000
>>> [ 0.683035] pci 0000:06:02.0: reg 0x10: [io 0xc080-0xc0ff]
>>> [ 0.688036] pci 0000:06:02.0: reg 0x14: [mem 0xfca01000-0xfca01fff]
>>> [ 0.700040] pci 0000:06:02.0: reg 0x20: [mem 0xfb404000-0xfb407fff 64bit 
>>> pref]
>>> 
>>> --------------------------------------
>>> 
>>> 
>>> 
>>> At 2022-11-04 05:28:47, "Laine Stump" <laine at laine.org> wrote:
>>>>On 11/3/22 6:35 AM, longguang.yue wrote:
>>>>> Hi, all
>>>>>    I have add two pcie-to-pci-bridge controllers, how hotplug devices 
>>>>> onto this two controllers?  and libvirt knows to plug onto pci-bridge 
>>>>> when there are no pcie-ports.
>>>>> defaultly libvirt plugs devices onto pcie-port.
>>>>
>>>>Do you mean you want hotplugged devices to be placed on the 
>>>>pcie-to-pci-bridge even though there are unused pci-root-ports?
>>>>
>>>>You can force a new device to be hotplugged to a particular slot of a 
>>>>particular controller by including the <address> element in the XML of 
>>>>the device you attach - whatever controller/slot/function you specify 
>>>>there is what libvirt will use (practically speaking the function must 
>>>>be 0).
>>>>
>>>>My recollection is that, *if the device is a conventional PCI device 
>>>>(and not a PCIe device) then libvirt will anyway attempt to find a an 
>>>>open conventional PCI slot, which would be found only on the 
>>>>pcie-to-pci-bridge. However, very few devices are actually conventional 
>>>>PCI, so this will very rarely happen automatically (I haven't looked at 
>>>>that code in a long time, and retain no specific memory, but I think the 
>>>>only emulated device that is conventional PCI is a watchdog device or 
>>>>something)
>>>>
>>>>So, if libvirt is selecting a pcie-root-port to plug in a device, that's 
>>>>because the device is a PCIe device. If you want to force it to do 
>>>>otherwise, you'll need to manually specify the address (just set "bus" 
>>>>of the address to the "index" of the controller, set "slot" to an 
>>>>unoccupied slot (1-31), and set domain and function to 0).