how hotplug devices onto pcie-to-pci-bridge

Tue Nov 8 02:08:07 UTC 2022

>But in general it's correct that this isn't anything libvirt can easily fix.
>
>Rather than trying to make hotplug to a pcie-to-pci-bridge work, the 
>*proper* fix is to just allow the virtio devices to be hotplugged to 
>pcie-root-ports, which is what libvirt would do by default. Then they 
>will be recognized as virtio-1.0 devices and use a mode that doesn't 
>require any memory to be mapped for them.
>
Do I have to add as many pcie-root-ports as posibble in advance?


At 2022-11-08 02:53:55, "Laine Stump" <laine at laine.org> wrote:
>On 11/6/22 3:28 AM, longguang.yue wrote:
>> Hi, Laine Stump and Oleg Vasilev:
>> 1.  Laine, what is 'conventional PCI device'? in my case they are all 
>> virtio device.
>
>A "conventional PCI" device is one that is designed to work with the 
>original PCI spec, i.e. before PCI Express. Among other differences, 
>they lack a PCIe capabilities section in their config data.
>
>All virtio devices are PCI Express devices or conventional PCI, 
>depending on the type of slot they are plugged into (that's my 
>understanding of it anyway). Most other devices are always just one or 
>the other.
>
>On real hardware, a PCIe device could never be plugged into a 
>conventional PCI slot, nor could a conventional PCI device be plugged 
>into a PCIe slot. QEMU allows this though, and just kind of glosses over 
>the rough spots. It can lead to problems though, e.g. the PCIe extended 
>capabilities space is not visible and so things like bus width/speed are 
>not communicated properly to the guest OS (and can't be changed).
>
>> 2. Followed your guide,succeed in hotplug.  but there are lots of 
>> problems after hotplug. I think problems are more related to qemu.
>>    2.1) disk devices(qcow2,virio) hotpluged onto pci-bridge can not 
>> alloc bar resource.
>
>When you plug a virtio device into a conventional PCI slot, it uses the 
>older (pre-virtio-1.0) method of host<->device communication, which 
>memory-maps a region. However, if a pci-bridge has no devices plugged 
>into it when the guest is started, then the BIOS won't reserve space for 
>this memory-mapped region (or something like that - I never retain 
>details). Because it's using the older conventional PCI virtio that 
>requires this region to be mapped for proper operation, it fails.
>
>>    2.2) disk devices(qcow2,virio) are staticly pluged to pci-bridge can 
>> not be found by guest os.
>>           those devices are on pci bus, but can not be recognized by 
>> guest os, no block device are probed.
>
>No idea about that.
>
>But in general it's correct that this isn't anything libvirt can easily fix.
>
>Rather than trying to make hotplug to a pcie-to-pci-bridge work, the 
>*proper* fix is to just allow the virtio devices to be hotplugged to 
>pcie-root-ports, which is what libvirt would do by default. Then they 
>will be recognized as virtio-1.0 devices and use a mode that doesn't 
>require any memory to be mapped for them.
>
>
>> qemu qemu-6.2.0
>> -----hotplug error message--------
>> [ 1067.577384] pci 0000:06:02.0: [1af4:1001] type 00 class 0x010000
>> [ 1067.578312] pci 0000:06:02.0: reg 0x10: [io 0x0000-0x007f]
>> [ 1067.578794] pci 0000:06:02.0: reg 0x14: [mem 0x00000000-0x00000fff]
>> [ 1067.579419] pci 0000:06:02.0: reg 0x20: [mem 0x00000000-0x00003fff 
>> 64bit pref]
>> [ 1067.581592] pci 0000:06:02.0: BAR 4: no space for [mem size 
>> 0x00004000 64bit pref]
>> [ 1067.581942] pci 0000:06:02.0: BAR 4: failed to assign [mem size 
>> 0x00004000 64bit pref]
>> [ 1067.582264] pci 0000:06:02.0: BAR 1: no space for [mem size 0x00001000]
>> [ 1067.582625] pci 0000:06:02.0: BAR 1: failed to assign [mem size 
>> 0x00001000]
>> [ 1067.582942] pci 0000:06:02.0: BAR 0: no space for [io size 0x0080]
>> [ 1067.583253] pci 0000:06:02.0: BAR 0: failed to assign [io size 0x0080]
>> [ 1067.584019] PCI Interrupt Link [GSIE] enabled at IRQ 20
>> [ 1067.584749] virtio-pci 0000:06:02.0: virtio_pci: leaving for legacy 
>> driver
>> [ 1067.585296] virtio-pci: probe of 0000:06:02.0 failed with error -12
>> 
>> -------staticly plug------------
>> [ 0.680299] pci 0000:06:02.0: [1af4:1001] type 00 class 0x010000
>> [ 0.683035] pci 0000:06:02.0: reg 0x10: [io 0xc080-0xc0ff]
>> [ 0.688036] pci 0000:06:02.0: reg 0x14: [mem 0xfca01000-0xfca01fff]
>> [ 0.700040] pci 0000:06:02.0: reg 0x20: [mem 0xfb404000-0xfb407fff 64bit 
>> pref]
>> 
>> --------------------------------------
>> 
>> 
>> 
>> At 2022-11-04 05:28:47, "Laine Stump" <laine at laine.org> wrote:
>>>On 11/3/22 6:35 AM, longguang.yue wrote:
>>>> Hi, all
>>>>    I have add two pcie-to-pci-bridge controllers, how hotplug devices 
>>>> onto this two controllers?  and libvirt knows to plug onto pci-bridge 
>>>> when there are no pcie-ports.
>>>> defaultly libvirt plugs devices onto pcie-port.
>>>
>>>Do you mean you want hotplugged devices to be placed on the 
>>>pcie-to-pci-bridge even though there are unused pci-root-ports?
>>>
>>>You can force a new device to be hotplugged to a particular slot of a 
>>>particular controller by including the <address> element in the XML of 
>>>the device you attach - whatever controller/slot/function you specify 
>>>there is what libvirt will use (practically speaking the function must 
>>>be 0).
>>>
>>>My recollection is that, *if the device is a conventional PCI device 
>>>(and not a PCIe device) then libvirt will anyway attempt to find a an 
>>>open conventional PCI slot, which would be found only on the 
>>>pcie-to-pci-bridge. However, very few devices are actually conventional 
>>>PCI, so this will very rarely happen automatically (I haven't looked at 
>>>that code in a long time, and retain no specific memory, but I think the 
>>>only emulated device that is conventional PCI is a watchdog device or 
>>>something)
>>>
>>>So, if libvirt is selecting a pcie-root-port to plug in a device, that's 
>>>because the device is a PCIe device. If you want to force it to do 
>>>otherwise, you'll need to manually specify the address (just set "bus" 
>>>of the address to the "index" of the controller, set "slot" to an 
>>>unoccupied slot (1-31), and set domain and function to 0).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20221108/696ff11e/attachment.htm>