[libvirt-users] assigning PCI addresses with bus > 0x09

[Laine Stump]

On 12/20/18 9:20 AM, Daniel P. Berrangé wrote:
> On Thu, Dec 20, 2018 at 03:15:34PM +0100, Riccardo Ravaioli wrote:
>> Hi,
>>
>> My goal is to assign PCI addresses to a number of devices (network
>> interfaces, disks and PCI devices in PCI-passthrough) without delegating to
>> libvirt the generation of those values. This should give me more control
>> and for sure more predictability over the hardware configuration of a
>> virtual machine and consequently the name of the interfaces in it.  I'm
>> using libvirt 4.3.0 to create qemu/KVM virtual machines running Linux
>> (Debian Stretch).
>>
>> So, for every device of the type mentioned above, I add this line:
>> <address type='pci' domain='0x0000' bus='0x__' slot='0x__' function='0x0'/>,
>> ... with values from 00 to ff in the bus field, and from 00 to 1f in the
>> slot field, as described in the documentation.
>>
>> Long story short, I noticed that as soon as I assign values > 0x09 to the
>> bus field, the serial console hangs indefinitely, in both Debian and
>> Ubuntu. The VM seems to be started correctly and its state is "running"; in
>> the XML file created by libvirt, I see all controllers from 0 the largest
>> bus value I assigned, so everything from that side seems ok.
> 
> I guess the hang is that you hit some limit in PCI buses.

I think you're right. Each bus requires some amount of IO space, and I 
thought I recalled someone saying that all of the available IO space is 
exhausted after 7 or 8 buses. This was in relation to PCIe, where each 
root port is a bus, and can potentially take up IO space, so possibly in 
that context they were talking about the buses *after* the root bus and 
pcie-pci-bridge, which would bring us back to the same number you're 
getting.

For PCIe our solution was to turn off IO space usage on the 
pcie-root-ports, but you can't do that for conventional PCI buses, since 
the devices might actually need IO space to function properly and the 
standard requires that you provide it.


> 
> The real question though is why you need to create sooooo many PCI buses.
> Each bus can do 31 devices.  Do you really need to have more than 279
> devices in your VM ?

And if you do need more than 279 devices, do they all need to be 
hot-pluggable? If not, then you can put up to 8 devices on each slot 
(functions 0 - 7).

> 
> Regards,
> Daniel
>