[libvirt] How should libvirt apps enable virtio-pci for aarch64?

[Pavel Fedin]

 Hello!

> - The PCIe controller XML is:
>     <controller type='pci' index='0' model='pcie-root'/>
>     <controller type='pci' index='1' model='dmi-to-pci-bridge'/>
>     <controller type='pci' index='2' model='pci-bridge'/>
> I have no idea if that's always going to be the expected XML, maybe it's not
> wise to hardcode that in apps.

 Since we are discussing this, i have a question.
 Why do we construct exactly this thing? What is "dmi-to-pci-bridge" and why do we need it? I guess this is something PC-specific,
may be this layout has been copied from some real PC model, but i don't see any practical sense in it.
 Also, there are two problems with "pci-bridge":
1. Live migration of this thing in qemu is broken. After migration the bridge screws up, lspci says "invalid header", and i don't
know whether it actually works because i never attach anything behind it, because of (2).
2. After pcie-root we have PCI-X, which supports MSI-X. And after pci-bridge we seem to have a plain PCI, which supports only plain
MSI. The problem here is that virtio seems to work only with MSI-X in any advanced mode (multiqueue, vhost, etc). If i place it
behind the bridge (and default libvirt's logic is to place the device there), MSI-X will not work. The same applies to passthrough
VFIO devices. This is especially painful because on real-life ARM64 platforms builtin hardware seems to mandate MSI-X. For example
on ThunderX NIC driver simply does not support anything except MSI-X.

> * Next idea: Users specify something like like <address type='pci'/> and
> libvirt fills in the address for us.

 I like this one, and IMHO this would be nice to have regardless of the default. Manual assignment of PCI layout is a tedious
process, which is not always necessary. I think it is quite logical to allow the user just to say: "I want this device on the PCI
bus", and do the rest for him.
 Also, this would allow to have a simple "Address type" switch in e.g. virt-manager, so that i would be able to easily choose a way
which the device uses to connect to the system.
 
Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia