[libvirt] How should libvirt apps enable virtio-pci for aarch64?

[Cole Robinson]

On 12/07/2015 03:19 AM, Pavel Fedin wrote:
>  Hello!
> 
>> - The PCIe controller XML is:
>>     <controller type='pci' index='0' model='pcie-root'/>
>>     <controller type='pci' index='1' model='dmi-to-pci-bridge'/>
>>     <controller type='pci' index='2' model='pci-bridge'/>
>> I have no idea if that's always going to be the expected XML, maybe it's not
>> wise to hardcode that in apps.
> 
>  Since we are discussing this, i have a question.
>  Why do we construct exactly this thing? What is "dmi-to-pci-bridge" and why do we need it? I guess this is something PC-specific,
> may be this layout has been copied from some real PC model, but i don't see any practical sense in it.

This is likely just a side effect of the libvirt code requesting PCIe for
aarch64, but the original PCIe support was added for the x86 q35 layout.

>  Also, there are two problems with "pci-bridge":
> 1. Live migration of this thing in qemu is broken. After migration the bridge screws up, lspci says "invalid header", and i don't
> know whether it actually works because i never attach anything behind it, because of (2).

I didn't even know aarch64 migration was working...

> 2. After pcie-root we have PCI-X, which supports MSI-X. And after pci-bridge we seem to have a plain PCI, which supports only plain
> MSI. The problem here is that virtio seems to work only with MSI-X in any advanced mode (multiqueue, vhost, etc). If i place it
> behind the bridge (and default libvirt's logic is to place the device there), MSI-X will not work. The same applies to passthrough
> VFIO devices. This is especially painful because on real-life ARM64 platforms builtin hardware seems to mandate MSI-X. For example
> on ThunderX NIC driver simply does not support anything except MSI-X.
> 

Maybe this is just a factor of libvirt specifying the wrong bits on the
aarch64 command line. Do you have a working qemu commandline outside of libvirt?

>> * Next idea: Users specify something like like <address type='pci'/> and
>> libvirt fills in the address for us.
> 
>  I like this one, and IMHO this would be nice to have regardless of the default. Manual assignment of PCI layout is a tedious
> process, which is not always necessary. I think it is quite logical to allow the user just to say: "I want this device on the PCI
> bus", and do the rest for him.

Agreed, I'll look into it in addition to the user PCI controller bits.

>  Also, this would allow to have a simple "Address type" switch in e.g. virt-manager, so that i would be able to easily choose a way
> which the device uses to connect to the system.
>  

Currently I don't see a compelling reason to add this in the UI really. Sure
it's probably useful for virt dev testing to switching between mmio and pci
but long term everyone is going to use pci so I don't think end users are
going to be tweaking this. I think extending virt-xml to handle address bits
should cover it.

Thanks,
Cole