[libvirt] [PATCH v5 0/4] qemu: Allow PCI virtio on ARM "virt" machine

[Laine Stump]

(Alex - I cc'ed you because I addressed a question or two your way down
towards the bottom).

On 08/11/2015 02:52 AM, Pavel Fedin wrote:
>  Hello!
>
>> The original patches to support pcie-root severely restricted what could
>> plug into what because in real hardware you can't plug a PCI device into
>> a PCIe slot (physically it doesn't work)
>  But how do you know whether the device is PCI or PCIe ? I don't see anything like this in the code, i see that for example "all network cards are PCI", which is, BTW, not true in the real world.

Two years ago when I first added support for q35-based machinetypes and
the first pcie controllers,  I had less information than I do now. When
I looked in the ouput of "qemu-kvm -device ?" I saw that each device
listed the type of bus it connected to (PCI or ISA), and assumed that
even though at the time qemu didn't differentiate between PCI and PCIe
there, since the two things *are* different in the real world eventually
they likely would. I wanted the libvirt code to be prepared for that
eventuality. Of course every example device (except the PCIe controllers
themselves) ends up with the flag saying that it can connect to a PCI
bus, not PCIe).

Later I was told that, unlike the real world where, if nothing else, the
physical slots themselves limit you, any normal PCI device in qemu could
be plugged into a PCI or PCIe slot. There still are several restrictions
though, which showed themselves as more complicated than just the naive
PCI vs PCIe that I originally imagined - just look at the restrictions
on the different PCIe controllers:

("pcie-sw-up-port" == "pcie-switch-upstream-port", "pcie-sw-dn-port" ==
"pcie-switch-downstream-port")

name                 upstream               downstream
-----------------    -----------------      -------------------
pcie-root            none                   any endpoint
                                            pcie-root-port
                                            dmi-to-pci-bridge
                                            pci-bridge
                                            31 ports NO hotplug

dmi-to-pci-bridge    pcie-root              any endpoint device
                     pcie-root-port         pcie-sw-up-port
                     pcie-sw-dn-port
                     NO hotplug             32 ports NO hotplug

pcie-root-port       pcie-root-only         any endpoint
                     NO hotplug             dmi-to-pci-bridge
                                            pcie-sw-up-port
                                            1 port hotpluggable

pcie-sw-up-port      pcie-root-port         pcie-sw-dn-port
                     pcie-sw-dn-port        32 ports "kind of" hotpluggable
                     "kind of" hotpluggable

pcie-sw-dn-port      pcie-sw-up-port        any endpoint
                     "kind of" hotplug      pcie-sw-up-port
                                            1 port hotpluggable

pci-bridge           pci-root               any endpoint
                     pcie-root              pci-bridge
                     dmi-to-pci-bridge      32 ports hotpluggable
                     pcie-root-port
                     pcie-sw-dn-port
                     NO hotplug (now)

So the original restrictions I placed on what could plugin where were
*too* restrictive for endpoint devices, but other restrictions were
useful, and the framework came in handy as I learned the restrictions of
each new pci controller model.


>> The behavior now is that if libvirt is auto-assigning a slot for a
>> device, it will put it into a hotpluggable true PCI slot, but if you
>> manually assign it to a slot that is non-hotpluggable and/or PCIe, it
>> will be allowed.
>  But when i tried to manually assign virtio-PCI to PCIe i simply got "Device requires standard PCI slot" and that was all. I had to make patch N4 in order to overcome this.

I'm glad you pointed out this patch (I had skipped over it), because

1) that patch is completely unnecessary ever since commits 1e15be1 and
9a12b6c were pushed upstream, and

2) that patch will cause problems with auto-assignment of addresses for
virtio-net devices on Q35 machines (they will be put on pcie-root
instead of the pci-bridge).

I have verified that (1) is true - I removed your patch, built and
installed new libvirt, and tried adding a new virtio-net device to
Cole's aarch64 example domain with a manually set pci address on both
bus 0 (pcie-root) and bus 1 (dmi-to-pci-bridge), and both were successful.

I just sent a patch that reverts your patch 4. Please test it to verify
my claims and let me know so I can push it.

  https://www.redhat.com/archives/libvir-list/2015-August/msg00488.html


>> BTW, I'm still wondering if the arm machinetype really does support the
>> supposedly Interl-x86-specific i82801b11-bridge device
>  Yes, it works fine. Just devices behind it cannot get MSI-X enabled.

I'm not expert enough to know for sure, but that sounds like a bug. Alex?

I do recall that a very long time ago when I first tried out
dmi-to-pci-bridge and Q35, I found a qemu bug that made it impossible to
use network devices (and they were probably virtio-net, as that's what I
usually use) attached to a pci-bridge on a Q35 machine. Maybe the same
bug? I don't remember what the exact problem was (but again, I think
Alex will remember the problem).

I can say that currently (and for almost the last two years) there is no
problem using a virtio-net adapter that is connected to a q35 machine in
this way:


   pcie-root --> dmi-to-pci-bridge --> pci-bridge --> virtio-net

>  By the way, you should be using virtio-pci with PC guests for a while, does it also suffer from this restriction there ?

See above  :-)

>
>> (and the new > controller devices - ioh3420 (pcie-root-port), x3130-upstream
>> (pcie-switch-upstream-port), and xio3130-downstream
>> (pcie-switch-downstream-port).
>  Didn't try that, but don't see why they would not work. PCI is just PCI after all, everything behing the controller is pretty much standard and arch-independent.

Alex (yeah, I know, I need to stop being such an Alex-worshipper on PCI
topics :-P) has actually expressed concern that we are using all of
these Intel-chipset-specific PCI controllers, and thinks that we should
instead create some generic devices that have different PCI device IDs
than those. Among other problems, those Intel-specific devices only
exist in real hardware at specific addresses, and he's concerned that us
putting them at a different address might confuse some OS; better to
have a device that functions similarly, but IDs itself differently so
the OS won't make any potentially incorrect assumptions (that type of
problem is likely a non-issue for your use of these devices though,
since you don't really have any "legacy OS" you have to deal with).