[libvirt-users] Using hostdev to plug a PCI-E host device into Q35 pcie-root port

[Laine Stump]

On 11/15/2013 03:35 PM, Thomas Kuther wrote:
> Hello,
>
> I'm trying to migrate a working qemu command line configuration to
> libvirt.
> The part I'm currently failing on is:
>
> $ qemu-system-x86_64 -M Q35 ... -device vfio-pci,host=05:00.0,bus=pcie.0
>
> The right way to translate this into libvirt XML seems to be using
> <hostdev>, but I seem to be unable to plug it into the pcie-root port
>
> This is how the interesting part looks like when I let "virsh edit"
> generate an <address>
>
>     <controller type='pci' index='0' model='pcie-root'/>
>     <controller type='pci' index='1' model='dmi-to-pci-bridge'>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> function='0x0'/>
>     </controller>
>     <controller type='pci' index='2' model='pci-bridge'>
>       <address type='pci' domain='0x0000' bus='0x01' slot='0x01'
> function='0x0'/>
>     </controller>
>     [...]
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <driver name='vfio'/>
>       <source>
>         <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x02' slot='0x06'
> function='0x0'/>
>     </hostdev>
>     [...]
>
> To my understanding, this will plug the host device into the
> pci-bridge controller.
> The guest OS doesn't boot with this and resets right after bios.

Ugh. That's very unfortunate. This is the first report I've heard of
something failing in such a bad way due to being plugged into a
pci-bridge slot; up until now I'd only heard that there is some extra
PCIe functionality that would be missing if a device was plugged into a
PCI slot vs. PCIe.

Can I ask what type of device this is?

>
> Manually setting
> <address type='pci' domain='0x0000' bus='0x00' slot='0x1E'
> function='0x0'/>
> cause XML validation failure.
>
> Is there any way in libvirt XML to plug a host's PCI-E device directly
> into the pcie-root port, like it works on qemu command line?


I'm sorry to say, no. With very few (and specific) exceptions, libvirt
insists that all guest devices be plugged into a hot-pluggable PCI slot
- this eliminates both the PCIe "root complex" (a.k.a. pcie.0) as well
as the dmi-to-pci-controller that is plugged into pcie.0 (because
pci-to-dmi controllers' slots don't support hot-plug).

This is done because, for now, almost all devices that qemu knows about
are PCI (no PCI-e) devices, and if we allowed plugging them into pcie.0
now, then on the day in the future when qemu begins enforcing the
difference between PCI and PCIe (currently it doesn't), the world would
be full of libvirt configs that would no longer work.

There was some discussion about this a month or two ago either on
libvir-list or maybe it was the qemu-devel list. We decided that qemu
needs to provide some sort of introspection of the devices' connection
types so that libvirt can determine what device can plug into which
slots; at that time we'll be able to allow exactly what's proper in each
case. In the meantime we're stuck with being overly cautious in order to
prevent future catastrophe.

>
> I'm aware I could use something like
>
>   <qemu:commandline>
>     <qemu:arg value='-device'/>
>     <qemu:arg value='vfio-pci,host=05:00.0,bus=pcie.0'/>
>   </qemu:commandline>
>
> but I insist on running the VM as non-root, and if I got that right I
> need to configure at least one vfio device (or memory locking) in
> order for libvirt to set a proper RLIMIT_MEMLOCK value.
>
> Any help would be be appreciated.

For now at least, you'll need to let it plug into the pci-bridge device
pci.2 (which, as you've found, libvirt will automatically find when you
don't specify any address). Unfortunately that doesn't do you much good,
since that particular device you're assigning actually requires that it
be plugged into the PCIe bus.

I'm wondering as I type if possibly we could relax the enforcement of
the "PCI only" rule such that we allow explicitly placing any device on
any type of bus, but only auto-assign to a plain PCI slot. That may be a
reasonable compromise until qemu has the required new device/controller
introspection info available.