Not able to add pcie card to guest: Operation not permitted

[Mauricio Tavares]

On Fri, Apr 24, 2020 at 9:30 PM Mauricio Tavares <raubvogel at gmail.com> wrote:
>
> On Fri, Apr 24, 2020 at 4:35 PM Peter Crowther
> <peter.crowther at melandra.com> wrote:
> >
> > On Fri, 24 Apr 2020 at 21:10, Mauricio Tavares <raubvogel at gmail.com> wrote:
> >>
> >> Let's say I have libvirt
> >>
> >> [root at vmhost2 ~]# virsh version
> >> [...]
> >>
> >> Running hypervisor: QEMU 2.12.0
> >> [root at vmhost2 ~]#
> >> [...]
> >
> > When I try to start the guest I get the following error message:
> >>
> >>
> >> [root at vmhost2 ~]# virsh start testfedora
> >> error: Failed to start domain testfedora
> >> error: internal error: qemu unexpectedly closed the monitor:
> >> 2020-04-24T20:01:35.341020Z qemu-kvm: -device
> >> vfio-pci,host=01:00.0,id=hostdev0,bus=pci.8,addr=0x0: vfio error:
> >> 0000:01:00.0: failed to setup INTx fd: Operation not permitted
> >>
> >> [root at vmhost2 ~]#
> >>
> >> Why is it telling me that is not permitted?
> >>
> > The guest will be running as qemu on the host.  Does qemu have appropriate permissions in the host, and does that include in any hardening like SElinux that you're running?
> >
>
> I tried with selinux in permissive mode to see if it made a
> difference. Not much.
>
> [root at vmhost2 ~]# getenforce
> Permissive
> [root at vmhost2 ~]# virsh start testfedora
> error: Failed to start domain testfedora
> error: internal error: qemu unexpectedly closed the monitor:
> 2020-04-25T00:43:36.621246Z qemu-kvm: -device
> vfio-pci,host=01:00.0,id=hostdev0,bus=pci.8,addr=0x0: vfio error:
> 0000:01:00.0: failed to setup INTx fd: Operation not permitted
>
> [root at vmhost2 ~]#
>
> For the fun of it, I swapped that card with another one (same speed,
> number of ports, diff brand), so it is on th every sam epci slot:
>
> [root at vmhost2 ~]# virsh nodedev-dumpxml pci_0000_01_00_0
> <device>
>   <name>pci_0000_01_00_0</name>
>   <path>/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0</path>
>   <parent>pci_0000_00_01_0</parent>
>   <driver>
>     <name>vfio-pci</name>
>   </driver>
>   <capability type='pci'>
>     <domain>0</domain>
>     <bus>1</bus>
>     <slot>0</slot>
>     <function>0</function>
>     <product id='0x4000' />
>     <vendor id='0x19ee'>Netronome Systems, Inc.</vendor>
>     <capability type='virt_functions' maxCount='64'/>
>     <iommuGroup number='1'>
>       <address domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
>       <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
>     </iommuGroup>
>     <pci-express>
>       <link validity='cap' port='0' speed='8' width='8'/>
>       <link validity='sta' speed='2.5' width='8'/>
>     </pci-express>
>   </capability>
> </device>
>
>
> [root at vmhost2 ~]#
>
> And it starts without an issue:
>
> [root at vmhost2 ~]# virsh start testfedora
> Domain testfedora started
>
> [root at vmhost2 ~]#
>
> Inside the guest:
>
> [root at testfedora ~]# dmesg |grep -i netronome
> [   12.327316] nfp: NFP PCIe Driver, Copyright (C) 2014-2017 Netronome Systems
> [   12.335036] nfp 0000:07:00.0: Netronome Flow Processor
> NFP4000/NFP5000/NFP6000 PCIe Card Probe
> [root at testfedora ~]#
>
> so I do not know what is going on.
>
      My last statement can be translated to "I am doing PCI
passthrough, which to me means I am passing the entire card --
whatever it is -- to the guest." Just for the sake of argument, I also
created a centos8 guest and had the same outcome. Who is doing the pci
passthrough thingie: libvirt or qemu? Just want to see where I should
put my efforts on.

I have the

Also, I decided for the fun of it to create a docker container in the
same host and pass the card to it. That seemed to have worked better

[root at ce3077ee015c /]# ls /sys/bus/pci/devices/0000\:01\:00.0/
aer_dev_correctable       infiniband_srp/           pools
aer_dev_fatal             infiniband_verbs/         power/
aer_dev_nonfatal          iommu/                    remove
ari_enabled               iommu_group/              rescan
broken_parity_status      irq                       reset
class                     local_cpulist             resource
config                    local_cpus                resource0
consistent_dma_mask_bits  max_link_speed            resource2
current_link_speed        max_link_width            resource2_wc
current_link_width        mlx4_port1                revision
d3cold_allowed            mlx4_port1_mtu            rom
device                    mlx4_port2                subsystem/
dma_mask_bits             mlx4_port2_mtu            subsystem_device
driver/                   modalias                  subsystem_vendor
driver_override           msi_bus                   uevent
enable                    msi_irqs/                 vendor
infiniband/               net/                      vpd
infiniband_mad/           numa_node

[root at ce3077ee015c /]# cat /sys/bus/pci/devices/0000\:01\:00.0/device
0x1003
[root at ce3077ee015c /]#

Of course the test is to configure card to use or to program it, but I
have so far been more successful. Which makes me even more confused.

> > Cheers,
> >
> > - Peter
> >