Not able to add pcie card to guest: Operation not permitted

[Mauricio Tavares]

On Fri, Apr 24, 2020 at 4:35 PM Peter Crowther
<peter.crowther at melandra.com> wrote:
>
> On Fri, 24 Apr 2020 at 21:10, Mauricio Tavares <raubvogel at gmail.com> wrote:
>>
>> Let's say I have libvirt
>>
>> [root at vmhost2 ~]# virsh version
>> [...]
>>
>> Running hypervisor: QEMU 2.12.0
>> [root at vmhost2 ~]#
>> [...]
>
> When I try to start the guest I get the following error message:
>>
>>
>> [root at vmhost2 ~]# virsh start testfedora
>> error: Failed to start domain testfedora
>> error: internal error: qemu unexpectedly closed the monitor:
>> 2020-04-24T20:01:35.341020Z qemu-kvm: -device
>> vfio-pci,host=01:00.0,id=hostdev0,bus=pci.8,addr=0x0: vfio error:
>> 0000:01:00.0: failed to setup INTx fd: Operation not permitted
>>
>> [root at vmhost2 ~]#
>>
>> Why is it telling me that is not permitted?
>>
> The guest will be running as qemu on the host.  Does qemu have appropriate permissions in the host, and does that include in any hardening like SElinux that you're running?
>

I tried with selinux in permissive mode to see if it made a
difference. Not much.

[root at vmhost2 ~]# getenforce
Permissive
[root at vmhost2 ~]# virsh start testfedora
error: Failed to start domain testfedora
error: internal error: qemu unexpectedly closed the monitor:
2020-04-25T00:43:36.621246Z qemu-kvm: -device
vfio-pci,host=01:00.0,id=hostdev0,bus=pci.8,addr=0x0: vfio error:
0000:01:00.0: failed to setup INTx fd: Operation not permitted

[root at vmhost2 ~]#

For the fun of it, I swapped that card with another one (same speed,
number of ports, diff brand), so it is on th every sam epci slot:

[root at vmhost2 ~]# virsh nodedev-dumpxml pci_0000_01_00_0
<device>
  <name>pci_0000_01_00_0</name>
  <path>/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0</path>
  <parent>pci_0000_00_01_0</parent>
  <driver>
    <name>vfio-pci</name>
  </driver>
  <capability type='pci'>
    <domain>0</domain>
    <bus>1</bus>
    <slot>0</slot>
    <function>0</function>
    <product id='0x4000' />
    <vendor id='0x19ee'>Netronome Systems, Inc.</vendor>
    <capability type='virt_functions' maxCount='64'/>
    <iommuGroup number='1'>
      <address domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
      <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </iommuGroup>
    <pci-express>
      <link validity='cap' port='0' speed='8' width='8'/>
      <link validity='sta' speed='2.5' width='8'/>
    </pci-express>
  </capability>
</device>


[root at vmhost2 ~]#

And it starts without an issue:

[root at vmhost2 ~]# virsh start testfedora
Domain testfedora started

[root at vmhost2 ~]#

Inside the guest:

[root at testfedora ~]# dmesg |grep -i netronome
[   12.327316] nfp: NFP PCIe Driver, Copyright (C) 2014-2017 Netronome Systems
[   12.335036] nfp 0000:07:00.0: Netronome Flow Processor
NFP4000/NFP5000/NFP6000 PCIe Card Probe
[root at testfedora ~]#

so I do not know what is going on.

> Cheers,
>
> - Peter
>