[libvirt] SR-IOV with libvirt

[Laine Stump]

On 02/20/2015 05:33 PM, Florin Stingaciu wrote:
> Hello, 
>
> For the last little while I've been trying to configure a VM with a
> passthrough interface using SR-IOV. I've managed to configure SR-IOV
> on the hypervisor correctly. I'm working with a Mellanox ConnectX3
> card with two infiniband ports and hypervisors running Centos 7. I can
> see the PCI devices (ie. virtual functions) using the `virsh
> nodedev-list` command. 
>
> The majority of tutorials online (including the tutorials on the
> libvirt site), suggest one configures an SR-IOV passthrough interface
> in the following manner in the VM definition file:
>
>     <interface type='hostdev' managed='yes'>
>       <mac address='52:54:00:d7:cd:aa'/>
>       <source>
>         <address type='pci' domain='0x0000' bus='0x01' slot='0x00'
> function='0x2'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
>     </interface>
>
> For example the above configuration should configure the second VF as
> a passthrough interface for this VM. When I try to launch the VM, I
> get the following error: 
>
>     error: Failed to start domain test
>     error: internal error: missing IFLA_VF_INFO in netlink response
>
> I've done research on this error quite extensively. This used to be a
> very popular error a while back for scenarios where the total number
> of VFs is > 64. This is not the case here as I only have 8.  
>
> I've was never able to overcome this error but I did manage to get
> this working another way. The VM will boot just fine and detect a new
> interface if you define one of the VFs as a PCI in the following
> manner within the VM definition file:
>
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0x01' slot='0x00' function='0x2'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> function='0x0'/>
>     </hostdev> 
>
> As you can see the from the source bus and function, it's the same VF
> I was trying with the definition above. I'm wondering if there is any
> particular reason for this or how could I get around this error.
> Normally, I would just use the second configuration, however I'm about
> to begin work within an openstack environment where the sriovnicswitch
> ml2 plugin will be used for networking. This plugin, when configuring
> passthrough interfaces uses the first definition that does not work
> and spits out that error. 
>
> Considering even the official libvirt tutorials suggest the first way,
> I feel like I must be missing something, or something broke and noone
> noticed in one of the latest updates. Any help would be greatly
> appreciated.
>

No, the problem is with the drivers for the particular cards you are
using. To my understanding, the Mellanox cards have a bit of a different
architecture, and were not initially designed to be SRIOV-compliant (or
maybe they 'comply differently', I'm not sure of the exact details).

I would recommend looking to Mellanox for updated drivers for the card
for CentOS7.

Alternately, you could test it with the RHEL7.1 beta, which will have
newer Mellanox bits than whatever CentOS7 you're currently working with.