[libvirt] libvirt VF indexing issue

Tue Oct 22 13:50:57 UTC 2013

On 10/17/2013 10:43 AM, Niilona wrote:
> libvirt points to wrong VF -channel, when trying to change MAC -address.
>
> libvirt v1 1.0.2
> linux driver : ixgbe 3.17.3
>
> Symptom description
> ----------------------------------
>
>
> 1.) How the VF channels are seen in the Host ( correct in increasing
> address order )
>
> command : ls -la  /sys/bus/pci/devices/0000\:04\:00.0/
> ..................
>
> lrwxrwxrwx  1 root root      0 Oct 17 12:25 virtfn0 -> ../0000:04:10.0/
> lrwxrwxrwx  1 root root      0 Oct 17 12:25 virtfn1 -> ../0000:04:10.2/
> lrwxrwxrwx  1 root root      0 Oct 17 12:25 virtfn10 -> ../0000:04:12.4/
> lrwxrwxrwx  1 root root      0 Oct 17 12:25 virtfn11 -> ../0000:04:12.6/
> lrwxrwxrwx  1 root root      0 Oct 17 12:25 virtfn12 -> ../0000:04:13.0/
> lrwxrwxrwx  1 root root      0 Oct 17 12:25 virtfn13 -> ../0000:04:13.2/
> lrwxrwxrwx  1 root root      0 Oct 17 12:25 virtfn2 -> ../0000:04:10.4/
> lrwxrwxrwx  1 root root      0 Oct 17 12:25 virtfn3 -> ../0000:04:10.6/
> lrwxrwxrwx  1 root root      0 Oct 17 12:25 virtfn4 -> ../0000:04:11.0/
> lrwxrwxrwx  1 root root      0 Oct 17 12:25 virtfn5 -> ../0000:04:11.2/
> lrwxrwxrwx  1 root root      0 Oct 17 12:25 virtfn6 -> ../0000:04:11.4/
> lrwxrwxrwx  1 root root      0 Oct 17 12:25 virtfn7 -> ../0000:04:11.6/
> lrwxrwxrwx  1 root root      0 Oct 17 12:25 virtfn8 -> ../0000:04:12.0/
> lrwxrwxrwx  1 root root      0 Oct 17 12:25 virtfn9 -> ../0000:04:12.2/
>
> -----------------------
>
> 2.) How the libvirt sees them
>
> virsh nodedev-dumpxml pci_0000_04_00_0
> <device>
>   <name>pci_0000_04_00_0</name>
>   <parent>pci_0000_00_03_0</parent>
>   <driver>
>     <name>ixgbe</name>
>   </driver>
>   <capability type='pci'>
>     <domain>0</domain>
>     <bus>4</bus>
>     <slot>0</slot>
>     <function>0</function>
>     <product id='0x10fb'>82599EB 10-Gigabit SFI/SFP+ Network
> Connection</product>
>     <vendor id='0x8086'>Intel Corporation</vendor>
>     <capability type='virt_functions'>
>       <address domain='0x0000' bus='0x04' slot='0x12' function='0x4'/>
>       <address domain='0x0000' bus='0x04' slot='0x12' function='0x6'/>
>       <address domain='0x0000' bus='0x04' slot='0x13' function='0x0'/>
>       <address domain='0x0000' bus='0x04' slot='0x13' function='0x2'/>
>       <address domain='0x0000' bus='0x04' slot='0x10' function='0x0'/>
>       <address domain='0x0000' bus='0x04' slot='0x10' function='0x2'/>
>       <address domain='0x0000' bus='0x04' slot='0x10' function='0x4'/>
>       <address domain='0x0000' bus='0x04' slot='0x10' function='0x6'/>
>       <address domain='0x0000' bus='0x04' slot='0x11' function='0x0'/>
>       <address domain='0x0000' bus='0x04' slot='0x11' function='0x2'/>
>       <address domain='0x0000' bus='0x04' slot='0x11' function='0x4'/>
>       <address domain='0x0000' bus='0x04' slot='0x11' function='0x6'/>
>       <address domain='0x0000' bus='0x04' slot='0x12' function='0x0'/>
>       <address domain='0x0000' bus='0x04' slot='0x12' function='0x2'/>
>     </capability>
>   </capability>
> </device>
>
> 3.) Libvirt sets MAC -address in slot/position 5 instead of 1
>
> ip link show eth10
> 87: eth10: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 2100 qdisc mq
> state UP mode DEFAULT qlen 1000
>     link/ether 00:1b:21:b9:a5:60 brd ff:ff:ff:ff:ff:ff
>     vf 0 MAC 92:e2:6b:53:a7:c2, spoof checking off <------ this is the
> slot, MAC tried to set
>     vf 1 MAC 0a:04:38:2c:79:35, spoof checking off
>     vf 2 MAC b2:46:b6:91:44:2e, spoof checking off
>     vf 3 MAC de:b6:28:da:20:44, spoof checking off
>     vf 4 MAC 02:00:00:80:00:01, spoof checking off <--------- MAC sets
> in wrong slot
>     vf 5 MAC 02:00:00:80:00:02, spoof checking off
>     vf 6 MAC 02:00:00:80:00:20, spoof checking off
>     vf 7 MAC 02:00:00:80:00:30, spoof checking off
>     vf 8 MAC 52:54:00:de:f8:08, spoof checking off
>     vf 9 MAC 5e:b0:d0:34:d6:cc, spoof checking off
>     vf 10 MAC d2:88:5c:2d:db:b5, spoof checking off
>     vf 11 MAC 6a:06:8a:9c:31:8b, spoof checking off
>     vf 12 MAC 4a:38:be:8f:b3:bf, spoof checking off
>     vf 13 MAC b6:c1:d4:12:d3:3d, spoof checking off
>
> 5.) Question : Why this happens in ixgbe -driver and when we try to
> use more than 10 VF's/physical port. ?

1) You haven't included any of the libvirt config that led to this
situation, thus it isn't possible for us to see the supposed mismatch
between which VF was assigned to the guest and which VF had its MAC
address changed. Please include the <interface> section of the guest
config, as well as any applicable <network> config (if you are
allocating the VF from a network pool).

2) I have noticed in the past that the MAC address shown for a VF by "ip
link show" did not match the MAC address seen by the guest (and used for
the packets that egress from the guest). Have you verified that the
guest truly isn't seeing the device with the proper MAC address? Take a
look at the output of ip link show on the guest.

3) Are you running the stock igxbe driver? There was a recent case of a
user whose VFs were not getting the proper vlan ID set, and after much
investigation, someone noticed that this user had installed some forked
version of the ixgbe driver which was faulty.