[libvirt] sr-iov ethernet VF is not assigned correctly to guest

Scott ph3lpz at gmail.com
Mon Mar 26 15:12:56 UTC 2012 

Hi,  I'm hoping someone could possibly show me what I may be doing wrong here, and why I'm not seeing the expected results

Thanks in advance!

Server Information:HP Product Name: ProLiant DL165 G7
Ethernet Card Information: Intel 82599EB 10-Gigabit SFI/SFP+
uname: Linux 3.0.0-16-server x86_64 GNU/Linux
OS Release: Ubuntu 11.10
qemu-kvm version 0.14.1+noroms-0ubuntu6.2

First, here's a list of my VFs:
# ip link show:
2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000
   link/ether 00:1b:21:7d:70:16 brd ff:ff:ff:ff:ff:ff
   vf 0 MAC 02:22:25:00:00:00
   vf 1 MAC 02:22:25:00:00:01
   vf 2 MAC 02:22:25:00:00:02
   vf 3 MAC 02:22:25:00:00:03
   vf 4 MAC 02:22:25:00:00:04
   vf 5 MAC 02:22:25:00:00:05
   vf 6 MAC 02:22:25:00:00:06
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN qlen 1000
   link/ether 00:1b:21:7d:70:16 brd ff:ff:ff:ff:ff:ff
   vf 0 MAC 02:22:25:01:00:00
   vf 1 MAC 02:22:25:01:00:01
   vf 2 MAC 02:22:25:01:00:02
   vf 3 MAC 02:22:25:01:00:03
   vf 4 MAC 02:22:25:01:00:04
   vf 5 MAC 02:22:25:01:00:05
   vf 6 MAC 02:22:25:01:00:06

# lspci |grep 05
05:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
05:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

# virsh nodedev-list |grep 05
pci_0000_05_00_0
pci_0000_05_00_1
pci_0000_05_10_0
pci_0000_05_10_1
pci_0000_05_10_2
pci_0000_05_10_3
pci_0000_05_10_4
pci_0000_05_10_5
pci_0000_05_10_6
pci_0000_05_10_7
pci_0000_05_11_0
pci_0000_05_11_1
pci_0000_05_11_2
pci_0000_05_11_3
pci_0000_05_11_4
pci_0000_05_11_5


4 Now detach 05.10.0 and 05.11.0 from the host:
# virsh nodedev-dettach pci_0000_05_10_0
Device pci_0000_05_10_0 dettached

# virsh nodedev-dettach pci_0000_05_11_0
Device pci_0000_05_11_0 dettached

5. Get the host, slot, function information:
# virsh nodedev-dumpxml pci_0000_05_10_0|grep -e bus -e slot -e fun |head -3
   <bus>5</bus>
   <slot>16</slot>
   <function>0</function>

# virsh nodedev-dumpxml pci_0000_05_11_0|grep -e bus -e slot -e fun |head -3
   <bus>5</bus>
   <slot>17</slot>
   <function>0</function>


== GUEST-A ==
XML:
   ...
   <hostdev mode='subsystem' type='pci' managed='yes'>
     <source>
       <address domain='0x0000' bus='0x05' slot='0x10' function='0x0'/>
     </source>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
   </hostdev>
   <hostdev mode='subsystem' type='pci' managed='yes'>
     <source>
       <address domain='0x0000' bus='0x05' slot='0x11' function='0x0'/>
     </source>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
   </hostdev>
  ...

CLI: 
/usr/bin/kvm -S \
-M pc-0.14 \
-enable-kvm \
-m 256 \
-smp 1,sockets=1,cores=1,threads=1 \
-name GUEST-A \
-uuid 9bac7b2f-dbb4-0f2a-08e6-47018d6630db \
-nodefconfig -nodefaults 
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/HOST-A.monitor,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=readline \
-rtc base=utc \
-boot cn \ 
-drive file=/srv/node/sdc1/HOST-A.raw,if=none,id=drive-virtio-disk0,boot=on,format=raw,cache=none \
-device virtio-blk-pci,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-usb \
-vnc 0.0.0.0:0 \
-vga cirrus \
-device pci-assign,host=05:10.0,id=hostdev0,configfd=20,bus=pci.0,addr=0x4 \
-device pci-assign,host=05:11.0,id=hostdev1,configfd=21,bus=pci.0,addr=0x5 \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6

dmesg:

[    3.072509] igbvf 0000:00:04.0: Intel(R) 82576 Virtual Function
[    3.072514] igbvf 0000:00:04.0: Address: 02:22:25:00:00:00
[    3.072516] igbvf 0000:00:04.0: MAC: 1
[    3.241325] igbvf 0000:00:05.0: Intel(R) 82576 Virtual Function
[    3.241329] igbvf 0000:00:05.0: Address: 02:22:25:00:00:04
[    3.241332] igbvf 0000:00:05.0: MAC: 1

ip link:
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
   link/ether 02:22:25:00:00:00 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
   link/ether 02:22:25:00:00:04 brd ff:ff:ff:ff:ff:ff


Expected Results:
GUEST-A should have the following assignment:
eth0 should = eth0-vf0 (bus:5, slot:10, fun:0), mac: 02:22:25:00:00:00
eth1 should = eth1-vf0 (bus:5, slot:11, fun:0), mac: 02:22:25:01:00:00

More information about the libvir-list mailing list