<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv=Content-Type content="text/html; charset=us-ascii"> <meta name=Generator content="Microsoft Word 11 (filtered medium)"> <style>  </style> </head> <body lang=EN-US link=blue vlink=purple> <div class=Section1> <h3 style='mso-margin-top-alt:6.0pt;margin-right:0in;margin-bottom:0in; margin-left:0in;margin-bottom:.0001pt;background:white'>Hi <o:p></o:p></h3> <h3 style='mso-margin-top-alt:6.0pt;margin-right:0in;margin-bottom:0in; margin-left:0in;margin-bottom:.0001pt;background:white'><o:p> </o:p></h3> <h3 style='mso-margin-top-alt:6.0pt;margin-right:0in;margin-bottom:0in; margin-left:0in;margin-bottom:.0001pt;background:white'>I am trying to do passthrough to guest using sriov and libvirt<o:p></o:p></h3> <h3 style='mso-margin-top-alt:6.0pt;margin-right:0in;margin-bottom:0in; margin-left:0in;margin-bottom:.0001pt;background:white'><o:p> </o:p></h3> <h3 style='mso-margin-top-alt:6.0pt;margin-right:0in;margin-bottom:0in; margin-left:0in;margin-bottom:.0001pt;background:white'>I tried to use both techniques from here<o:p></o:p></h3> <a href="http://wiki.libvirt.org/page/Networking#Assignment_with_.3Chostdev.3E">http://wiki.libvirt.org/page/Networking#Assignment_with_.3Chostdev.3E</a><o:p></o:p> Assignment from a pool of SRIOV VFs in a libvirt <network> definition<o:p></o:p> Assignment with <interface type='hostdev'> (SRIOV devices only)<o:p></o:p> <h3 style='mso-margin-top-alt:6.0pt;margin-right:0in;margin-bottom:0in; margin-left:0in;margin-bottom:.0001pt;background:white'><o:p> </o:p></h3> <h3 style='mso-margin-top-alt:6.0pt;margin-right:0in;margin-bottom:0in; margin-left:0in;margin-bottom:.0001pt;background:white'><o:p> </o:p></h3> <network><o:p></o:p> <name>passthrough-eth6</name><o:p></o:p> <forward mode='hostdev' managed='yes'><o:p></o:p> <pf dev='eth6'/><o:p></o:p> </forward><o:p></o:p> </network><o:p></o:p> <h3 style='mso-margin-top-alt:6.0pt;margin-right:0in;margin-bottom:0in; margin-left:0in;margin-bottom:.0001pt;background:white'><o:p> </o:p></h3> <h3 style='mso-margin-top-alt:6.0pt;margin-right:0in;margin-bottom:0in; margin-left:0in;margin-bottom:.0001pt;background:white'><o:p> </o:p></h3> <h3 style='mso-margin-top-alt:6.0pt;margin-right:0in;margin-bottom:0in; margin-left:0in;margin-bottom:.0001pt;background:white'>I launch the vm, there is a vm with virt-install …. --network network=passthrough-eth6 <o:p></o:p></h3> <h3 style='mso-margin-top-alt:6.0pt;margin-right:0in;margin-bottom:0in; margin-left:0in;margin-bottom:.0001pt;background:white'>I inject traffic with a generator so I defined it in promisc mode without an ip address.(ifconfig eth6 promisc up)<o:p></o:p></h3> <h3 style='mso-margin-top-alt:6.0pt;margin-right:0in;margin-bottom:0in; margin-left:0in;margin-bottom:.0001pt;background:white'>In my physical device eth6 i can see RX increasing. But in the guest side nothing arrive in the RX? Why?<o:p></o:p></h3> <h3 style='mso-margin-top-alt:6.0pt;margin-right:0in;margin-bottom:0in; margin-left:0in;margin-bottom:.0001pt;background:white'>I don’t know in which log to look at to understand the problem<o:p></o:p></h3> <h3 style='mso-margin-top-alt:6.0pt;margin-right:0in;margin-bottom:0in; margin-left:0in;margin-bottom:.0001pt;background:white'>I defined the network-script/eth1 file in the guest is defined as static (not dhcp) as the interface it is connected to in host have no ip.<o:p></o:p></h3> <h3 style='mso-margin-top-alt:6.0pt;margin-right:0in;margin-bottom:0in; margin-left:0in;margin-bottom:.0001pt;background:white'>I have a another interface on the host connected via ovs and libvirt and it is working fine – you can see it in log as vnet0(thanks to this mailing-list help).<o:p></o:p></h3> <o:p> </o:p> In the guest part I attached the log but I think interesting part is <o:p></o:p> Jan 27 17:03:21 localhost kernel: ixgbevf 0000:00:04.0: NIC Link is Up, 10 Gbps<o:p></o:p> <o:p> </o:p> I also attached dmesg output from guest<o:p></o:p> <h3 style='mso-margin-top-alt:6.0pt;margin-right:0in;margin-bottom:0in; margin-left:0in;margin-bottom:.0001pt;background:white'><o:p> </o:p></h3> Here is the log from the host side, I can’t see an error. /var/log/message <o:p></o:p> Jan 27 12:02:29 1235 avahi-daemon[2598]: Withdrawing address record for fe80::fc54:ff:fe92:7352 on vnet0.<o:p></o:p> Jan 27 12:02:29 1235 kernel: device vnet0 left promiscuous mode<o:p></o:p> Jan 27 12:02:30 1235 kernel: ixgbe 0000:15:00.0: setting MAC a2:fc:a8:89:2a:8f on VF 0<o:p></o:p> Jan 27 12:02:30 1235 ovs-vsctl: 00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --if-exists del-port vnet0<o:p></o:p> Jan 27 12:02:30 1235 ovs-vswitchd: 00180|bridge|ERR|bridge virbr0: mirror mirror-1-to-3 does not specify output; ignoring<o:p></o:p> Jan 27 12:02:31 1235 ntpd[2905]: Deleting interface #28 vnet0, fe80::fc54:ff:fe92:7352#123, interface stats: received=0, sent=0, dropped=0, active_time=981 secs<o:p></o:p> Jan 27 12:02:31 1235 ntpd[2905]: peers refreshed<o:p></o:p> Jan 27 12:02:43 1235 kernel: ixgbe 0000:15:00.0: Reload the VF driver to make this change effective.<o:p></o:p> Jan 27 12:02:43 1235 kernel: kvm: 7957: cpu0 unhandled rdmsr: 0x345<o:p></o:p> Jan 27 12:02:43 1235 kernel: kvm: 7957: cpu0 unhandled wrmsr: 0x680 data 0<o:p></o:p> …<o:p></o:p> Jan 27 12:02:51 1235 ovs-vsctl: 00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --may-exist add-port br3 vnet0 -- set Interface vnet0 "external-ids:attached-mac=\"52:54:00:43:BF:8C\"" -- set Interface vnet0 "external-ids:iface-id=\"309ed431-417f-2843-89b8-179a066aa595\"" -- set Interface vnet0 "external-ids:vm-id=\"870e3b65-953b-01ee-84a3-c2d05300c2d9\"" -- set Interface vnet0 external-ids:iface-status=active<o:p></o:p> Jan 27 12:02:51 1235 ovs-vswitchd: 00183|bridge|ERR|bridge virbr0: mirror mirror-1-to-3 does not specify output; ignoring<o:p></o:p> Jan 27 12:02:51 1235 kernel: device vnet0 entered promiscuous mode<o:p></o:p> Jan 27 12:02:51 1235 kernel: ixgbe 0000:15:00.0: setting MAC 52:54:00:d9:f1:bd on VF 0<o:p></o:p> Jan 27 12:02:51 1235 kernel: ixgbe 0000:15:00.0: Reload the VF driver to make this change effective.<o:p></o:p> Jan 27 12:02:51 1235 kernel: pci-stub 0000:15:10.0: enabling device (0000 -> 0002)<o:p></o:p> Jan 27 12:02:52 1235 avahi-daemon[2598]: Registering new address record for fe80::fc54:ff:fe43:bf8c on vnet0.*.<o:p></o:p> Jan 27 12:02:54 1235 ntpd[2905]: Listen normally on 29 vnet0 fe80::fc54:ff:fe43:bf8c UDP 123<o:p></o:p> Jan 27 12:02:54 1235 ntpd[2905]: peers refreshed<o:p></o:p> Jan 27 12:03:17 1235 kernel: __ratelimit: 23 callbacks suppressed<o:p></o:p> Jan 27 12:03:17 1235 kernel: kvm: 8006: cpu0 disabled perfctr wrmsr: 0xc1 data 0xabcd<o:p></o:p> Jan 27 12:03:20 1235 kernel: ixgbe 0000:15:00.0: eth6: VF Reset msg received from vf 0<o:p></o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> And this is the /var/log/libvirt/qemu/probe8.log from the host<o:p></o:p> 2015-01-27 17:02:51.762+0000: starting up<o:p></o:p> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name probe8 -S -M rhel6.6.0 -enable-kvm -m 4096 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 870e3b65-953b-01ee-84a3-c2d05300c2d9 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/probe8.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -drive file=/data/KVM/comp_VNF_MaveriQ_Probe2-cloud8.qcow2,if=none,id=drive-ide0-0-0,format=qcow2,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive file=/var/lib/libvirt/images/configuration8.iso,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,fd=25,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:43:bf:8c,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 0.0.0.0:0 -vga cirrus -device pci-assign,host=15:10.0,id=hostdev0,configfd=26,bus=pci.0,addr=0x4 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on<o:p></o:p> char device redirected to /dev/pts/1<o:p></o:p> <o:p> </o:p> I can see that virtual function succeeded because I got the following output<o:p></o:p> virsh net-dumpxml passthrough-eth6<o:p></o:p> <network connections='1'><o:p></o:p> <name>passthrough-eth6</name><o:p></o:p> <uuid>f8fd37cd-3215-e4b0-17b1-0b5515140db9</uuid><o:p></o:p> <forward mode='hostdev' managed='yes'><o:p></o:p> <pf dev='eth6'/><o:p></o:p> <address type='pci' domain='0x0000' bus='0x15' slot='0x10' function='0x0'/><o:p></o:p> <address type='pci' domain='0x0000' bus='0x15' slot='0x10' function='0x2'/><o:p></o:p> <address type='pci' domain='0x0000' bus='0x15' slot='0x10' function='0x4'/><o:p></o:p> <address type='pci' domain='0x0000' bus='0x15' slot='0x10' function='0x6'/><o:p></o:p> …<o:p></o:p> </forward><o:p></o:p> </network><o:p></o:p> <o:p> </o:p> lspci | grep Intel<o:p></o:p> 15:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)<o:p></o:p> 15:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)<o:p></o:p> 15:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)<o:p></o:p> 15:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)<o:p></o:p> ….<o:p></o:p> <o:p> </o:p> <o:p> </o:p> And Intel-IOMMU: enabled in the dmesg<o:p></o:p> <o:p> </o:p> From host dmesg<o:p></o:p> <o:p> </o:p> ixgbe 0000:15:00.0: setting MAC 52:54:00:97:c6:86 on VF 0<o:p></o:p> ixgbe 0000:15:00.0: Reload the VF driver to make this change effective.<o:p></o:p> device vnet0 entered promiscuous mode<o:p></o:p> pci-stub 0000:15:10.0: enabling device (0000 -> 0002)<o:p></o:p> assign device 0:15:10.0<o:p></o:p> vnet0: no IPv6 routers present<o:p></o:p> __ratelimit: 23 callbacks suppressed<o:p></o:p> kvm: 6896: cpu0 disabled perfctr wrmsr: 0xc1 data 0xabcd<o:p></o:p> ixgbe 0000:15:00.0: eth6: VF Reset msg received from vf 0<o:p></o:p> pci-stub 0000:15:10.0: irq 62 for MSI/MSI-X<o:p></o:p> pci-stub 0000:15:10.0: irq 63 for MSI/MSI-X<o:p></o:p> pci-stub 0000:15:10.0: irq 62 for MSI/MSI-X<o:p></o:p> pci-stub 0000:15:10.0: irq 63 for MSI/MSI-X<o:p></o:p> ixgbe 0000:15:00.0: eth6: VF Reset msg received from vf 3<o:p></o:p> pci-stub 0000:15:10.6: claimed by stub<o:p></o:p> device vnet0 left promiscuous mode<o:p></o:p> pci-stub 0000:15:10.0: restoring config space at offset 0x1 (was 0x100000, writing 0x100004)<o:p></o:p> ixgbe 0000:15:00.0: setting MAC a2:fc:a8:89:2a:8f on VF 0<o:p></o:p> ixgbe 0000:15:00.0: Reload the VF driver to make this change effective.<o:p></o:p> kvm: 7207: cpu0 unhandled rdmsr: 0x345<o:p></o:p> kvm: 7207: cpu0 unhandled wrmsr: 0x680 data 0<o:p></o:p> …<o:p></o:p> ixgbe 0000:15:00.0: setting MAC 52:54:00:16:b6:0c on VF 3<o:p></o:p> ixgbe 0000:15:00.0: Reload the VF driver to make this change effective.<o:p></o:p> device vnet0 entered promiscuous mode<o:p></o:p> pci-stub 0000:15:10.6: enabling device (0000 -> 0002)<o:p></o:p> assign device 0:15:10.6<o:p></o:p> vnet0: no IPv6 routers present<o:p></o:p> __ratelimit: 24 callbacks suppressed<o:p></o:p> kvm: 7254: cpu0 disabled perfctr wrmsr: 0xc1 data 0xabcd<o:p></o:p> ixgbe 0000:15:00.0: eth6: VF Reset msg received from vf 3<o:p></o:p> pci-stub 0000:15:10.6: irq 62 for MSI/MSI-X<o:p></o:p> pci-stub 0000:15:10.6: irq 63 for MSI/MSI-X<o:p></o:p> pci-stub 0000:15:10.6: irq 62 for MSI/MSI-X<o:p></o:p> pci-stub 0000:15:10.6: irq 63 for MSI/MSI-X<o:p></o:p> device vnet0 left promiscuous mode<o:p></o:p> pci-stub 0000:15:10.6: restoring config space at offset 0x1 (was 0x100000, writing 0x100004)<o:p></o:p> ixgbe 0000:15:00.0: setting MAC ba:c1:bd:19:c1:d5 on VF 3<o:p></o:p> ixgbe 0000:15:00.0: Reload the VF driver to make this change effective.<o:p></o:p> device vnet0 entered promiscuous mode<o:p></o:p> ixgbe 0000:15:00.0: setting MAC 52:54:00:e2:39:62 on VF 0<o:p></o:p> ixgbe 0000:15:00.0: Reload the VF driver to make this change effective.<o:p></o:p> pci-stub 0000:15:10.0: enabling device (0000 -> 0002)<o:p></o:p> assign device 0:15:10.0<o:p></o:p> vnet0: no IPv6 routers present<o:p></o:p> kvm: 7639: cpu0 disabled perfctr wrmsr: 0xc1 data 0xabcd<o:p></o:p> ixgbe 0000:15:00.0: eth6: VF Reset msg received from vf 0<o:p></o:p> pci-stub 0000:15:10.0: irq 62 for MSI/MSI-X<o:p></o:p> pci-stub 0000:15:10.0: irq 63 for MSI/MSI-X<o:p></o:p> pci-stub 0000:15:10.0: irq 62 for MSI/MSI-X<o:p></o:p> pci-stub 0000:15:10.0: irq 63 for MSI/MSI-X<o:p></o:p> device vnet0 left promiscuous mode<o:p></o:p> pci-stub 0000:15:10.0: restoring config space at offset 0x1 (was 0x100000, writing 0x100004)<o:p></o:p> ixgbe 0000:15:00.0: setting MAC a2:fc:a8:89:2a:8f on VF 0<o:p></o:p> ixgbe 0000:15:00.0: Reload the VF driver to make this change effective.<o:p></o:p> kvm: 7957: cpu0 unhandled rdmsr: 0x345<o:p></o:p> kvm: 7957: cpu0 unhandled wrmsr: 0x680 data 0<o:p></o:p> …<o:p></o:p> device vnet0 entered promiscuous mode<o:p></o:p> ixgbe 0000:15:00.0: setting MAC 52:54:00:d9:f1:bd on VF 0<o:p></o:p> ixgbe 0000:15:00.0: Reload the VF driver to make this change effective.<o:p></o:p> pci-stub 0000:15:10.0: enabling device (0000 -> 0002)<o:p></o:p> assign device 0:15:10.0<o:p></o:p> vnet0: no IPv6 routers present<o:p></o:p> __ratelimit: 23 callbacks suppressed<o:p></o:p> kvm: 8006: cpu0 disabled perfctr wrmsr: 0xc1 data 0xabcd<o:p></o:p> ixgbe 0000:15:00.0: eth6: VF Reset msg received from vf 0<o:p></o:p> pci-stub 0000:15:10.0: irq 62 for MSI/MSI-X<o:p></o:p> …<o:p></o:p> </div> </body> </html>