<div dir="ltr">Hi Alex,<div><br></div><div>Thank you for the quick response.</div><div><br></div><div>i have 370+G RAM in the system</div><div><br></div><div><div>[root@localhost vcr]# cat /proc/meminfo</div><div>MemTotal: 396231416 kB</div><div>MemFree: 272301316 kB</div><div><br></div><div>the reason i moved towards pci-e, sr-iov, vfio is primarily that reason. i can get upto 1.8Gbps iMix traffic performance on vhost-virtio-passthrough mode, i am looking for upwards to 4Gbps performance per vNIC. </div><div><br></div><div>the error doesn't say what;s wrong,with my config.</div><div><br></div><div>appreciate your help.</div><div><br></div><div>-chintu-</div><div><br></div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, May 27, 2016 at 11:49 AM, Alex Williamson <span dir="ltr"><<a href="mailto:alex.williamson@redhat.com" target="_blank">alex.williamson@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On Fri, 27 May 2016 11:15:41 -0400<br>
chintu hetam <<a href="mailto:rometoroam@gmail.com">rometoroam@gmail.com</a>> wrote:<br>
<br>
> I have tried few things with libvirt, directly spawning VM with qemu-kvm<br>
> even tried withe centos7 as a guest and in all cases vfio fails.<br>
><br>
> here are the details!!<br>
><br>
> 1) Device: Intel X710/XL710<br>
> 2) virt xml snapshot:<br>
><br>
> <hostdev mode='subsystem' type='pci' managed='yes'><br>
> <driver name='vfio'/><br>
> <source><br>
> <address domain='0x0000' bus='0xaa' slot='0x02' function='0x0'/><br>
> </source><br>
> <alias name='hostdev0'/><br>
> <rom bar='on'/><br>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x09'<br>
> function='0x0'/><br>
> </hostdev><br>
> <hostdev mode='subsystem' type='pci' managed='yes'><br>
> <driver name='vfio'/><br>
> <source><br>
> <address domain='0x0000' bus='0xaa' slot='0x06' function='0x0'/><br>
> </source><br>
> <alias name='hostdev1'/><br>
> <rom bar='on'/><br>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b'<br>
> function='0x0'/><br>
> </hostdev><br>
><br>
> 3) ip link show snapshot:<br>
> 18: enp170s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop portid<br>
> 00e0ed5d884c state DOWN mode DEFAULT group default qlen 1000<br>
> link/ether 00:e0:ed:5d:88:4c brd ff:ff:ff:ff:ff:ff<br>
> vf 0 MAC aa:bb:cc:dd:ee:ff, spoof checking on, link-state auto<br>
> vf 1 MAC aa:bb:cc:dd:ee:fe, spoof checking on, link-state auto<br>
> vf 2 MAC aa:bb:cc:dd:ee:fd, spoof checking on, link-state auto<br>
> vf 3 MAC aa:bb:cc:dd:ee:fc, spoof checking on, link-state auto<br>
> 19: enp170s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop portid<br>
> 00e0ed5d884d state DOWN mode DEFAULT group default qlen 1000<br>
> link/ether 00:e0:ed:5d:88:4d brd ff:ff:ff:ff:ff:ff<br>
> vf 0 MAC aa:bb:cc:dd:ed:ff, spoof checking on, link-state auto<br>
> vf 1 MAC aa:bb:cc:dd:ed:fe, spoof checking on, link-state auto<br>
> vf 2 MAC aa:bb:cc:dd:ed:fd, spoof checking on, link-state auto<br>
> vf 3 MAC aa:bb:cc:dd:ed:fc, spoof checking on, link-state auto<br>
><br>
> 4) various system information<br>
> [root@localhost vcr]# modinfo i40evf<br>
> filename:<br>
> /lib/modules/4.4.6-300.fc23.x86_64/updates/drivers/net/ethernet/intel/i40evf/i40evf.ko<br>
> version: 1.5.14<br>
> license: GPL<br>
> description: Intel(R) 40-10 Gigabit Ethernet Virtual Function Driver<br>
> author: Intel Corporation, <<a href="mailto:linux.nics@intel.com">linux.nics@intel.com</a>><br>
> srcversion: 522158BAB323575B03D48C8<br>
> alias: pci:v00008086d0000154Csv*sd*bc*sc*i*<br>
> depends:<br>
> vermagic: 4.4.6-300.fc23.x86_64 SMP mod_unload<br>
><br>
> [root@localhost vcr]# modinfo i40e<br>
> filename:<br>
> /lib/modules/4.4.6-300.fc23.x86_64/kernel/drivers/net/ethernet/intel/i40e/i40e.ko.xz<br>
> version: 1.3.46-k<br>
> license: GPL<br>
> description: Intel(R) Ethernet Connection XL710 Network Driver<br>
> author: Intel Corporation, <<a href="mailto:e1000-devel@lists.sourceforge.net">e1000-devel@lists.sourceforge.net</a>><br>
> srcversion: A3FBA4AB3B91C574EA1FD38<br>
> alias: pci:v00008086d00001588sv*sd*bc*sc*i*<br>
> alias: pci:v00008086d00001587sv*sd*bc*sc*i*<br>
> alias: pci:v00008086d000037D2sv*sd*bc*sc*i*<br>
> alias: pci:v00008086d000037D1sv*sd*bc*sc*i*<br>
> alias: pci:v00008086d000037D0sv*sd*bc*sc*i*<br>
> alias: pci:v00008086d00001587sv*sd*bc*sc*i*<br>
> alias: pci:v00008086d00001589sv*sd*bc*sc*i*<br>
> alias: pci:v00008086d00001586sv*sd*bc*sc*i*<br>
> alias: pci:v00008086d00001585sv*sd*bc*sc*i*<br>
> alias: pci:v00008086d00001584sv*sd*bc*sc*i*<br>
> alias: pci:v00008086d00001583sv*sd*bc*sc*i*<br>
> alias: pci:v00008086d00001581sv*sd*bc*sc*i*<br>
> alias: pci:v00008086d00001580sv*sd*bc*sc*i*<br>
> alias: pci:v00008086d0000157Fsv*sd*bc*sc*i*<br>
> alias: pci:v00008086d00001574sv*sd*bc*sc*i*<br>
> alias: pci:v00008086d00001572sv*sd*bc*sc*i*<br>
> depends: ptp,vxlan<br>
> intree: Y<br>
> vermagic: 4.4.6-300.fc23.x86_64 SMP mod_unload<br>
> parm: debug:Debug level (0=none,...,16=all) (int)<br>
><br>
> However, in libvirt xml if i change driver to kvm it works but then as i<br>
> understand it's e1000 equivalent and vfio is virtio performance equivalent .<br>
<br>
</div></div>Not remotely, in this context 'kvm' vs 'vfio' are both types of device<br>
assignment, but kvm is deprecated here and will be going away at some<br>
point (only legacy kvm device assignment, not the remainder of kvm). If<br>
you want virtio performance equivalent, use virtio, PCI device<br>
assignment offers better performance than virtio.<br>
<div><div class="h5"><br>
> 5) lspci -tvv snapshot<br>
><br>
><br>
> | +-03.0-[aa-ab]--+-00.0 Intel Corporation Ethernet Controller<br>
> X710 for 10GbE SFP+<br>
> | | +-00.1 Intel Corporation Ethernet Controller<br>
> X710 for 10GbE SFP+<br>
> | | +-00.2 Intel Corporation Ethernet Controller<br>
> X710 for 10GbE SFP+<br>
> | | +-00.3 Intel Corporation Ethernet Controller<br>
> X710 for 10GbE SFP+<br>
> | | +-02.0 Intel Corporation XL710/X710 Virtual<br>
> Function<br>
> | | +-02.1 Intel Corporation XL710/X710 Virtual<br>
> Function<br>
> | | +-02.2 Intel Corporation XL710/X710 Virtual<br>
> Function<br>
> | | +-02.3 Intel Corporation XL710/X710 Virtual<br>
> Function<br>
> | | +-06.0 Intel Corporation XL710/X710 Virtual<br>
> Function<br>
> | | +-06.1 Intel Corporation XL710/X710 Virtual<br>
> Function<br>
> | | +-06.2 Intel Corporation XL710/X710 Virtual<br>
> Function<br>
> | | \-06.3 Intel Corporation XL710/X710 Virtual<br>
> Function<br>
><br>
> 6) FreeBSD guest virtual function driver version is 1.2.11<br>
><br>
> 7) error log doesn't say much other than resource unavailable<br>
> 2016-05-27 15:08:28.804+0000: starting up libvirt version: 1.2.18.2,<br>
> package: 3.fc23 (Fedora Project, 2016-03-17-21:40:48,<br>
> <a href="http://buildvm-22.phx2.fedoraproject.org" rel="noreferrer" target="_blank">buildvm-22.phx2.fedoraproject.org</a>), qemu version: 2.4.1 (qemu-2.4.1-9.fc23)<br>
> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin<br>
> QEMU_AUDIO_DRV=spice /usr/bin/qemu-kvm -name generic -S -machine<br>
> pc-i440fx-2.4,accel=kvm,usb=off,vmport=off,mem-merge=off -cpu SandyBridge<br>
> -m 32768 -mem-prealloc -mem-path /dev/hugepages/libvirt/qemu -realtime<br>
> mlock=on -smp 8,sockets=8,cores=1,threads=1 -uuid<br>
> d8f61bc1-dd9a-470f-a26f-824e5e2b8359 -no-user-config -nodefaults -chardev<br>
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/generic.monitor,server,nowait<br>
> -mon chardev=charmonitor,id=monitor,mode=control -rtc<br>
> base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet<br>
> -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1<br>
> -boot order=c,menu=on,strict=on -device<br>
> ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device<br>
> ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6<br>
> -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1<br>
> -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2<br>
> -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive<br>
> file=/home/vcr/Appliance-4.6.0.0_5-vtnet0.img,if=none,id=drive-virtio-disk0,format=raw<br>
> -device<br>
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk0,id=virtio-disk0<br>
> -drive<br>
> file=/dev/disk/by-id/scsi-36001405fdc0e99da3aa4b37b1403085c,if=none,id=drive-virtio-disk1,format=raw<br>
> -device<br>
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0xa,drive=drive-virtio-disk1,id=virtio-disk1<br>
> -netdev tap,fd=24,id=hostnet0,vhost=on,vhostfd=25 -device<br>
> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:30:a0:c0,bus=pci.0,addr=0x3<br>
> -chardev pty,id=charserial0 -device<br>
> isa-serial,chardev=charserial0,id=serial0 -chardev<br>
> spicevmc,id=charchannel0,name=vdagent -device<br>
> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0<br>
> -spice<br>
> port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on<br>
> -device<br>
> qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2<br>
> -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device<br>
> hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev<br>
> spicevmc,id=charredir0,name=usbredir -device<br>
> usb-redir,chardev=charredir0,id=redir0 -chardev<br>
> spicevmc,id=charredir1,name=usbredir -device<br>
> usb-redir,chardev=charredir1,id=redir1 -device<br>
> vfio-pci,host=aa:02.0,id=hostdev0,bus=pci.0,addr=0x9,rombar=1 -device<br>
> vfio-pci,host=aa:06.0,id=hostdev1,bus=pci.0,addr=0xb,rombar=1 -device<br>
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on<br>
> char device redirected to /dev/pts/2 (label charserial0)<br>
> qemu: qemu_thread_create: Resource temporarily unavailable<br>
> 2016-05-27 15:09:20.735+0000: shutting down<br>
><br>
> Any help will be appreciated.<br>
<br>
</div></div>You're creating a 32G VM, how much memory does the host have? Anything<br>
in dmesg? Thanks,<br>
<br>
Alex<br>
</blockquote></div><br></div>