[vfio-users] vfio fails Guest FreeBSD9.3 host Fedora 23

Alex Williamson alex.williamson at redhat.com
Fri May 27 15:49:16 UTC 2016 

On Fri, 27 May 2016 11:15:41 -0400
chintu hetam <rometoroam at gmail.com> wrote:

> I have tried few things with libvirt, directly spawning VM with qemu-kvm
> even tried withe centos7 as a guest and in all cases vfio fails.
> 
> here are the details!!
> 
> 1) Device: Intel X710/XL710
> 2) virt xml snapshot:
> 
>    <hostdev mode='subsystem' type='pci' managed='yes'>
>       <driver name='vfio'/>
>       <source>
>         <address domain='0x0000' bus='0xaa' slot='0x02' function='0x0'/>
>       </source>
>       <alias name='hostdev0'/>
>       <rom bar='on'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
> function='0x0'/>  
>     </hostdev>
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <driver name='vfio'/>
>       <source>
>         <address domain='0x0000' bus='0xaa' slot='0x06' function='0x0'/>
>       </source>
>       <alias name='hostdev1'/>
>       <rom bar='on'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x0b'
> function='0x0'/>  
>     </hostdev>
> 
> 3) ip link show snapshot:
> 18: enp170s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop portid
> 00e0ed5d884c state DOWN mode DEFAULT group default qlen 1000
>     link/ether 00:e0:ed:5d:88:4c brd ff:ff:ff:ff:ff:ff
>     vf 0 MAC aa:bb:cc:dd:ee:ff, spoof checking on, link-state auto
>     vf 1 MAC aa:bb:cc:dd:ee:fe, spoof checking on, link-state auto
>     vf 2 MAC aa:bb:cc:dd:ee:fd, spoof checking on, link-state auto
>     vf 3 MAC aa:bb:cc:dd:ee:fc, spoof checking on, link-state auto
> 19: enp170s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop portid
> 00e0ed5d884d state DOWN mode DEFAULT group default qlen 1000
>     link/ether 00:e0:ed:5d:88:4d brd ff:ff:ff:ff:ff:ff
>     vf 0 MAC aa:bb:cc:dd:ed:ff, spoof checking on, link-state auto
>     vf 1 MAC aa:bb:cc:dd:ed:fe, spoof checking on, link-state auto
>     vf 2 MAC aa:bb:cc:dd:ed:fd, spoof checking on, link-state auto
>     vf 3 MAC aa:bb:cc:dd:ed:fc, spoof checking on, link-state auto
> 
> 4) various system information
> [root at localhost vcr]# modinfo i40evf
> filename:
> /lib/modules/4.4.6-300.fc23.x86_64/updates/drivers/net/ethernet/intel/i40evf/i40evf.ko
> version:        1.5.14
> license:        GPL
> description:    Intel(R) 40-10 Gigabit Ethernet Virtual Function Driver
> author:         Intel Corporation, <linux.nics at intel.com>
> srcversion:     522158BAB323575B03D48C8
> alias:          pci:v00008086d0000154Csv*sd*bc*sc*i*
> depends:
> vermagic:       4.4.6-300.fc23.x86_64 SMP mod_unload
> 
> [root at localhost vcr]# modinfo i40e
> filename:
> /lib/modules/4.4.6-300.fc23.x86_64/kernel/drivers/net/ethernet/intel/i40e/i40e.ko.xz
> version:        1.3.46-k
> license:        GPL
> description:    Intel(R) Ethernet Connection XL710 Network Driver
> author:         Intel Corporation, <e1000-devel at lists.sourceforge.net>
> srcversion:     A3FBA4AB3B91C574EA1FD38
> alias:          pci:v00008086d00001588sv*sd*bc*sc*i*
> alias:          pci:v00008086d00001587sv*sd*bc*sc*i*
> alias:          pci:v00008086d000037D2sv*sd*bc*sc*i*
> alias:          pci:v00008086d000037D1sv*sd*bc*sc*i*
> alias:          pci:v00008086d000037D0sv*sd*bc*sc*i*
> alias:          pci:v00008086d00001587sv*sd*bc*sc*i*
> alias:          pci:v00008086d00001589sv*sd*bc*sc*i*
> alias:          pci:v00008086d00001586sv*sd*bc*sc*i*
> alias:          pci:v00008086d00001585sv*sd*bc*sc*i*
> alias:          pci:v00008086d00001584sv*sd*bc*sc*i*
> alias:          pci:v00008086d00001583sv*sd*bc*sc*i*
> alias:          pci:v00008086d00001581sv*sd*bc*sc*i*
> alias:          pci:v00008086d00001580sv*sd*bc*sc*i*
> alias:          pci:v00008086d0000157Fsv*sd*bc*sc*i*
> alias:          pci:v00008086d00001574sv*sd*bc*sc*i*
> alias:          pci:v00008086d00001572sv*sd*bc*sc*i*
> depends:        ptp,vxlan
> intree:         Y
> vermagic:       4.4.6-300.fc23.x86_64 SMP mod_unload
> parm:           debug:Debug level (0=none,...,16=all) (int)
> 
> However, in libvirt xml if i change driver to kvm it works but then as i
> understand it's e1000 equivalent and vfio is virtio performance equivalent .

Not remotely, in this context 'kvm' vs 'vfio' are both types of device
assignment, but kvm is deprecated here and will be going away at some
point (only legacy kvm device assignment, not the remainder of kvm).  If
you want virtio performance equivalent, use virtio, PCI device
assignment offers better performance than virtio.
 
> 5) lspci -tvv snapshot
> 
> 
>  |           +-03.0-[aa-ab]--+-00.0  Intel Corporation Ethernet Controller
> X710 for 10GbE SFP+
>  |           |               +-00.1  Intel Corporation Ethernet Controller
> X710 for 10GbE SFP+
>  |           |               +-00.2  Intel Corporation Ethernet Controller
> X710 for 10GbE SFP+
>  |           |               +-00.3  Intel Corporation Ethernet Controller
> X710 for 10GbE SFP+
>  |           |               +-02.0  Intel Corporation XL710/X710 Virtual
> Function
>  |           |               +-02.1  Intel Corporation XL710/X710 Virtual
> Function
>  |           |               +-02.2  Intel Corporation XL710/X710 Virtual
> Function
>  |           |               +-02.3  Intel Corporation XL710/X710 Virtual
> Function
>  |           |               +-06.0  Intel Corporation XL710/X710 Virtual
> Function
>  |           |               +-06.1  Intel Corporation XL710/X710 Virtual
> Function
>  |           |               +-06.2  Intel Corporation XL710/X710 Virtual
> Function
>  |           |               \-06.3  Intel Corporation XL710/X710 Virtual
> Function
> 
> 6) FreeBSD guest virtual function driver version is 1.2.11
> 
> 7) error log doesn't say much other than resource unavailable
> 2016-05-27 15:08:28.804+0000: starting up libvirt version: 1.2.18.2,
> package: 3.fc23 (Fedora Project, 2016-03-17-21:40:48,
> buildvm-22.phx2.fedoraproject.org), qemu version: 2.4.1 (qemu-2.4.1-9.fc23)
> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
> QEMU_AUDIO_DRV=spice /usr/bin/qemu-kvm -name generic -S -machine
> pc-i440fx-2.4,accel=kvm,usb=off,vmport=off,mem-merge=off -cpu SandyBridge
> -m 32768 -mem-prealloc -mem-path /dev/hugepages/libvirt/qemu -realtime
> mlock=on -smp 8,sockets=8,cores=1,threads=1 -uuid
> d8f61bc1-dd9a-470f-a26f-824e5e2b8359 -no-user-config -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/generic.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc
> base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet
> -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1
> -boot order=c,menu=on,strict=on -device
> ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device
> ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6
> -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1
> -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2
> -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive
> file=/home/vcr/Appliance-4.6.0.0_5-vtnet0.img,if=none,id=drive-virtio-disk0,format=raw
> -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk0,id=virtio-disk0
> -drive
> file=/dev/disk/by-id/scsi-36001405fdc0e99da3aa4b37b1403085c,if=none,id=drive-virtio-disk1,format=raw
> -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0xa,drive=drive-virtio-disk1,id=virtio-disk1
> -netdev tap,fd=24,id=hostnet0,vhost=on,vhostfd=25 -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:30:a0:c0,bus=pci.0,addr=0x3
> -chardev pty,id=charserial0 -device
> isa-serial,chardev=charserial0,id=serial0 -chardev
> spicevmc,id=charchannel0,name=vdagent -device
> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
> -spice
> port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on
> -device
> qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2
> -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
> hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev
> spicevmc,id=charredir0,name=usbredir -device
> usb-redir,chardev=charredir0,id=redir0 -chardev
> spicevmc,id=charredir1,name=usbredir -device
> usb-redir,chardev=charredir1,id=redir1 -device
> vfio-pci,host=aa:02.0,id=hostdev0,bus=pci.0,addr=0x9,rombar=1 -device
> vfio-pci,host=aa:06.0,id=hostdev1,bus=pci.0,addr=0xb,rombar=1 -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
> char device redirected to /dev/pts/2 (label charserial0)
> qemu: qemu_thread_create: Resource temporarily unavailable
> 2016-05-27 15:09:20.735+0000: shutting down
> 
> Any help will be appreciated.

You're creating a 32G VM, how much memory does the host have?  Anything
in dmesg?  Thanks,

Alex

More information about the vfio-users mailing list