[vfio-users] Cannot passthrough ixgbe virtual functions to QEMU VM

[Vincenzo Maffione]

Hi,
  I'm having an insanely hard time trying to pass an SR-IOV Virtual
Function (VF) to a QEMU VM.

I've followed the instructions at http://www.linux-kvm.org/page/
10G_NIC_performance:_VFIO_vs_virtio
and I successfully managed to create one (or more) VFs on my ixgbe NIC.

My host kernel is Linux 4.10.8, patched with the ACS override patch, as
suggested here
https://wiki.archlinux.org/index.php/PCI_passthrough_via_
OVMF#Bypassing_the_IOMMU_groups_.28ACS_override_patch.29

I use QEMU upstream master branch, updated today.

And my boot parameters are

BOOT_IMAGE=../vmlinuz-linux-vfio root=UUID=1c777330-3f00-4ab8-8d5d-d4b2b5d860b0
rw intel_iommu=on hugepages=2048 pci=assign-busses
pcie_acs_override=downstream initrd=../initramfs-linux-vfio.img

Thanks to the ACS patched kernel, I'm able to successfully passthrough two
different ixgbe physical functions (PF) to two different QEMU machines. Due
to the IOMMU groups limitations of my hardware, that was not possible on
the vanilla 4.10.8 kernel.

Once I create a VF from one of the PFs I have, I unbind that from the
ixgbevf driver, and bind it to the vfio-pci driver. Then I use the
following QEMU cmdline to do the PCI passthrough (the same command that I
use to successfully passthrough PFs):

# qemu-system-x86_64 /path/to/image.qcow2 -enable-kvm -smp 2 -m 2G -vga std
-nographic -snapshot -device e1000,netdev=mgmt,mac=00:AA:BB:CC:0a:99
-netdev user,id=mgmt,hostfwd=tcp::20010-:22  -device vfio-pci,host=02:10.1

where 02:10.1 is the PCI identifier of the VF.

However, QEMU immediately crashes, reporting a full register dump:

************************************************************
******************************************
qemu-system-x86_64: VFIO_MAP_DMA: -14
qemu-system-x86_64: vfio_dma_map(0x56017b4b75b0, 0xfebf0000, 0x4000,
0x7fcb60724000) = -14 (Bad address)
qemu: hardware error: vfio: DMA mapping failed, unable to continue
CPU #0:
EAX=00000103 EBX=80002004 ECX=00000103 EDX=00000cfc
ESI=00002000 EDI=00000000 EBP=7ffab670 ESP=00006d34
EIP=000eda90 EFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
SS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
FS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
GS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
GDT=     000f7070 00000037
IDT=     000f70ae 00000000
CR0=00000011 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=00000000000000000000000000000000 XMM01=000000000000000000000000000000
00
XMM02=00000000000000000000000000000000 XMM03=000000000000000000000000000000
00
XMM04=00000000000000000000000000000000 XMM05=000000000000000000000000000000
00
XMM06=00000000000000000000000000000000 XMM07=000000000000000000000000000000
00
CPU #1:
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000663
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 ffff0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=00000000000000000000000000000000 XMM01=000000000000000000000000000000
00
XMM02=00000000000000000000000000000000 XMM03=000000000000000000000000000000
00
XMM04=00000000000000000000000000000000 XMM05=000000000000000000000000000000
00
XMM06=00000000000000000000000000000000 XMM07=000000000000000000000000000000
00
QEMU terminated with an exception
************************************************************
******************************************

I've seen similar errors have been reported in various mailing lists and
blog, but I was not able to relate them to my specific problem, nor find
any solution.

Can anyone help?

Thanks,
  Vincenzo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20170414/89066948/attachment.htm>