[vfio-users] fitlet I211 PCI passthrough

Sat Mar 26 15:15:04 UTC 2016

On Sat, Mar 26, 2016 at 5:28 AM, YAEGASHI Takeshi <yaegashi at debian.org>
wrote:

> Hello,
>
> I have a fitlet http://www.fit-pc.com/web/products/fitlet/ with 4 GbE
> (I211) ports and wanted to passthrough some of them to KVM guests, ie
> assign one I211 to each guests.  I've tried various configurations
> with libvirt/kvm but no luck so far.
>
> After reading
> http://vfio.blogspot.jp/2014/08/iommu-groups-inside-and-out.html
> I've got sure that there's a problem in IOMMU groups.  Actually all of
> I211 and AMD's PCI bridge [1022:156b] share the same iommu_group 2.
>
> $ lspci -tv
> -[0000:00]-+-00.0  Advanced Micro Devices, Inc. [AMD] Device 1566
>            +-00.2  Advanced Micro Devices, Inc. [AMD] Device 1567
>            +-01.0  Advanced Micro Devices, Inc. [AMD/ATI] Mullins [Radeon
> R6 Graphics]
>            +-01.1  Advanced Micro Devices, Inc. [AMD/ATI] Kabini HDMI/DP
> Audio
>            +-02.0  Advanced Micro Devices, Inc. [AMD] Device 156b
>            +-02.2-[01]----00.0  Intel Corporation I211 Gigabit Network
> Connection
>            +-02.3-[02]----00.0  Intel Corporation I211 Gigabit Network
> Connection
>            +-02.4-[03]----00.0  Intel Corporation I211 Gigabit Network
> Connection
>            +-02.5-[04]----00.0  Intel Corporation I211 Gigabit Network
> Connection
>            +-08.0  Advanced Micro Devices, Inc. [AMD] Device 1537
>            +-10.0  Advanced Micro Devices, Inc. [AMD] FCH USB XHCI
> Controller
>            +-11.0  Advanced Micro Devices, Inc. [AMD] FCH SATA Controller
> [AHCI mode]
>            +-12.0  Advanced Micro Devices, Inc. [AMD] FCH USB EHCI
> Controller
>            +-13.0  Advanced Micro Devices, Inc. [AMD] FCH USB EHCI
> Controller
>            +-14.0  Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller
>            +-14.2  Advanced Micro Devices, Inc. [AMD] FCH Azalia Controller
>            +-14.3  Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge
>            +-14.7  Advanced Micro Devices, Inc. [AMD] FCH SD Flash
> Controller
>            +-18.0  Advanced Micro Devices, Inc. [AMD] Device 1580
>            +-18.1  Advanced Micro Devices, Inc. [AMD] Device 1581
>            +-18.2  Advanced Micro Devices, Inc. [AMD] Device 1582
>            +-18.3  Advanced Micro Devices, Inc. [AMD] Device 1583
>            +-18.4  Advanced Micro Devices, Inc. [AMD] Device 1584
>            \-18.5  Advanced Micro Devices, Inc. [AMD] Device 1585
>
> $ find /sys/kernel/iommu_groups -type l
> /sys/kernel/iommu_groups/0/devices/0000:00:00.0
> /sys/kernel/iommu_groups/1/devices/0000:00:01.0
> /sys/kernel/iommu_groups/1/devices/0000:00:01.1
> /sys/kernel/iommu_groups/2/devices/0000:00:02.0
> /sys/kernel/iommu_groups/2/devices/0000:00:02.2
> /sys/kernel/iommu_groups/2/devices/0000:00:02.3
> /sys/kernel/iommu_groups/2/devices/0000:00:02.4
> /sys/kernel/iommu_groups/2/devices/0000:00:02.5
> /sys/kernel/iommu_groups/2/devices/0000:01:00.0
> /sys/kernel/iommu_groups/2/devices/0000:02:00.0
> /sys/kernel/iommu_groups/2/devices/0000:03:00.0
> /sys/kernel/iommu_groups/2/devices/0000:04:00.0
> /sys/kernel/iommu_groups/3/devices/0000:00:08.0
> /sys/kernel/iommu_groups/4/devices/0000:00:10.0
> /sys/kernel/iommu_groups/5/devices/0000:00:11.0
> /sys/kernel/iommu_groups/6/devices/0000:00:12.0
> /sys/kernel/iommu_groups/7/devices/0000:00:13.0
> /sys/kernel/iommu_groups/8/devices/0000:00:14.0
> /sys/kernel/iommu_groups/8/devices/0000:00:14.2
> /sys/kernel/iommu_groups/8/devices/0000:00:14.3
> /sys/kernel/iommu_groups/8/devices/0000:00:14.7
> /sys/kernel/iommu_groups/9/devices/0000:00:18.0
> /sys/kernel/iommu_groups/9/devices/0000:00:18.1
> /sys/kernel/iommu_groups/9/devices/0000:00:18.2
> /sys/kernel/iommu_groups/9/devices/0000:00:18.3
> /sys/kernel/iommu_groups/9/devices/0000:00:18.4
> /sys/kernel/iommu_groups/9/devices/0000:00:18.5
>
> I'm running Ubuntu 14.04 with xenial kernel 4.4.0-13, libvirt 1.2.2,
> qemu 2.0.0.  Using vfio-pci simply failed:
>
> qemu-system-x86_64: -device
> vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x3: vfio: error, group 2
> is not viable, please ensure all devices within the iommu_group are bound
> to their vfio bus driver.
> qemu-system-x86_64: -device
> vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to get
> group 2
> qemu-system-x86_64: -device
> vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x3: Device initialization
> failed.
> qemu-system-x86_64: -device
> vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x3: Device 'vfio-pci'
> could not be initialized
>
> Using legacy KVM device assignment with <driver name='kvm'/> also
> failed with unclear reason "Invalid argument":
>
> qemu-system-x86_64: -device
> pci-assign,configfd=24,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x2: Failed
> to assign device "hostdev0" : Invalid argument
> qemu-system-x86_64: -device
> pci-assign,configfd=24,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x2: Device
> initialization failed.
> qemu-system-x86_64: -device
> pci-assign,configfd=24,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x2: Device
> 'kvm-pci-assign' could not be initialized
>
> After some googling I've learned that legacy KVM assignment also
> refuses to work for this configuration since kernel 4.2
> https://lkml.org/lkml/2015/11/12/661 .  With older kernel 3.19 this
> configuration worked as expected.
>
> Is there any chance for me to achieve single I211 passthrough with
> recent kernels?  Because the hardware has no flexibility on the bus
> topologies, the only possible way would be patching the kernel (ACS
> override or PCI quirks).  Is it safe?  Does anyone have any info on
> the IOMMU capability of device/bridge/chipset in question?
>

The grouping occurs the way it does because 00:02.* is a multifunction
device.  Without native hardware support for ACS on those functions, we
must assume that routing within the multifunction device can result in
non-IOMMU translated peer-to-peer DMA between the downstream endpoints.
You can use the ACS override patch, but the reason that patch is not
upstream is because it is potentially unsafe, you may be declaring
isolation where none really exists and the VMs could interact with each
other in unexpected ways.  The only real solution to enabling assignment of
these devices is to work with AMD to determine whether this internal
re-routing is possible and if not, add quirks to the kernel to expose the
devices as isolated.  Thanks,

Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160326/07143edd/attachment.htm>