[vfio-users] fitlet I211 PCI passthrough

Bob Dawes xochipilli4 at yahoo.com
Sat Mar 26 13:38:39 UTC 2016


There's rarely much harm to hardware in playing around with anything 
other than the fan control / bios / soldering, though obviously you can 
kill your OS installation (but that's just one reason why we love 
VM's!). I wouldn't expect much success as the four network devices are 
likely together because they share resource/control functions. As you 
say - you will have to trick the system into thinking they are isolated 
using the ACS isolation patch. Maybe somebody more experienced can give 
a clearer answer though as I've never used that patch but I'm deeply 
inclined to recommend a virtualised solution as it's robust .. scalable 
and far easier to maintain.

For example, just run the 4 network cards on the host and make virtual 
networks or bridge for the clients. Routing is a complex problem and has 
to be handled somehwere so it's better you know where plus with 4 
network cards I can imagine the the performance impact (latency or 
throughput) of using virtual nets/bridges on all computers/routers will 
round off to 0 when measured against the overhead introduced by the 
cards not having unique 8GT/s connections to their destinations.

I can think of cases where it would be worth the effort but I suggest 
you have a dig around in some of the network virtualisation documents 
(arch is quite a good place to start if I remember). It gets very scary 
very quickly even before you try to pass them through separately and if 
you've got 20,000 of these you'll need more knowledgeable help than me.

On 26/03/16 11:28, YAEGASHI Takeshi wrote:
> Hello,
>
> I have a fitlet http://www.fit-pc.com/web/products/fitlet/ with 4 GbE
> (I211) ports and wanted to passthrough some of them to KVM guests, ie
> assign one I211 to each guests.  I've tried various configurations
> with libvirt/kvm but no luck so far.
>
> After reading http://vfio.blogspot.jp/2014/08/iommu-groups-inside-and-out.html
> I've got sure that there's a problem in IOMMU groups.  Actually all of
> I211 and AMD's PCI bridge [1022:156b] share the same iommu_group 2.
>
> $ lspci -tv
> -[0000:00]-+-00.0  Advanced Micro Devices, Inc. [AMD] Device 1566
>             +-00.2  Advanced Micro Devices, Inc. [AMD] Device 1567
>             +-01.0  Advanced Micro Devices, Inc. [AMD/ATI] Mullins [Radeon R6 Graphics]
>             +-01.1  Advanced Micro Devices, Inc. [AMD/ATI] Kabini HDMI/DP Audio
>             +-02.0  Advanced Micro Devices, Inc. [AMD] Device 156b
>             +-02.2-[01]----00.0  Intel Corporation I211 Gigabit Network Connection
>             +-02.3-[02]----00.0  Intel Corporation I211 Gigabit Network Connection
>             +-02.4-[03]----00.0  Intel Corporation I211 Gigabit Network Connection
>             +-02.5-[04]----00.0  Intel Corporation I211 Gigabit Network Connection
>             +-08.0  Advanced Micro Devices, Inc. [AMD] Device 1537
>             +-10.0  Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller
>             +-11.0  Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
>             +-12.0  Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller
>             +-13.0  Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller
>             +-14.0  Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller
>             +-14.2  Advanced Micro Devices, Inc. [AMD] FCH Azalia Controller
>             +-14.3  Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge
>             +-14.7  Advanced Micro Devices, Inc. [AMD] FCH SD Flash Controller
>             +-18.0  Advanced Micro Devices, Inc. [AMD] Device 1580
>             +-18.1  Advanced Micro Devices, Inc. [AMD] Device 1581
>             +-18.2  Advanced Micro Devices, Inc. [AMD] Device 1582
>             +-18.3  Advanced Micro Devices, Inc. [AMD] Device 1583
>             +-18.4  Advanced Micro Devices, Inc. [AMD] Device 1584
>             \-18.5  Advanced Micro Devices, Inc. [AMD] Device 1585
>
> $ find /sys/kernel/iommu_groups -type l
> /sys/kernel/iommu_groups/0/devices/0000:00:00.0
> /sys/kernel/iommu_groups/1/devices/0000:00:01.0
> /sys/kernel/iommu_groups/1/devices/0000:00:01.1
> /sys/kernel/iommu_groups/2/devices/0000:00:02.0
> /sys/kernel/iommu_groups/2/devices/0000:00:02.2
> /sys/kernel/iommu_groups/2/devices/0000:00:02.3
> /sys/kernel/iommu_groups/2/devices/0000:00:02.4
> /sys/kernel/iommu_groups/2/devices/0000:00:02.5
> /sys/kernel/iommu_groups/2/devices/0000:01:00.0
> /sys/kernel/iommu_groups/2/devices/0000:02:00.0
> /sys/kernel/iommu_groups/2/devices/0000:03:00.0
> /sys/kernel/iommu_groups/2/devices/0000:04:00.0
> /sys/kernel/iommu_groups/3/devices/0000:00:08.0
> /sys/kernel/iommu_groups/4/devices/0000:00:10.0
> /sys/kernel/iommu_groups/5/devices/0000:00:11.0
> /sys/kernel/iommu_groups/6/devices/0000:00:12.0
> /sys/kernel/iommu_groups/7/devices/0000:00:13.0
> /sys/kernel/iommu_groups/8/devices/0000:00:14.0
> /sys/kernel/iommu_groups/8/devices/0000:00:14.2
> /sys/kernel/iommu_groups/8/devices/0000:00:14.3
> /sys/kernel/iommu_groups/8/devices/0000:00:14.7
> /sys/kernel/iommu_groups/9/devices/0000:00:18.0
> /sys/kernel/iommu_groups/9/devices/0000:00:18.1
> /sys/kernel/iommu_groups/9/devices/0000:00:18.2
> /sys/kernel/iommu_groups/9/devices/0000:00:18.3
> /sys/kernel/iommu_groups/9/devices/0000:00:18.4
> /sys/kernel/iommu_groups/9/devices/0000:00:18.5
>
> I'm running Ubuntu 14.04 with xenial kernel 4.4.0-13, libvirt 1.2.2,
> qemu 2.0.0.  Using vfio-pci simply failed:
>
> qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x3: vfio: error, group 2 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
> qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to get group 2
> qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x3: Device initialization failed.
> qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x3: Device 'vfio-pci' could not be initialized
>
> Using legacy KVM device assignment with <driver name='kvm'/> also
> failed with unclear reason "Invalid argument":
>
> qemu-system-x86_64: -device pci-assign,configfd=24,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x2: Failed to assign device "hostdev0" : Invalid argument
> qemu-system-x86_64: -device pci-assign,configfd=24,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x2: Device initialization failed.
> qemu-system-x86_64: -device pci-assign,configfd=24,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x2: Device 'kvm-pci-assign' could not be initialized
>
> After some googling I've learned that legacy KVM assignment also
> refuses to work for this configuration since kernel 4.2
> https://lkml.org/lkml/2015/11/12/661 .  With older kernel 3.19 this
> configuration worked as expected.
>
> Is there any chance for me to achieve single I211 passthrough with
> recent kernels?  Because the hardware has no flexibility on the bus
> topologies, the only possible way would be patching the kernel (ACS
> override or PCI quirks).  Is it safe?  Does anyone have any info on
> the IOMMU capability of device/bridge/chipset in question?
>
> Regards,




More information about the vfio-users mailing list