[vfio-users] fitlet I211 PCI passthrough
Bob Dawes
xochipilli4 at yahoo.com
Sat Mar 26 13:38:39 UTC 2016
There's rarely much harm to hardware in playing around with anything
other than the fan control / bios / soldering, though obviously you can
kill your OS installation (but that's just one reason why we love
VM's!). I wouldn't expect much success as the four network devices are
likely together because they share resource/control functions. As you
say - you will have to trick the system into thinking they are isolated
using the ACS isolation patch. Maybe somebody more experienced can give
a clearer answer though as I've never used that patch but I'm deeply
inclined to recommend a virtualised solution as it's robust .. scalable
and far easier to maintain.
For example, just run the 4 network cards on the host and make virtual
networks or bridge for the clients. Routing is a complex problem and has
to be handled somehwere so it's better you know where plus with 4
network cards I can imagine the the performance impact (latency or
throughput) of using virtual nets/bridges on all computers/routers will
round off to 0 when measured against the overhead introduced by the
cards not having unique 8GT/s connections to their destinations.
I can think of cases where it would be worth the effort but I suggest
you have a dig around in some of the network virtualisation documents
(arch is quite a good place to start if I remember). It gets very scary
very quickly even before you try to pass them through separately and if
you've got 20,000 of these you'll need more knowledgeable help than me.
On 26/03/16 11:28, YAEGASHI Takeshi wrote:
> Hello,
>
> I have a fitlet http://www.fit-pc.com/web/products/fitlet/ with 4 GbE
> (I211) ports and wanted to passthrough some of them to KVM guests, ie
> assign one I211 to each guests. I've tried various configurations
> with libvirt/kvm but no luck so far.
>
> After reading http://vfio.blogspot.jp/2014/08/iommu-groups-inside-and-out.html
> I've got sure that there's a problem in IOMMU groups. Actually all of
> I211 and AMD's PCI bridge [1022:156b] share the same iommu_group 2.
>
> $ lspci -tv
> -[0000:00]-+-00.0 Advanced Micro Devices, Inc. [AMD] Device 1566
> +-00.2 Advanced Micro Devices, Inc. [AMD] Device 1567
> +-01.0 Advanced Micro Devices, Inc. [AMD/ATI] Mullins [Radeon R6 Graphics]
> +-01.1 Advanced Micro Devices, Inc. [AMD/ATI] Kabini HDMI/DP Audio
> +-02.0 Advanced Micro Devices, Inc. [AMD] Device 156b
> +-02.2-[01]----00.0 Intel Corporation I211 Gigabit Network Connection
> +-02.3-[02]----00.0 Intel Corporation I211 Gigabit Network Connection
> +-02.4-[03]----00.0 Intel Corporation I211 Gigabit Network Connection
> +-02.5-[04]----00.0 Intel Corporation I211 Gigabit Network Connection
> +-08.0 Advanced Micro Devices, Inc. [AMD] Device 1537
> +-10.0 Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller
> +-11.0 Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
> +-12.0 Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller
> +-13.0 Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller
> +-14.0 Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller
> +-14.2 Advanced Micro Devices, Inc. [AMD] FCH Azalia Controller
> +-14.3 Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge
> +-14.7 Advanced Micro Devices, Inc. [AMD] FCH SD Flash Controller
> +-18.0 Advanced Micro Devices, Inc. [AMD] Device 1580
> +-18.1 Advanced Micro Devices, Inc. [AMD] Device 1581
> +-18.2 Advanced Micro Devices, Inc. [AMD] Device 1582
> +-18.3 Advanced Micro Devices, Inc. [AMD] Device 1583
> +-18.4 Advanced Micro Devices, Inc. [AMD] Device 1584
> \-18.5 Advanced Micro Devices, Inc. [AMD] Device 1585
>
> $ find /sys/kernel/iommu_groups -type l
> /sys/kernel/iommu_groups/0/devices/0000:00:00.0
> /sys/kernel/iommu_groups/1/devices/0000:00:01.0
> /sys/kernel/iommu_groups/1/devices/0000:00:01.1
> /sys/kernel/iommu_groups/2/devices/0000:00:02.0
> /sys/kernel/iommu_groups/2/devices/0000:00:02.2
> /sys/kernel/iommu_groups/2/devices/0000:00:02.3
> /sys/kernel/iommu_groups/2/devices/0000:00:02.4
> /sys/kernel/iommu_groups/2/devices/0000:00:02.5
> /sys/kernel/iommu_groups/2/devices/0000:01:00.0
> /sys/kernel/iommu_groups/2/devices/0000:02:00.0
> /sys/kernel/iommu_groups/2/devices/0000:03:00.0
> /sys/kernel/iommu_groups/2/devices/0000:04:00.0
> /sys/kernel/iommu_groups/3/devices/0000:00:08.0
> /sys/kernel/iommu_groups/4/devices/0000:00:10.0
> /sys/kernel/iommu_groups/5/devices/0000:00:11.0
> /sys/kernel/iommu_groups/6/devices/0000:00:12.0
> /sys/kernel/iommu_groups/7/devices/0000:00:13.0
> /sys/kernel/iommu_groups/8/devices/0000:00:14.0
> /sys/kernel/iommu_groups/8/devices/0000:00:14.2
> /sys/kernel/iommu_groups/8/devices/0000:00:14.3
> /sys/kernel/iommu_groups/8/devices/0000:00:14.7
> /sys/kernel/iommu_groups/9/devices/0000:00:18.0
> /sys/kernel/iommu_groups/9/devices/0000:00:18.1
> /sys/kernel/iommu_groups/9/devices/0000:00:18.2
> /sys/kernel/iommu_groups/9/devices/0000:00:18.3
> /sys/kernel/iommu_groups/9/devices/0000:00:18.4
> /sys/kernel/iommu_groups/9/devices/0000:00:18.5
>
> I'm running Ubuntu 14.04 with xenial kernel 4.4.0-13, libvirt 1.2.2,
> qemu 2.0.0. Using vfio-pci simply failed:
>
> qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x3: vfio: error, group 2 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
> qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to get group 2
> qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x3: Device initialization failed.
> qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x3: Device 'vfio-pci' could not be initialized
>
> Using legacy KVM device assignment with <driver name='kvm'/> also
> failed with unclear reason "Invalid argument":
>
> qemu-system-x86_64: -device pci-assign,configfd=24,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x2: Failed to assign device "hostdev0" : Invalid argument
> qemu-system-x86_64: -device pci-assign,configfd=24,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x2: Device initialization failed.
> qemu-system-x86_64: -device pci-assign,configfd=24,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x2: Device 'kvm-pci-assign' could not be initialized
>
> After some googling I've learned that legacy KVM assignment also
> refuses to work for this configuration since kernel 4.2
> https://lkml.org/lkml/2015/11/12/661 . With older kernel 3.19 this
> configuration worked as expected.
>
> Is there any chance for me to achieve single I211 passthrough with
> recent kernels? Because the hardware has no flexibility on the bus
> topologies, the only possible way would be patching the kernel (ACS
> override or PCI quirks). Is it safe? Does anyone have any info on
> the IOMMU capability of device/bridge/chipset in question?
>
> Regards,
More information about the vfio-users
mailing list