[vfio-users] IOMMU Group with PCI Bridge and 10gbe nic

[Alex Williamson]

On Thu, 14 Sep 2017 19:39:40 +0000 (GMT)
Allen Dial <adial at mac.com> wrote:

> Hi VFIO-Users,
> 
> I have 2 of 3 2 port NICs that I would like to passthrough to guest vms but am not sure how best to go about it as 1, they get divided up (not the biggest issue other than I don't know how I will tell which one is which physically out the back of the machine) but more importantly they each get grouped with pci bridges in their iommu groups.
> 
> Is it possible to pass 4 of the interfaces to vms and keep 2 of them for the host?
> 
> Below is some hopefully helpful information from the terminal:
> 
> group 39
> /sys/kernel/iommu_groups/39/devices/0000:42:02.0
> 42:02.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES24T3G2 PCI Express Gen2 Switch (rev 02)
> /sys/kernel/iommu_groups/39/devices/0000:43:00.0
> 43:00.0 Ethernet controller: MYRICOM Inc. Myri-10G Dual-Protocol NIC (rev 01)
> 
> group 40
> /sys/kernel/iommu_groups/40/devices/0000:42:04.0
> 42:04.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES24T3G2 PCI Express Gen2 Switch (rev 02)
> /sys/kernel/iommu_groups/40/devices/0000:44:00.0
> 44:00.0 Ethernet controller: MYRICOM Inc. Myri-10G Dual-Protocol NIC (rev 01)

Looks like the card likely includes a PCIe switch where the downstream
ports do not support ACS, so they get grouped with the endpoint below
it.  The real question is probably whether the downstream ports 42:02.0
and 42:04.0 should have been grouped together, putting both downstream
ports and both endpoints in the same group.  That probably didn't
happen because the downstream ports have different "slot" addresses.

> group 42
> /sys/kernel/iommu_groups/42/devices/0000:47:02.0
> 47:02.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES24T3G2 PCI Express Gen2 Switch (rev 02)
> /sys/kernel/iommu_groups/42/devices/0000:48:00.0
> 48:00.0 Ethernet controller: MYRICOM Inc. Myri-10G Dual-Protocol NIC (rev 01)
> 
> group 43
> /sys/kernel/iommu_groups/43/devices/0000:47:04.0
> 47:04.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES24T3G2 PCI Express Gen2 Switch (rev 02)
> /sys/kernel/iommu_groups/43/devices/0000:49:00.0
> 49:00.0 Ethernet controller: MYRICOM Inc. Myri-10G Dual-Protocol NIC (rev 01)

Set #2

> group 45
> /sys/kernel/iommu_groups/45/devices/0000:4b:02.0
> 4b:02.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES24T3G2 PCI Express Gen2 Switch (rev 02)
> /sys/kernel/iommu_groups/45/devices/0000:4c:00.0
> 4c:00.0 Ethernet controller: MYRICOM Inc. Myri-10G Dual-Protocol NIC (rev 01)
> 
> group 46
> /sys/kernel/iommu_groups/46/devices/0000:4b:04.0
> 4b:04.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES24T3G2 PCI Express Gen2 Switch (rev 02)
> /sys/kernel/iommu_groups/46/devices/0000:4d:00.0
> 4d:00.0 Ethernet controller: MYRICOM Inc. Myri-10G Dual-Protocol NIC (rev 01)

Set #3

As it is, there's no problem with assigning each NIC to a separate
guest or using them in the host.  It is arguable whether there should
really only be 3 groups here as peer-to-peer DMA between downstream
switch ports seems possible in this configuration.  If we fix that then
you'd need to assign NICs in sets of two.  As it is, each NIC is
assignable, for better or worse.  Thanks,

Alex