[vfio-users] Mobo reccommendations/suggestions ?

[Bob Dawes]

On 18/06/16 16:42, Alex Williamson wrote:
> On Sat, Jun 18, 2016 at 9:11 AM, Samuel Holland <samuel at sholland.org 
> <mailto:samuel at sholland.org>> wrote:
>
>     On 06/17/2016 02:00 PM, Kajim Shanti wrote:
>
>         If i decide to replace the mobo, which one would you reccommend as
>         giving  the least problems for devices passthrough?
>
>
>     ASRock generally has a good reputation for not breaking VT-d. Be aware
>     that any motherboard you buy with multiple PCIe x16 ports will
>     have them
>     all in the same IOMMU group.
>
>
> This is why the continuous recommendations for High End Desktop 
> Processors and Xeon E5, where this is not the case.  Just be aware 
> that standard Core-i3/5/7 are going to group all the processor root 
> ports together, which either limits your configurations or requires 
> the acs override patch.  The latter, I don't consider supportable.  
> There are plenty of usable configurations available for Core 
> processors, but for more than one discrete GPU, you're going to need 
> to use PCH-based root ports.
>
>
> _______________________________________________
> vfio-users mailing list
> vfio-users at redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users

I'd actually vote for cheaper variants of skylake hardware for consumer 
use as you aren't likely to get the x16 port grouping and there won't be 
lots of extra controllers with hardwired groups. The grouping problem 
for the x16 ports goes away because these cheaper boards don't offer 
sharing of the 16 CPU lanes but instead sub-divide the lanes between 
separate bridges as you add cards (i.e. 1 x16 card or 2 x8, etc.). The 
intel spec. says the host port will offer full isolation if there is no 
inter-device data sharing that doesn't go through the host port which 
your root bridges can either guarantee for you .. or in the cheaper case 
which you get for free by not letting them share lanes and grouping 
exceptions manually. For normal hardware this usually means you didn't 
plug the same device into multiple ports - or connect devices with a 
cable. ACS does offer optional services to cope even with those cases 
but simpler boards will tend to have isolation by that rule and Mine 
even has ACS registers that explicitly say it's isolated and give some 
optional services I've no idea about.

The exception is if you are running multiple Gen2 legacy devices that 
need x16 to hit their maximum throughput or naturally for SLI which is 
the only consumer use case I can think of that can hope to even saturate 
a single x8 Gen 3 port. SLI devices are clearly in the same IOMMU group 
so unless that's your intent don't buy the Z77 type boards.

I have little trouble from running two Vms with a different radeon card 
passed to each and the IGD on the host using no ACS isolation patches 
and qemu 2.6 / 4.7 kernel as well as a couple of minimal server VMs that 
run 24/7 but are low load. The board is an ASUS Z170 Pro (I forget which 
variant) with an i5 6600k with the IGD running on the host but no X. 
This is all firmly in the upper lower end of consumer grade territory 
and is only consistent with minimal VM use but it's great if you don't 
need scalability or high availability under all workloads. I don't see 
many crashes outside the Win10 VM and those never take other VMs with 
them. The IGD crashes the host a lot if you use it for accelerated X - 
which is why I don't. I'd turn it off if I could use a less cutting edge 
kernel.

For reference .. these are my IOMMU groupings and the relevant devices. 
If you want more specific information I can provide working examples 
from my hardware. I suspect it may vary a lot between the cheaper boards 
as only the first PEG port is in any way standardised.

IOMMU / Device
0    00:00.0 Host bridge [0600]: Intel Corporation Skylake Host 
Bridge/DRAM Registers [8086:191f] (rev 07)
1    00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe 
Controller (x16) [8086:1901] (rev 07)
2    00:02.0 VGA compatible controller [0300]: Intel Corporation HD 
Graphics 530 [8086:1912] (rev 06)
3    00:14.0 USB controller [0c03]: Intel Corporation Sunrise Point-H 
USB 3.0 xHCI Controller [8086:a12f] (rev 31)
4    00:16.0 Communication controller [0780]: Intel Corporation Sunrise 
Point-H CSME HECI #1 [8086:a13a] (rev 31)
5    00:17.0 SATA controller [0106]: Intel Corporation Sunrise Point-H 
SATA controller [AHCI mode] [8086:a102] (rev 31)
6    00:1b.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI 
Root Port #17 [8086:a167] (rev f1)
7    00:1c.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI 
Express Root Port #1 [8086:a110] (rev f1)
8    00:1c.4 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI 
Express Root Port #5 [8086:a114] (rev f1)
9    00:1d.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI 
Express Root Port #9 [8086:a118] (rev f1)
10  00:1f.0 ISA bridge [0601]: Intel Corporation Sunrise Point-H LPC 
Controller [8086:a145] (rev 31)
10  00:1f.2 Memory controller [0580]: Intel Corporation Sunrise Point-H 
PMC [8086:a121] (rev 31)
10  00:1f.3 Audio device [0403]: Intel Corporation Sunrise Point-H HD 
Audio [8086:a170] (rev 31)
10  00:1f.4 SMBus [0c05]: Intel Corporation Sunrise Point-H SMBus 
[8086:a123] (rev 31)
10  00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet 
Connection (2) I219-V [8086:15b8] (rev 31)
1    01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, 
Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X] [1002:6798]
1    01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] 
Tahiti XT HDMI Audio [Radeon HD 7970 Series] [1002:aaa0]
12  03:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 
3.1 Host Controller [1b21:1242]
13  04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, 
Inc. [AMD/ATI] Cayman XT [Radeon HD 6970] [1002:6718]
13  04:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] 
Cayman/Antilles HDMI Audio [Radeon HD 6900 Series] [1002:aa80]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160620/c862b18f/attachment.htm>