[vfio-users] Supermicro C7Z170-OCE, a rather interesing Skylake Motherboard for VFIO if PEX8747 downstream ACS works

[Zir Blazer]

For your information, here is a Motherboard that has a rather unique feature. The Supermicro C7Z170-OCE supports Skylake and has all the 16 PCIe Lanes coming from the Processor PCIe Controller going through the included PLX PEX8747 chip, as you can see in the Manual at Page 23 / 1 - 9 :
http://www.supermicro.com/products/motherboard/Core/Z170/C7Z170-OCE.cfm
http://www.ocaholic.ch/modules/news/article.php?storyid=13898http://www.supermicro.com/manuals/motherboard/Z170/MNL-1828.pdf

The PLX PEX8747 is a PCIe Lanes multiplexing chip. On Intel standard consumer platforms, with the Processor 16 PCIe Lanes you can usually do either a single PCIe 16x slot or a 8x/8x arrangement, which allows for using at maximum two nVidia Video Cards in SLI, since they require you to have at least 8 lanes per Video Card to use them that way (AMD Crossfire has no such restriction, so you could do 3-way in 8x/4x/4x mode with them). At some point of the Ivy Bridge generation, ultra high end consumer Motherboards started to include this chip because they could workaround that nVidia requeriments and thus include support for 3-way or 4-way SLI by providing for up to four 8x PCIe slots:
http://www.anandtech.com/show/6170/four-multigpu-z77-boards-from-280350-plx-pex-8747-featuring-gigabyte-asrock-ecs-and-evga
There are obviously drawbacks about this. First, you're still limited to 16 lanes between the PEX8747 and the Processor, so if you were actually using all possible bandwidth of the 16 lanes, you would have a bottleneck, so it doesn't do any magic there, it just dynamically switches bandwidth around between its attached devices according to needs. The second is that by default it adds latency, so you could actually be losing a bit of performance on a Motherboard with it if you don't use its capabilities, like for single Video Card purposes. Finally, the PLX chip is expensive and significantly increases a Motherboard cost. Actually, if I recall correctly, after the Ivy Bridge generation, these type of PLX chips dissapeared from consumer space because after Avago purchased PLX, their old multiplexing chips became a sort of premium product for Servers and increased its cost even further, causing them to be fully phased out of even ultra high end consumer products.
So basically, it initially looks like including a PLX chip is a waste of time since for anyone that wanted such a ultra high end consumer LGA 1155/1150/1151 Motherboard for 4-way SLI, he should have questioned himself instead why he is not aiming for LGA 2011/2011-3, since chances are that its around the same budget range anyways, and its better if you actually need the PCIe Lanes. Is that correct? Absolutely yes... BUT, it may have an use for VFIO.The PLX multiplexing chips supports PCIe ACS (Access Control Services). We know that Intel consumer platforms don't support ACS on the Processor PCIe Controller, but the question would be: Does ACS isolation applies downsteam of the PLX chip? Since it may be interesing to check that even if you can't properly isolate PCI devices on the standard 8x/4x/4x arrangement, if with everything behind the PLX chip, it may be possible to isolate them. In which case, you could do passthrough with VFIO without needing hacks on anything on such platform, as each Video Card would get its own IOMMU Group by default.
Indeed, if the PLX chip allows us to have a sort-of ACS on LGA 1151, it may be a rather interesing piece of Hardware. Problem is, the C7Z170-OCE is a premium version of the already 220 U$D C7Z170-SQ, here:http://www.tweaktown.com/reviews/7394/supermicro-c7z170-sq-intel-z170-motherboard-review/index.htmlChances are that its price will be around the 300 U$D mark, in which case, you're again better served by LGA 2011-3. But if you WANT Skylake and WANT VFIO passthrough without having to apply hacks for the no-isolation on Processor PCIe Controller issue, it may be a rather interesing Motherboard if it works that way. The only other Skylake Motherboard I'm aware that has a PEX8747 is the Gigabyte Z170X-Gaming G1, which cost an absurd 470 U$Ds.
So basically, I would like someone to confirm if PEX8747 ACS support works downstream, even on Intel consumer platforms. 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20151225/fb24d75a/attachment.htm>