<html><head></head><body>Alex,<br>
<br>
I am running the Arch VFIO kernel from aur and the ACS patch is working for my Skylake system. Will I not need the ACS patch with 4.7 or will I get a performance increase? <br><br><div class="gmail_quote">On May 9, 2016 9:25:09 AM PDT, Alex Williamson <alex.l.williamson@gmail.com> wrote:<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Mon, May 9, 2016 at 10:06 AM, Jonas Camillus Jeppesen <span dir="ltr"><<a href="mailto:jonascj@sdu.dk" target="_blank">jonascj@sdu.dk</a>></span> wrote:<br /><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>My IOMMU grouping of devices change depending on which pci-e
socket I insert my R9 290 GPU into. For the sake of purchasing a
new system I wanted to discuss the different groupings so I can
better choose my new hardware.<br />
</p>
<p>Here are the IOMMU group which contain my GPU with the GPU
inserted into the three different PCIE slots I have (for all
groups in the different configs see attachments).<br />
</p>
<p>#A: GPU in PCIE1<br />
IOMMU group 1<br />
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200
v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01]
(rev 06)<br />
01:00.0 VGA compatible controller [0300]: Advanced Micro
Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290] [1002:67b1]<br />
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc.
[AMD/ATI] Hawaii HDMI Audio [1002:aac8]<br />
<br />
#B: GPU in PCIE3<br />
IOMMU group 1<br />
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200
v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01]
(rev 06)<br />
00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200
v3/4th Gen Core Processor PCI Express x8 Controller [8086:0c05]
(rev 06)<br />
02:00.0 VGA compatible controller [0300]: Advanced Micro
Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290] [1002:67b1]<br />
02:00.1 Audio device [0403]: Advanced Micro Devices, Inc.
[AMD/ATI] Hawaii HDMI Audio [1002:aac8]<br />
<br />
#C: GPU in PCIE4<br />
IOMMU group 14<br />
04:00.0 VGA compatible controller [0300]: Advanced Micro
Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290] [1002:67b1]<br />
04:00.1 Audio device [0403]: Advanced Micro Devices, Inc.
[AMD/ATI] Hawaii HDMI Audio [1002:aac8]</p>
<p>I have had success with Arch linux and stock kernel 4.4.1 with
config #C (GPU in PCIE4), but not the others. Is that to be
expected without patching the kernel (i.e. the GPU needs to be in
a group all by itself)?<br />
</p>
<p>Can these groups be figured out without plugging the GPU into the
different slots and looking at
/sys/kernel/iommu_groups/; deduced from the chip set
specification, inspecting /sys/ in more clever way, or similar? If
yes, then it would be great to collect a list of hardware and its
groupings. That way it would be easier to decide which motherboard
and cpu to get for different setups.</p></div></blockquote><div><br /></div><div>It's very simple, unless you choose a Xeon E5+ or a High End Desktop Processor, then all of the processor based root ports will be grouped together. The difference between #A and #B in your example is simply that the BIOS disables the 00:01.1 root port if there's nothing installed in the slot (it can't disable 00:01.0 in configuration #B because function #0 must be present in order to discover function #1). In example #C you've installed the card in a PCH-based root port for which we have quirks in the kernel to enable isolation. We have quirks for most Intel chipsets to enable this:</div><div><br /></div><div><a href="http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/quirks.c#n3877">http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/quirks.c#n3877</a><br /></div><div><br /></div><div>Linux v4.7 should have qu!
irks for
Skylake PCH root ports:</div><div><br /></div><div><a href="https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/drivers/pci/quirks.c?id=1bf2bf229b64540f91ac6fa3af37c81249037a0b">https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/drivers/pci/quirks.c?id=1bf2bf229b64540f91ac6fa3af37c81249037a0b</a><br /></div><div><br /></div><div>Often the more high-end motherboard manufacturers will have block diagrams of the system layout so you can see which slots are hosted by which device. IIRC supermicro typically does this, not that I'm endorsing their products.</div><div><br /></div><div>All of the above configurations will work, #A and #B simply add restrictions that you can't use the additional CPU-based slots for anything else without (not recommended) using the ACS override patch. For an assigned GPU, it does not matter that the root port itself is included in the group. If however you were trying to make use of a multi-port N!
IC, with
each port assigned to a separate VM, or an SR-IOV device with multiple virtual functions, installing it in such a slot would be a very bad choice for those applications.</div></div></div></div>
<p style="margin-top: 2.5em; margin-bottom: 1em; border-bottom: 1px solid #000"></p><pre class="k9mail"><hr /><br />vfio-users mailing list<br />vfio-users@redhat.com<br /><a href="https://www.redhat.com/mailman/listinfo/vfio-users">https://www.redhat.com/mailman/listinfo/vfio-users</a><br /></pre></blockquote></div></body></html>