<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Tue, Nov 7, 2017 at 8:42 AM, Alex Williamson <span dir="ltr"><<a href="mailto:alex.l.williamson@gmail.com" target="_blank">alex.l.williamson@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="gmail-">On Tue, Nov 7, 2017 at 8:32 AM, Steve Freitas <span dir="ltr"><<a href="mailto:sflist@ihonk.com" target="_blank">sflist@ihonk.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor="#FFFFFF"><span> <div class="gmail-m_7149431608629266604m_-3761334914076420816moz-cite-prefix">On 11/07/2017 06:51 AM, Alex Williamson wrote:<br> </div> <blockquote type="cite"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote">On Mon, Nov 6, 2017 at 11:58 PM, Steve Freitas <span dir="ltr"><<a href="mailto:sflist@ihonk.com" target="_blank">sflist@ihonk.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span>On 11/06/2017 07:01 PM, Marcial Ascencio wrote:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Hi guys, i was planning to buy USB card with at least 2 usb ports to assign each one to each VM, so I'm asking some advice about a brand that works soft with Linux VFIO. Thank you so much in advance!<br> </blockquote> <br> </span> Hi Marcial,<br> <br> The Sonnet Allegro Pro worked great for me: <a href="https://amazon.com/Sonnet-Allegro-Pro-PCIe-card/dp/B00XPUHO10/" rel="noreferrer" target="_blank">https://amazon.com/Sonnet-Alle<wbr>gro-Pro-PCIe-card/dp/B00XPUHO1<wbr>0/</a><br> <br> I pass three of the four controllers through to different VMs, no ACS override necessary on my machine. VM reboots work without an issue. An earlier post on this mailing list indicated problems with this card with VM reboots, but that seems to have been resolved in more recent kernels. My host runs Ubuntu 17.04, kernel 4.10.<br> </blockquote> <div><br> </div> <div>Does the PCIe switch actually support ACS on the downstream ports for this card? I've seen some reports that certain switch configurations are incorrectly allowing separate iommu groups where they shouldn't. Fair warning, we're probably going to need to fix that and break some configurations that were incorrectly grouped. Thanks,</div> <div><br> </div> </div> </div> </div> </blockquote> <br></span> Yer breakin' my heart! :-)<br> <br> Good question, how do I check that?</div></blockquote><div><br></div></span><div>Send 'sudo lspci -vvv'</div></div></div></div> </blockquote></div><br></div><div class="gmail_extra">(lspci provided off-list)</div><div class="gmail_extra"><br></div><div class="gmail_extra">The card looks good to me:</div><div class="gmail_extra"><br></div><div class="gmail_extra">Your root port includes ACS support:</div><div class="gmail_extra"><br></div><div class="gmail_extra"><div class="gmail_extra">00:01.1 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1b (rev 07) (prog-if 00 [Normal decode])</div><div class="gmail_extra"> Bus: primary=00, secondary=02, subordinate=07, sec-latency=0</div><div class="gmail_extra"> Capabilities: [110 v1] Access Control Services</div><div class="gmail_extra"> ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-</div><div class="gmail_extra"> ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-</div><div class="gmail_extra"><br></div><div class="gmail_extra">The upstream switch port on the card (ACS is not relevant here):</div><div class="gmail_extra"><br></div><div class="gmail_extra">02:00.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode])</div><div class="gmail_extra"> Bus: primary=02, secondary=03, subordinate=07, sec-latency=0</div><div class="gmail_extra"><br></div><div class="gmail_extra">The downstream switch ports, each supports ACS:</div><div class="gmail_extra"><br></div><div class="gmail_extra">03:01.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode])</div><div class="gmail_extra"> Bus: primary=03, secondary=04, subordinate=04, sec-latency=0</div><div class="gmail_extra"> Capabilities: [520 v1] Access Control Services</div><div class="gmail_extra"> ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+</div><div class="gmail_extra"> ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-</div><div class="gmail_extra"><br></div><div class="gmail_extra">03:05.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode])</div><div class="gmail_extra"> Bus: primary=03, secondary=05, subordinate=05, sec-latency=0</div><div class="gmail_extra"> Capabilities: [520 v1] Access Control Services</div><div class="gmail_extra"> ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+</div><div class="gmail_extra"> ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-</div><div class="gmail_extra"><br></div><div class="gmail_extra">03:07.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode])</div><div class="gmail_extra"> Bus: primary=03, secondary=06, subordinate=06, sec-latency=0</div><div class="gmail_extra"> Capabilities: [520 v1] Access Control Services</div><div class="gmail_extra"> ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+</div><div class="gmail_extra"> ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-</div><div class="gmail_extra"><br></div><div class="gmail_extra">03:09.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode])</div><div class="gmail_extra"> Bus: primary=03, secondary=07, subordinate=07, sec-latency=0</div><div class="gmail_extra"> Capabilities: [520 v1] Access Control Services</div><div class="gmail_extra"> ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+</div><div class="gmail_extra"> ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-</div><div class="gmail_extra"><br></div><div class="gmail_extra">And then the four single function XHCI controllers.</div><div class="gmail_extra"><br></div><div class="gmail_extra">04:00.0 USB controller: Fresco Logic FL1100 USB 3.0 Host Controller (rev 10) (prog-if 30 [XHCI])</div><div class="gmail_extra"> Subsystem: Sonnet Technologies, Inc. FL1100 USB 3.0 Host Controller</div><div class="gmail_extra"><br></div><div class="gmail_extra">05:00.0 USB controller: Fresco Logic FL1100 USB 3.0 Host Controller (rev 10) (prog-if 30 [XHCI])</div><div class="gmail_extra"> Subsystem: Sonnet Technologies, Inc. FL1100 USB 3.0 Host Controller</div><div class="gmail_extra"><br></div><div class="gmail_extra">06:00.0 USB controller: Fresco Logic FL1100 USB 3.0 Host Controller (rev 10) (prog-if 30 [XHCI])</div><div class="gmail_extra"> Subsystem: Sonnet Technologies, Inc. FL1100 USB 3.0 Host Controller</div><div class="gmail_extra"><br></div><div class="gmail_extra">07:00.0 USB controller: Fresco Logic FL1100 USB 3.0 Host Controller (rev 10) (prog-if 30 [XHCI])</div><div class="gmail_extra"> Subsystem: Sonnet Technologies, Inc. FL1100 USB 3.0 Host Controller</div></div></div>