[vfio-users] Requesting USB Card Quad Bus advice

[Alex Williamson]

On Tue, Nov 7, 2017 at 8:42 AM, Alex Williamson <alex.l.williamson at gmail.com
> wrote:

> On Tue, Nov 7, 2017 at 8:32 AM, Steve Freitas <sflist at ihonk.com> wrote:
>
>> On 11/07/2017 06:51 AM, Alex Williamson wrote:
>>
>> On Mon, Nov 6, 2017 at 11:58 PM, Steve Freitas <sflist at ihonk.com> wrote:
>>
>>> On 11/06/2017 07:01 PM, Marcial Ascencio wrote:
>>>
>>>> Hi guys, i was planning to buy USB card with at least 2 usb ports to
>>>> assign each one to each VM, so I'm asking some advice about a brand that
>>>> works soft with Linux VFIO. Thank you so much in advance!
>>>>
>>>
>>> Hi Marcial,
>>>
>>> The Sonnet Allegro Pro worked great for me:
>>> https://amazon.com/Sonnet-Allegro-Pro-PCIe-card/dp/B00XPUHO10/
>>>
>>> I pass three of the four controllers through to different VMs, no ACS
>>> override necessary on my machine. VM reboots work without an issue. An
>>> earlier post on this mailing list indicated problems with this card with VM
>>> reboots, but that seems to have been resolved in more recent kernels. My
>>> host runs Ubuntu 17.04, kernel 4.10.
>>>
>>
>> Does the PCIe switch actually support ACS on the downstream ports for
>> this card?  I've seen some reports that certain switch configurations are
>> incorrectly allowing separate iommu groups where they shouldn't.  Fair
>> warning, we're probably going to need to fix that and break some
>> configurations that were incorrectly grouped.  Thanks,
>>
>>
>> Yer breakin' my heart! :-)
>>
>> Good question, how do I check that?
>>
>
> Send 'sudo lspci -vvv'
>

(lspci provided off-list)

The card looks good to me:

Your root port includes ACS support:

00:01.1 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root
Port 1b (rev 07) (prog-if 00 [Normal decode])
     Bus: primary=00, secondary=02, subordinate=07, sec-latency=0
     Capabilities: [110 v1] Access Control Services
         ACSCap:    SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+
EgressCtrl- DirectTrans-
         ACSCtl:    SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+
EgressCtrl- DirectTrans-

The upstream switch port on the card (ACS is not relevant here):

02:00.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI
Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode])
     Bus: primary=02, secondary=03, subordinate=07, sec-latency=0

The downstream switch ports, each supports ACS:

03:01.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI
Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode])
     Bus: primary=03, secondary=04, subordinate=04, sec-latency=0
     Capabilities: [520 v1] Access Control Services
         ACSCap:    SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+
EgressCtrl+ DirectTrans+
         ACSCtl:    SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+
EgressCtrl- DirectTrans-

03:05.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI
Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode])
     Bus: primary=03, secondary=05, subordinate=05, sec-latency=0
     Capabilities: [520 v1] Access Control Services
         ACSCap:    SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+
EgressCtrl+ DirectTrans+
         ACSCtl:    SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+
EgressCtrl- DirectTrans-

03:07.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI
Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode])
     Bus: primary=03, secondary=06, subordinate=06, sec-latency=0
     Capabilities: [520 v1] Access Control Services
         ACSCap:    SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+
EgressCtrl+ DirectTrans+
         ACSCtl:    SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+
EgressCtrl- DirectTrans-

03:09.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI
Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode])
     Bus: primary=03, secondary=07, subordinate=07, sec-latency=0
     Capabilities: [520 v1] Access Control Services
         ACSCap:    SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+
EgressCtrl+ DirectTrans+
         ACSCtl:    SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+
EgressCtrl- DirectTrans-

And then the four single function XHCI controllers.

04:00.0 USB controller: Fresco Logic FL1100 USB 3.0 Host Controller (rev
10) (prog-if 30 [XHCI])
     Subsystem: Sonnet Technologies, Inc. FL1100 USB 3.0 Host Controller

05:00.0 USB controller: Fresco Logic FL1100 USB 3.0 Host Controller (rev
10) (prog-if 30 [XHCI])
     Subsystem: Sonnet Technologies, Inc. FL1100 USB 3.0 Host Controller

06:00.0 USB controller: Fresco Logic FL1100 USB 3.0 Host Controller (rev
10) (prog-if 30 [XHCI])
     Subsystem: Sonnet Technologies, Inc. FL1100 USB 3.0 Host Controller

07:00.0 USB controller: Fresco Logic FL1100 USB 3.0 Host Controller (rev
10) (prog-if 30 [XHCI])
     Subsystem: Sonnet Technologies, Inc. FL1100 USB 3.0 Host Controller
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20171107/2fa3cf9b/attachment.htm>