[vfio-users] Bus reset trouble with Titan-X

Alex Williamson alex.williamson at redhat.com
Wed Oct 19 17:44:52 UTC 2016


On Wed, 19 Oct 2016 12:16:30 -0500
Kevin Vasko <kvasko at gmail.com> wrote:

> Ah, ok. My bad.
> 
> 
> Ran
> 
> #: setpci -s 3:00.0 82.w=8:8
> 
> SltSta: Status: AttnBtn- PowerFit- MRL- CmdCplt- PresDet+ Interlock-
>             Changed: MRL- PresDet- LinkState-
> 
> #: setpci -s 3:00.0 78.w=20:20
> 
> StlSta: Status AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
>            Changed: MRL- PresDet- LinkState-
> 
> 
> When I run lspci -vvs 3:00.0 it is currently in this state
> 
> StlSta: Status AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
>            Changed: MRL- PresDet- LinkState-
> 
> I didn't realize that I was needing to look at "PresDet", sorry. It does
> look like it is different than before so I assume the setpci commands
> changed it somewhere.
> 
> The device (GPU) is still in the "(rev ff) (prog-if ff)" state.

Ok, it would have been a long shot, the Presence Detect Changed bit
really should not have been having any effect on re-establishing the
link, it was just a notable difference between the working and
non-working examples.
 
> Do you think this could be a GPU issue? I have not tried a different GPU in
> the system. Would it be worthwhile trying an NVidia M4000 to see if I get
> the same results or do you think there is a problem with the PLX Riser?

I can only speculate here, but I wouldn't expect PCIe link
characteristics to be significantly different between consumer and
workstation class cards.  If you have one on hand, it certainly doesn't
hurt to try though.  Perhaps performing the same test with a non-NVIDIA
card installed might be more enlightening, preferably a card with
similar PCIe width and speed, but any sort of data point might be
useful.

I will note that NVIDIA does make use of PLX PCIe switches on some of
their devices, both the GRID K1 and Tesla M60 (probably others as well)
make use of a PLX PEX 8747 switch to pack multiple GPUs onto a single
card.  So there might be a reasonable expectation of PLX switches
working with NVIDIA devices.  What sort of tuning or special
configuration NVIDIA does on those since the switch is onboard the
card, I have no idea.  Thanks,

Alex




More information about the vfio-users mailing list