<div dir="ltr">Sure thing. I'm attaching all of the logs I have to let you get a bigger picture (and anyone that might run into a similar issue). Hopefully I didn't mess anything up.<div><br></div><div>Unfortunately, I've seen almost every single device fail at one point or another. I was thinking it might be isolated to a single PLX Riser card but I have now seen devices fail on every single parent device at one time or another. Based on that, I don't think I could narrow it down to a single PCISlot/PLX Riser that is the culprit. Unless both of these boards are bad, my conclusion is that this indicates a problem with the hardware as well. I completely agree that if the PCI Bus reset isn't working properly, nothing is going to work.</div><div><br></div><div>I sent these steps to the manufacturer to see if they could reproduce the issue on their end. If they can then they will need to investigate on their end why the problem exists. If they can't, it is possible we have a bad set of boards in this machine. </div><div><br><div>Thank you so much for your help. Really appreciate it.</div><div><br></div><div>-Kevin</div><div><br></div><div><div>#: lspci -tv</div><div><br></div><div><div> | +-1f.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU</div><div> | \-1f.2 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU</div><div> \-[0000:00]-+-00.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMI2</div><div> +-01.0-[01]--</div><div> +-02.0-[02-08]----00.0-[03-08]--+-00.0-[04]--+-00.0 NVIDIA Corporation GM200 [GeForce GTX TITAN X]</div><div> | | \-00.1 NVIDIA Corporation Device 0fb0</div><div> | +-04.0-[05]--+-00.0 NVIDIA Corporation GM200 [GeForce GTX TITAN X]</div><div> | | \-00.1 NVIDIA Corporation Device 0fb0</div><div> | +-08.0-[06]--+-00.0 NVIDIA Corporation GM200 [GeForce GTX TITAN X]</div><div> | | \-00.1 NVIDIA Corporation Device 0fb0</div><div> | +-0c.0-[07]--+-00.0 NVIDIA Corporation GM200 [GeForce GTX TITAN X]</div><div> | | \-00.1 NVIDIA Corporation Device 0fb0</div><div> | \-14.0-[08]----00.0 Mellanox Technologies MT27500 Family [ConnectX-3]</div><div> +-03.0-[09-12]----00.0-[0a-12]--+-08.0-[0b-11]----00.0-[0c-11]--+-00.0-[0d]--+-00.0 NVIDIA Corporation GM200 [GeForce GTX TITAN X]</div><div> | | | \-00.1 NVIDIA Corporation Device 0fb0</div><div> | | +-04.0-[0e]--+-00.0 NVIDIA Corporation GM200 [GeForce GTX TITAN X]</div><div> | | | \-00.1 NVIDIA Corporation Device 0fb0</div><div> | | +-08.0-[0f]--+-00.0 NVIDIA Corporation GM200 [GeForce GTX TITAN X]</div><div> | | | \-00.1 NVIDIA Corporation Device 0fb0</div><div> | | +-0c.0-[10]--+-00.0 NVIDIA Corporation GM200 [GeForce GTX TITAN X]</div><div> | | | \-00.1 NVIDIA Corporation Device 0fb0</div><div> | | \-14.0-[11]--+-00.0 Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet Multi Function</div><div> | | +-00.1 Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet Multi Function</div><div> | | +-00.2 Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet Multi Function</div><div> | | +-00.3 Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet Multi Function</div><div> | | +-00.4 Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet Multi Function</div><div> | | +-00.5 Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet Multi Function</div><div> | | +-00.6 Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet Multi Function</div><div> | | \-00.7 Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet Multi Function</div><div> | \-10.0-[12]--</div><div> +-05.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management</div><div> +-05.1 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Hot Plug</div><div> +-05.2 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 RAS, Control Status and Global Errors</div></div><div><br></div><div><br></div><div># showing which ones are in failed state</div><div>:# lspci -vnnn | grep NVIDIA </div><div><br></div><div><div>04:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX TITAN X] [10de:17c2] (rev ff) (prog-if ff)</div><div>04:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fb0] (rev ff) (prog-if ff)</div><div>05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX TITAN X] [10de:17c2] (rev ff) (prog-if ff)</div><div>05:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fb0] (rev ff) (prog-if ff)</div><div>06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX TITAN X] [10de:17c2] (rev a1) (prog-if 00 [VGA controller])</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Subsystem: NVIDIA Corporation Device [10de:1132]</div><div>06:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fb0] (rev a1)</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Subsystem: NVIDIA Corporation Device [10de:1132]</div><div>07:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX TITAN X] [10de:17c2] (rev a1) (prog-if 00 [VGA controller])</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Subsystem: NVIDIA Corporation Device [10de:1132]</div><div>07:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fb0] (rev a1)</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Subsystem: NVIDIA Corporation Device [10de:1132]</div><div>0d:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX TITAN X] [10de:17c2] (rev a1) (prog-if 00 [VGA controller])</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Subsystem: NVIDIA Corporation Device [10de:1132]</div><div>0d:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fb0] (rev a1)</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Subsystem: NVIDIA Corporation Device [10de:1132]</div><div>0e:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX TITAN X] [10de:17c2] (rev a1) (prog-if 00 [VGA controller])</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Subsystem: NVIDIA Corporation Device [10de:1132]</div><div>0e:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fb0] (rev a1)</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Subsystem: NVIDIA Corporation Device [10de:1132]</div><div>0f:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX TITAN X] [10de:17c2] (rev a1) (prog-if 00 [VGA controller])</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Subsystem: NVIDIA Corporation Device [10de:1132]</div><div>0f:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fb0] (rev a1)</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Subsystem: NVIDIA Corporation Device [10de:1132]</div><div>10:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX TITAN X] [10de:17c2] (rev a1) (prog-if 00 [VGA controller])</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Subsystem: NVIDIA Corporation Device [10de:1132]</div><div>10:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fb0] (rev a1)</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Subsystem: NVIDIA Corporation Device [10de:1132]</div></div><div><br></div><div><br></div><div>#showing parent bridge of a device that has a failed</div><div>#:lspci -vvvs 03:00</div><div><div>03:00.0 PCI bridge: PLX Technology, Inc. Device 8796 (rev ab) (prog-if 00 [Normal decode])</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Latency: 0, Cache Line Size: 32 bytes</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Bus: primary=03, secondary=04, subordinate=04, sec-latency=0</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>I/O behind bridge: 00009000-00009fff</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Memory behind bridge: c5000000-c60fffff</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Prefetchable memory behind bridge: 000038ffe0000000-000038fff1ffffff</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [40] Power Management version 3</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Address: 00000000fee003b8 Data: 0000</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Masking: 000000ff Pending: 00000000</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DevCap:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>MaxPayload 2048 bytes, PhantFunc 0</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ExtTag- RBE+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DevCtl:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>MaxPayload 256 bytes, MaxReadReq 128 bytes</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DevSta:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>LnkCap:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Port #0, Speed 8GT/s, Width x16, ASPM not supported, Exit Latency L0s <4us, L1 <8us</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ClockPM- Surprise- LLActRep- BwNot-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>LnkCtl:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ASPM Disabled; Disabled- CommClk-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>LnkSta:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>SltCap:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Slot #32, PowerLimit 75.000W; Interlock- NoCompl-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>SltCtl:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>SltSta:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Changed: MRL- PresDet+ LinkState-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message ARIFwd+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> Compliance De-emphasis: -6dB</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 3577</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [100 v1] Device Serial Number ab-87-00-10-b5-df-0e-00</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [fb4 v1] Advanced Error Reporting</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>UESta:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>UEMsk:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>UESvrt:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>CESta:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>CEMsk:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>AERCap:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [138 v1] Power Budgeting <?></div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [10c v1] #19</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [148 v1] Virtual Channel</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Caps:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>LPEVC=0 RefClk=100ns PATEntryBits=8</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Arb:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Fixed- WRR32- WRR64- WRR128-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Ctrl:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ArbSelect=Fixed</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Status:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>InProgress-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>VC0:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Caps:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>PATOffset=03 MaxTimeSlots=1 RejSnoopTrans-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Arb:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Ctrl:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Enable+ ID=0 ArbSelect=WRR64 TC/VC=01</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Status:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>NegoPending+ InProgress-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Port Arbitration Table <?></div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [e00 v1] #12</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [f24 v1] Access Control Services</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ACSCap:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ACSCtl:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?></div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Kernel driver in use: pcieport</div><div><br></div></div><div>#showing secondary device of 03:00 (parent) which is in failed state</div><div>#: lspci -vvvs 04:00.0</div><div><div>04:00.0 VGA compatible controller: NVIDIA Corporation GM200 [GeForce GTX TITAN X] (rev ff) (prog-if ff)</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>!!! Unknown header type 7f</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Kernel driver in use: pci-stub</div><div><br></div></div><div>#showing secondary device of 03:00 (parent) of .1 device (audio adapter) that is in failed state</div><div>#: lspci -vvvs 04:00.1</div><div><div>04:00.1 Audio device: NVIDIA Corporation Device 0fb0 (rev ff) (prog-if ff)</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>!!! Unknown header type 7f</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Kernel driver in use: pci-stub</div><div><br></div></div><div><br></div><div>#showing parent device that has a NON failed device<br></div><div>#: lspci -vvvs 03:08</div><div><div>03:08.0 PCI bridge: PLX Technology, Inc. Device 8796 (rev ab) (prog-if 00 [Normal decode])</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Latency: 0, Cache Line Size: 32 bytes</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Bus: primary=03, secondary=06, subordinate=06, sec-latency=0</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>I/O behind bridge: 00007000-00007fff</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Memory behind bridge: c1000000-c20fffff</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Prefetchable memory behind bridge: 000038ffa0000000-000038ffb1ffffff</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [40] Power Management version 3</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Address: 00000000fee003f8 Data: 0000</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Masking: 000000ff Pending: 00000000</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DevCap:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>MaxPayload 2048 bytes, PhantFunc 0</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ExtTag- RBE+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DevCtl:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>MaxPayload 256 bytes, MaxReadReq 128 bytes</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DevSta:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>LnkCap:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Port #8, Speed 8GT/s, Width x16, ASPM not supported, Exit Latency L0s <4us, L1 <8us</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ClockPM- Surprise- LLActRep- BwNot-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>LnkCtl:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ASPM Disabled; Disabled- CommClk-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>LnkSta:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Speed 8GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>SltCap:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Slot #32, PowerLimit 75.000W; Interlock- NoCompl-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>SltCtl:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>SltSta:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Changed: MRL- PresDet- LinkState-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message ARIFwd+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> Compliance De-emphasis: -6dB</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 3577</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [100 v1] Device Serial Number ab-87-00-10-b5-df-0e-00</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [fb4 v1] Advanced Error Reporting</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>UESta:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>UEMsk:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>UESvrt:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>CESta:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>CEMsk:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>AERCap:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [138 v1] Power Budgeting <?></div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [10c v1] #19</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [148 v1] Virtual Channel</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Caps:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>LPEVC=0 RefClk=100ns PATEntryBits=8</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Arb:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Fixed- WRR32- WRR64- WRR128-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Ctrl:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ArbSelect=Fixed</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Status:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>InProgress-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>VC0:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Caps:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>PATOffset=03 MaxTimeSlots=1 RejSnoopTrans-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Arb:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Ctrl:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Enable+ ID=0 ArbSelect=WRR64 TC/VC=01</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Status:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>NegoPending- InProgress-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Port Arbitration Table <?></div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [e00 v1] #12</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [f24 v1] Access Control Services</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ACSCap:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ACSCtl:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?></div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Kernel driver in use: pcieport</div><div><br></div></div><div><div>#showing secondary device of 03:08 which is NON failed state</div><div>#: lspci -vvvs 06:00.0</div><div><div><div>06:00.0 VGA compatible controller: NVIDIA Corporation GM200 [GeForce GTX TITAN X] (rev a1) (prog-if 00 [VGA controller])</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Subsystem: NVIDIA Corporation Device 1132</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Interrupt: pin A routed to IRQ 5</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Region 0: Memory at c1000000 (32-bit, non-prefetchable) [disabled] [size=16M]</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Region 1: Memory at 38ffa0000000 (64-bit, prefetchable) [disabled] [size=256M]</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Region 3: Memory at 38ffb0000000 (64-bit, prefetchable) [disabled] [size=32M]</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Region 5: I/O ports at 7000 [disabled] [size=128]</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Expansion ROM at c2000000 [disabled] [size=512K]</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [60] Power Management version 3</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Address: 0000000000000000 Data: 0000</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [78] Express (v2) Legacy Endpoint, MSI 00</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DevCap:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DevCtl:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Report errors: Correctable- Non-Fatal- Fatal- Unsupported-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>MaxPayload 256 bytes, MaxReadReq 512 bytes</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DevSta:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>LnkCap:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Port #8, Speed 8GT/s, Width x16, ASPM not supported, Exit Latency L0s <1us, L1 <4us</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ClockPM+ Surprise- LLActRep- BwNot-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>LnkCtl:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ASPM Disabled; RCB 64 bytes Disabled- CommClk-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>LnkSta:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR+, OBFF Via message</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> Compliance De-emphasis: -6dB</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [100 v1] Virtual Channel</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Caps:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>LPEVC=0 RefClk=100ns PATEntryBits=1</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Arb:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Fixed- WRR32- WRR64- WRR128-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Ctrl:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ArbSelect=Fixed</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Status:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>InProgress-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>VC0:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Caps:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Arb:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Ctrl:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Enable+ ID=0 ArbSelect=Fixed TC/VC=01</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Status:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>NegoPending- InProgress-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [250 v1] Latency Tolerance Reporting</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Max snoop latency: 0ns</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Max no snoop latency: 0ns</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [258 v1] L1 PM Substates</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> PortCommonModeRestoreTime=255us PortTPowerOnTime=10us</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [128 v1] Power Budgeting <?></div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [420 v2] Advanced Error Reporting</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>UESta:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>UEMsk:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>UESvrt:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>CESta:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>CEMsk:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>AERCap:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?></div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [900 v1] #19</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Kernel driver in use: pci-stub</div><div><br></div></div><div><br></div></div><div>#showing secondary device of 03:08 of .1 device (audio adapter) that is in NON failed state</div><div>#: lspci -vvvs 06:00.1</div><div><div>06:00.1 Audio device: NVIDIA Corporation Device 0fb0 (rev a1)</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Subsystem: NVIDIA Corporation Device 1132</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Latency: 0, Cache Line Size: 32 bytes</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Interrupt: pin B routed to IRQ 3</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Region 0: Memory at c2080000 (32-bit, non-prefetchable) [size=16K]</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [60] Power Management version 3</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Address: 0000000000000000 Data: 0000</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [78] Express (v2) Endpoint, MSI 00</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DevCap:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DevCtl:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Report errors: Correctable- Non-Fatal- Fatal- Unsupported-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>MaxPayload 256 bytes, MaxReadReq 512 bytes</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DevSta:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>LnkCap:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Port #8, Speed 8GT/s, Width x16, ASPM not supported, Exit Latency L0s <1us, L1 <4us</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ClockPM+ Surprise- LLActRep- BwNot-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>LnkCtl:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ASPM Disabled; RCB 64 bytes Disabled- CommClk-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>LnkSta:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR+, OBFF Via message</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Capabilities: [100 v2] Advanced Error Reporting</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>UESta:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>UEMsk:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>UESvrt:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>CESta:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>CEMsk:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>AERCap:<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Kernel driver in use: pci-stub</div><div><br></div></div></div><div><br></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 18, 2016 at 6:03 PM, Alex Williamson <span dir="ltr"><<a href="mailto:alex.williamson@redhat.com" target="_blank">alex.williamson@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Tue, 18 Oct 2016 17:48:59 -0500<br> <span class="">Kevin Vasko <<a href="mailto:kvasko@gmail.com">kvasko@gmail.com</a>> wrote:<br> <br> > Alex,<br> ><br> </span><div><div class="h5">> I think I was able to do it successfully and was scucessfully able to make<br> > the thing fail. It went from (rev a1) to (rev ff) with response of the<br> > header error.<br> ><br> > Instead of doing all devices I just did 1 at a time.<br> ><br> > this was the output of<br> ><br> > # lspci -tv<br> ><br> > +-02.0-[02-08]----00.0-[03-08]<wbr>--+-00.0-[04]--+--00.0 NVIDIA Corporation<br> > GM200 [GeForce GTX TITAN X]<br> > | \-00.1<br> > NVIDIA Corporation Device efb0<br> > +-04.0-[05]--+--00.0 NVIDIA<br> > Corporation GM200 [GeForce GTX TITAN X]<br> > | \-00.1<br> > NVIDIA Corporation Device efb0<br> > +-08.0-[06]--+--00.0 NVIDIA<br> > Corporation GM200 [GeForce GTX TITAN X]<br> > | \-00.1<br> > NVIDIA Corporation Device efb0<br> > +-0c.0-[07]--+--00.0 NVIDIA<br> > Corporation GM200 [GeForce GTX TITAN X]<br> > | \-00.1<br> > NVIDIA Corporation Device efb0<br> > +-14.0-[08]----00.0 Mellanox<br> > Technologies MT27600 Family [ConnectX-3]<br> > +-03.0-[09-12]----00.0-[0a-12]<wbr>--+-08.0-[0b-11]----00.0-[0c-<wbr>11]--+--00.0-[0d]--+-00.0<br> > NVIDIA Corporation GM200 [GeForce GTX TITAN X]<br> ><br> > | \-00.1 NVIDIA Corporation Device 0fb0<br> ><br> > +--04.0-[0e]--+-00.0 NVIDIA Corporation GM200 [GeForce GTX TITAN<br> > X]<br> ><br> > | \-00.1 NVIDIA Corporation Device 0fb0<br> ><br> > +--08.0-[0f]--+-00.0 NVIDIA Corporation GM200 [GeForce GTX TITAN<br> > X]<br> ><br> > | \-00.1 NVIDIA Corporation Device 0fb0<br> ><br> > +--0c.0-[10]--+-00.0 NVIDIA Corporation GM200 [GeForce GTX TITAN<br> > X]<br> ><br> > | \-00.1 NVIDIA Corporation Device 0fb0<br> ><br> > I tried the first device<br> > # virsh nodedev-detach --driver=kvm pci_0000_04_00_0<br> > Device pci_0000_04_00_0 detached<br> ><br> > # virsh nodedev-detach --driver=kvm pci_0000_04_00_1<br> > Device pci_0000_04_00_1 detached<br> ><br> > In the script I put<br> ><br> > DEVS=(<br> > 03:00.0<br> > 04<br> > )<br> ><br> > Ran it 100 times and got no error.<br> ><br> > Ran it for a different device 05<br> ><br> ><br> ><br> > # virsh nodedev-detach --driver=kvm pci_0000_05_00_0<br> > Device pci_0000_05_00_0 detached<br> ><br> > # virsh nodedev-detach --driver=kvm pci_0000_05_00_1<br> > Device pci_0000_05_00_1 detached<br> ><br> > DEVS=(<br> > 03:04.0<br> > 05:<br> > )<br> ><br> ><br> > I saw this.<br> ><br> > #: for i in $(seq 1 100); do ./reset.sh; done<br> > 05:00.0 VGA compatible controller: NVIDIA Corporation GM200 [GeForce GTX<br> > TITAN X] (rev a1)<br> > 05:00.1 Audio device: NVIDIA Corporation Device 0fb0 (rev a1)<br> > 05:00.0 VGA compatible controller: NVIDIA Corporation GM200 [GeForce GTX<br> > TITAN X] (rev a1)<br> > 05:00.1 Audio device: NVIDIA Corporation Device 0fb0 (rev a1)<br> > 05:00.0 VGA compatible controller: NVIDIA Corporation GM200 [GeForce GTX<br> > TITAN X] (rev ff)<br> > 05:00.1 Audio device: NVIDIA Corporation Device 0fb0 (rev ff)<br> ><br> > I repeated this with another device on the system.<br> ><br> > I assume this indicates that that the device is not resetting properly? The<br> > question is where do I go from here? Would this indicate a problem with the<br> > PCI Reset code or a problematic hardware?<br> <br> </div></div>Right, the PCIe link is not coming back for some reason, that seems<br> like a hardware issue. Can you attach the output of 'sudo lspci -vvvs<br> 3:04.0' when you're in this state (replace with the appropriate parent<br> bridge depending on the failed device), maybe we can see if that<br> downstream port is stuck in training.<br> <br> What I would do next is to test each card repeatedly. Do only some<br> cards fail? If so, swap a working card and a non-working card, does<br> the failure follow the card or the slot? I'm not sure what the result<br> is going to be, but if we can't rely on a PCI bus reset then you're<br> really not going to have any repeat-ability with assigning the GPUs.<br> Thanks,<br> <br> Alex<br> </blockquote></div><br></div>