<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 18, 2016 at 11:04 AM, Kevin Vasko <span dir="ltr"><<a href="mailto:kvasko@gmail.com" target="_blank">kvasko@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Alex,<div><br></div><div>(crossing fingers this goes into the correct thread). <br><div><br></div><div>I upgraded this machine to 4.4.0-42-generic.</div><div><br></div><div>I spawned a single VM with 1 GPU immediately after the kernel upgrade. It works. It attached properly and in the VM when I ran lspci, it showed up properly.</div><div><br></div><div>I deleted that VM and started up the system with 4x GPUs, and then it started exhibiting the same issue. Three of the GPUs attached properly. </div><div><br></div><div>This appears to be that it was not resolved with upgrading the kernel. If you don't mind providing instructions on resetting the bus to see if I can narrow this down further (what you were talking about yesterday) that would be appreciated. Any other suggestions would be greatly appreciated as well.</div><div><br></div><div>Here are the logs of the 4 GPU attachment that failed. <br></div><div><br></div><div>On the host.</div><div><br></div><div>/etc/var/log/libvirt/qemu/<wbr>instance-00000185.log</div><div><br></div><div>this shows the /usr/bin/kvm command issuing the connection of the following devices</div><div><br></div><div>-device vfio-pci,host=0f:00.0,id=<wbr>hostdev0,bus=pci.0,addr=0x5 </div><div>-device vfio-pci,host=10:00.0,id=<wbr>hostdev1,bus=pci.0,addr=0x6 </div><div>-device vfio-pci,host=0e:00.0,id=<wbr>hostdev2,bus=pci.0,addr=0x7 </div><div>-device vfio-pci,host=0d:00.0,id=<wbr>hostdev3,bus=pci.0,addr=0x8</div><div><p class="MsoNormal" style="color:rgb(0,0,0);font-family:"times new roman";font-size:medium"><br></p><p class="MsoNormal" style="color:rgb(0,0,0);font-family:"times new roman";font-size:medium">lspci -vnnn -d 10de:17c2 (on the host, I omitted the other 4 GPUs)</p><p class="MsoNormal" style="color:rgb(0,0,0);font-family:"times new roman";font-size:medium"><br></p><p class="MsoNormal" style="color:rgb(0,0,0);font-family:"times new roman";font-size:medium"></p><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"></div><p></p><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3">0d:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX TITAN X] [10de:17c2] (rev a1) (prog-if 00 [VGA controller])</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> subsystem: NVIDIA Corporation Device [10de:1132]</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Flags: fast devsel, IRQ 28</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Memory at b9000000 (32-bit, non-prefetchable) [size=16M]</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Memory at 38ff20000000 (64-bit, prefetchable) [size=256M]</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Memory at 38ff30000000 (64-bit, prefetchable) [size=32M]</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> I/O ports at 3000 [size=128]</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Expansion ROM at ba000000 [disabled] [size=512k]</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Capabilities: [60] Power Management version 3</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Capabilities: [68] MSI: Enable-1 Count=1/1 Maskable- 64bit+</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Capabilities: [78] Express Legacy Endpoint, MSI 00</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Capabilities: [100] Express Legacy Endpoint, MSI 00</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Capabilities: [250] Latency Tolerance Reporting</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Capabilities: [258] L1 PM Substates</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Capabilities: [128] Power Budgeting <?></font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Capabilities: [420] Advanced Error Reporting</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?></font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Capabilities: [900] #19</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Kernel driver in use: vfio-pci</font></p></div><p class="MsoNormal" style="color:rgb(0,0,0);font-family:"times new roman";font-size:medium"></p><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"></div><p></p><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3">0e:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX TITAN X] [10de:17c2] (rev a1) (prog-if 00 [VGA controller])</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> subsystem: NVIDIA Corporation Device [10de:1132]</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Flags: fast devsel, IRQ 28</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Memory at b9000000 (32-bit, non-prefetchable) [size=16M]</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Memory at 38ff20000000 (64-bit, prefetchable) [size=256M]</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Memory at 38ff30000000 (64-bit, prefetchable) [size=32M]</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> I/O ports at 3000 [size=128]</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Expansion ROM at ba000000 [disabled] [size=512k]</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Capabilities: [60] Power Management version 3</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Capabilities: [68] MSI: Enable-1 Count=1/1 Maskable- 64bit+</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Capabilities: [78] Express Legacy Endpoint, MSI 00</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Capabilities: [100] Express Legacy Endpoint, MSI 00</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Capabilities: [250] Latency Tolerance Reporting</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Capabilities: [258] L1 PM Substates</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Capabilities: [128] Power Budgeting <?></font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Capabilities: [420] Advanced Error Reporting</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?></font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Capabilities: [900] #19</font></p><p class="MsoNormal" style="margin:0px"><font color="#000000" face="times new roman" size="3"> Kernel driver in use: vfio-pci</font></p></div><p class="MsoNormal" style="color:rgb(0,0,0);font-family:"times new roman";font-size:medium"><br></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3">0f:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX TITAN X] [10de:17c2] (rev ff) (prog-if ff)</font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3"> !!! Unknown header type 7f</font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3"> Kernel driver in use: vfio-pci</font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3"><br></font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3">10:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX TITAN X] [10de:17c2] (rev a1) (prog-if 00 [VGA controller])</font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3"> subsystem: NVIDIA Corporation Device [10de:1132]</font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3"> Flags: fast devsel, IRQ 28</font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3"> Memory at b9000000 (32-bit, non-prefetchable) [size=16M]</font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3"> Memory at 38ff20000000 (64-bit, prefetchable) [size=256M]</font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3"> Memory at 38ff30000000 (64-bit, prefetchable) [size=32M]</font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3"> I/O ports at 3000 [size=128]</font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3"> Expansion ROM at ba000000 [disabled] [size=512k]</font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3"> Capabilities: [60] Power Management version 3</font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3"> Capabilities: [68] MSI: Enable-1 Count=1/1 Maskable- 64bit+</font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3"> Capabilities: [78] Express Legacy Endpoint, MSI 00</font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3"> Capabilities: [100] Express Legacy Endpoint, MSI 00</font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3"> Capabilities: [250] Latency Tolerance Reporting</font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3"> Capabilities: [258] L1 PM Substates</font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3"> Capabilities: [128] Power Budgeting <?></font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3"> Capabilities: [420] Advanced Error Reporting</font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3"> Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?></font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3"> Capabilities: [900] #19</font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3"> Kernel driver in use: vfio-pci</font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3"><br></font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3">On the VM guest:</font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3"><br></font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3">lspci</font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3"><br></font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3">00:06.0 VGA compatible controller: NVIDIA Corporation GM200 [GeForce GTX TITAN X] (rev a1)</font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3">00:07.0 VGA compatible controller: NVIDIA Corporation GM200 [GeForce GTX TITAN X] (rev a1)</font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3"></font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3">00:08.0 VGA compatible controller: NVIDIA Corporation GM200 [GeForce GTX TITAN X] (rev a1)</font></p><div><br></div><div>dmesg</div><p class="MsoNormal"><font color="#000000" face="times new roman" size="3"><br></font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3">[ 0.787786] pci 0000:00:05.0: [10de:17c2] type 7f class 0xffffff</font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3">[ 0.788970] pci 0000:00:06.0: [10de:17c2] type 00 class 0x030000</font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3">[ 0.855192] pci 0000:00:07.0: [10de:17c2] type 00 class 0x030000</font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3"></font></p><p class="MsoNormal"><font color="#000000" face="times new roman" size="3">[ 0.925003] pci 0000:00:08.0: [10de:17c2] type 00 class 0x030000</font></p></div><div><br></div><div><br></div><div><br></div><div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 17, 2016 at 11:10 PM, Kevin Vasko <span dir="ltr"><<a href="mailto:kvasko@gmail.com" target="_blank">kvasko@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thanks. I'm an idiot. I just replied to the email directly after the subscription and wasn't paying attention. Thank you for correcting it.<div><br></div><div>I was originally running 3.13.0-86-generic upgraded to the 3.19 version to try before I posted this, but got the same results. I'll try a newer version of the kernel and see what happens.</div><div><br></div><div>Sorry to be dense but what do you mean by "retrain properly"? I assume you mean that once it fails to reset it just never recovers? </div><div><br></div><div>We have 2 other machines that I've never seen this problem with so what what you are saying makes sense. This system does have a slightly more specialized PCI bus to be able to stick 8 cards on a single bus (at least that is my understanding), so at this point, either I'm hitting a bug that is fixed in the kernel, or this PCI bus is not doing something that vfio-pci is expecting (would be my speculation).</div><div><br></div><div>I'll report back my findings tomorrow. </div><div><br></div><div>Thanks for the help.</div><span class="m_3859021223680115108gmail-HOEnZb"><font color="#888888"><div><br></div><div>-Kevin</div></font></span><div><div class="m_3859021223680115108gmail-h5"><div><br></div><div><br></div><div><br><div><br></div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 17, 2016 at 5:53 PM, Alex Williamson <span dir="ltr"><<a href="mailto:alex.williamson@redhat.com" target="_blank">alex.williamson@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">(generally a good idea to have a useful subject line)<br> <br> On Mon, 17 Oct 2016 16:26:15 -0500<br> Kevin Vasko <<a href="mailto:kvasko@gmail.com" target="_blank">kvasko@gmail.com</a>> wrote:<br> ><br> > Any suggestions on debugging a !!! Unknown header type 7f?<br> ><br> <br> This usually means that the device didn't come back from bus reset and<br> re-reading the PCI config space where the device was just gives a -1<br> response. lspci tries to interpret that bogus data and gives results<br> like you see. You might try a newer kernel, we've probably fixed some<br> things in the bus reset path since v3.19. It looks like you continue<br> to see the bogus data once it gets into this state, so it's probably<br> not a "simple" device coming out of reset too slowly problem. Possibly<br> the PCIe link doesn't retrain properly sometimes after a bus reset. If<br> a new kernel doesn't help, I could give you instructions for performing<br> a bus reset with setpci and you could test how reliably you can reset<br> the device and read config space after. Thanks,<br> <br> Alex<br> </blockquote></div><br></div></div></div></div></div> </blockquote></div><br></div></div></div></div> </blockquote></div><br></div></div>