<div dir="ltr">Alex,<div><br></div><div>I think I was able to do it successfully and was scucessfully able to make the thing fail. It went from (rev a1) to (rev ff) with response of the header error.</div><div><br></div><div>Instead of doing all devices I just did 1 at a time. </div><div><br></div><div>this was the output of </div><div><br></div><div><span style="font-size:12.8px"># lspci -tv</span><br></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">+-02.0-[02-08]----00.0-[03-08]--+-00.0-[04]--+--00.0 NVIDIA Corporation GM200 [GeForce GTX TITAN X]</span></div><div><span style="font-size:12.8px"> | \-00.1 NVIDIA Corporation Device efb0</span></div><div><span style="font-size:12.8px"> </span><span style="font-size:12.8px">+-04.0-[05]--+--00.0 NVIDIA Corporation GM200 [GeForce GTX TITAN X]</span><span style="font-size:12.8px"> </span></div><div><span style="font-size:12.8px"> | \-00.1 NVIDIA Corporation Device efb0</span><span style="font-size:12.8px"><br></span></div><div><div><span style="font-size:12.8px"> </span><span style="font-size:12.8px">+-08.0-[06]--+--00.0 NVIDIA Corporation GM200 [GeForce GTX TITAN X]</span><span style="font-size:12.8px"> </span></div><div><span style="font-size:12.8px"> | \-00.1 NVIDIA Corporation Device efb0</span></div></div><div><div><span style="font-size:12.8px"> </span><span style="font-size:12.8px">+-0c.0-[07]--+--00.0 NVIDIA Corporation GM200 [GeForce GTX TITAN X]</span><span style="font-size:12.8px"> </span></div><div><span style="font-size:12.8px"> | \-00.1 NVIDIA Corporation Device efb0</span></div></div><div><div><span style="font-size:12.8px"> </span><span style="font-size:12.8px">+-14.0-[08]----00.0 Mellanox Technologies MT27600 Family [ConnectX-3]</span></div></div><div><span style="font-size:12.8px">+-03.0-[09-12]----00.0-[0a-12]--+-08.0-[0b-11]----00.0-[0c-11]--+--00.0-[0d]--+-00.0 NVIDIA Corporation GM200 [GeForce GTX TITAN X]</span></div><div><span style="font-size:12.8px"> | \-00.1 NVIDIA Corporation Device 0fb0</span></div><div><span style="font-size:12.8px"> </span><span style="font-size:12.8px">+--04.0-[0e]--+-00.0 NVIDIA Corporation GM200 [GeForce GTX TITAN X]</span></div><div><span style="font-size:12.8px"> | \-00.1 NVIDIA Corporation Device 0fb0</span></div><div><span style="font-size:12.8px"> +--08.0-[0f]--+-00.0 NVIDIA Corporation GM200 [GeForce GTX TITAN X]</span></div><div><span style="font-size:12.8px"> | \-00.1 NVIDIA Corporation Device 0fb0</span></div><div><span style="font-size:12.8px"> +--0c.0-[10]--+-00.0 NVIDIA Corporation GM200 [GeForce GTX TITAN X]</span></div><div><span style="font-size:12.8px"> | \-00.1 NVIDIA Corporation Device 0fb0</span><span style="font-size:12.8px"> </span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">I tried the first device</span></div><div><span style="font-size:12.8px"># virsh nodedev-detach --driver=kvm pci_0000_04_00_0</span><br style="font-size:12.8px"><span style="font-size:12.8px">Device pci_0000_04_00_0 detached</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px"># virsh nodedev-detach --driver=kvm pci_0000_04_00_1</span><br style="font-size:12.8px"><span style="font-size:12.8px">Device pci_0000_04_00_1 detached</span><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">In the script I put</span></div><div><span style="font-size:12.8px"><br></span></div><div>DEVS=(<br> 03:00.0</div><div> 04<br>)<span style="font-size:12.8px"><br></span></div><div><br></div><div>Ran it 100 times and got no error.</div><div><br></div><div>Ran it for a different device 05</div><div><br></div><div><br></div><div><br></div><div><div><span style="font-size:12.8px"># virsh nodedev-detach --driver=kvm pci_0000_05_00_0</span><br style="font-size:12.8px"><span style="font-size:12.8px">Device pci_0000_05_00_0 detached</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px"># virsh nodedev-detach --driver=kvm pci_0000_05_00_1</span><br style="font-size:12.8px"><span style="font-size:12.8px">Device pci_0000_05_00_1 detached</span></div></div><div><br></div><div><div>DEVS=(<br> 03:04.0</div><div> 05:<br>)<span style="font-size:12.8px"><br></span></div></div><div><br></div><div><br></div><div>I saw this.</div><div><br></div><div>#: for i in $(seq 1 100); do ./reset.sh; done</div><div>05:00.0 VGA compatible controller: NVIDIA Corporation GM200 [GeForce GTX TITAN X] (rev a1)</div><div>05:00.1 Audio device: NVIDIA Corporation Device 0fb0 (rev a1)</div><div><div>05:00.0 VGA compatible controller: NVIDIA Corporation GM200 [GeForce GTX TITAN X] (rev a1)</div><div>05:00.1 Audio device: NVIDIA Corporation Device 0fb0 (rev a1)</div></div><div><div>05:00.0 VGA compatible controller: NVIDIA Corporation GM200 [GeForce GTX TITAN X] (rev ff)</div><div>05:00.1 Audio device: NVIDIA Corporation Device 0fb0 (rev ff)</div></div><div><br></div><div>I repeated this with another device on the system.</div><div><br></div><div>I assume this indicates that that the device is not resetting properly? The question is where do I go from here? Would this indicate a problem with the PCI Reset code or a problematic hardware? </div><div><br></div><div><br></div><div>-Kevin</div><div><br></div><div><br></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px"><br></span></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 18, 2016 at 11:49 AM, Alex Williamson <span dir="ltr"><<a href="mailto:alex.williamson@redhat.com" target="_blank">alex.williamson@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-">On Tue, 18 Oct 2016 11:04:14 -0500<br>
Kevin Vasko <<a href="mailto:kvasko@gmail.com">kvasko@gmail.com</a>> wrote:<br>
<br>
> Alex,<br>
><br>
> (crossing fingers this goes into the correct thread).<br>
><br>
> I upgraded this machine to 4.4.0-42-generic.<br>
><br>
> I spawned a single VM with 1 GPU immediately after the kernel upgrade. It<br>
> works. It attached properly and in the VM when I ran lspci, it showed up<br>
> properly.<br>
><br>
> I deleted that VM and started up the system with 4x GPUs, and then it<br>
> started exhibiting the same issue. Three of the GPUs attached properly.<br>
><br>
> This appears to be that it was not resolved with upgrading the kernel. If<br>
> you don't mind providing instructions on resetting the bus to see if I can<br>
> narrow this down further (what you were talking about yesterday) that would<br>
> be appreciated. Any other suggestions would be greatly appreciated as well.<br>
<br>
</span>Ok, you're going to need to identify the parent bridge for the GPUs.<br>
You can do this with 'lspci -tv'. If you need help, send the output of<br>
that command. Here's an example:<br>
<br>
# lspci -tv<br>
-[0000:00]-+-00.0 Intel Corporation 5520/5500/X58 I/O Hub to ESI Port<br>
+-01.0-[01]--+-00.0 Intel Corporation 82576 Gigabit Network Connection<br>
| \-00.1 Intel Corporation 82576 Gigabit Network Connection<br>
+-03.0-[02]----00.0 Fresco Logic FL1100 USB 3.0 Host Controller<br>
+-07.0-[03]--+-00.0 Intel Corporation Ethernet Controller X710 for 10GbE SFP+<br>
| \-00.1 Intel Corporation Ethernet Controller X710 for 10GbE SFP+<br>
...<br>
<br>
Say I want to do a bus reset on the X710 ethernet devices at 03:00.0<br>
and 03:00.1. This should be similar to a GPU and companion audio<br>
device. The parent bridge is device 00:07.0. I can double check this<br>
by running lspci on this device:<br>
<br>
# lspci -vs 00:07.0<br>
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 22) (prog-if 00 [Normal decode])<br>
Flags: bus master, fast devsel, latency 0, IRQ 27<br>
Bus: primary=00, secondary=03, subordinate=03, sec-latency=0<br>
^^^^^^^^^^^^<br>
<br>
The secondary bus is 03, thus it's the parent device of 03:00.[01].<br>
<br>
Prior to performing a bus reset, attach all the affected devices to a<br>
driver that isn't going to be making use of the devices, for instance<br>
pci-stub. We can do this with virsh using:<br>
<br>
# virsh nodedev-detach --driver=kvm pci_0000_03_00_0<br>
Device pci_0000_03_00_0 detached<br>
<br>
# virsh nodedev-detach --driver=kvm pci_0000_03_00_1<br>
Device pci_0000_03_00_1 detached<br>
<br>
The "--driver=kvm" simply selects pci-stub rather than vfio-pci, which<br>
would otherwise be the default.<br>
<br>
Also note that after a bus reset, the downstream devices are not going<br>
to be usable until after a system reboot. Our goal is to see how<br>
reliably we can perform a bus reset and have the devices re-appear, we<br>
cannot make use of them beyond running lspci on them without a system<br>
reboot.<br>
<br>
Ok, so for each GPU you should know the parent bridge, the address of<br>
the GPUs themselves, and each GPU and companion audio device should be<br>
bound to pci-stub.<br>
<br>
Using the dual port NICs as stand-ins for your GPUs, we need a script<br>
like this:<br>
<br>
# cat reset.sh<br>
#!/bin/sh<br>
<br>
DEVS=(<br>
00:01.0 # parent of 01:<br>
01: # affected devices of 01.0<br>
00:07.0 # parent of 03:<br>
03: # affected devices of 07.0<br>
# change the entries above for your system<br>
# you will have more devices here, a parent bridge<br>
# followed by the bus of the affected GPU, 0f:, 10:, 0e:, 0d:<br>
)<br>
<br>
i=0<br>
<br>
while [ $i -lt ${#DEVS[@]} ]; do<br>
setpci -s ${DEVS[$i]} 3e.w=40:40 # Set 2ndary bus reset bit<br>
sleep 0.2<br>
setpci -s ${DEVS[$i]} 3e.w=0:40 # Clear 2ndary bus reset bit<br>
sleep 1<br>
# when this reports abnormally, we've failed<br>
lspci -s ${DEVS[$(( $i + 1 ))]}<br>
i=$(( $i + 2 ))<br>
done<br>
<br>
Don't forget to chmod 755 the script. Run it once and it should<br>
produce something like this (of course with your GPUs instead of my<br>
NICs):<br>
<br>
# ./reset.sh<br>
01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)<br>
01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)<br>
03:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)<br>
03:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)<br>
<br>
If that works, then run it 100 times:<br>
<br>
# for i in $(seq 1 100); do ./reset.sh; done<br>
<br>
If you start seeing "(rev ff) (prog-if ff)" then the device has<br>
failed. (left as an exercise to the reader to automatically stop on<br>
this condition ;) Please report what you find and remember that it's<br>
expected that you will need to reboot the system after performing this<br>
test to get the devices back into a workable state. We're not saving<br>
and restoring the state of the devices around reset. Thanks,<br>
<br>
Alex<br>
</blockquote></div><br></div></div>