[vfio-users] How to hard-reset a PCI-e device?

[Taiidan at gmx.com]

On 02/15/2017 08:30 PM, Alex Williamson wrote:
> On Wed, Feb 15, 2017 at 5:53 PM,Taiidan at gmx.com  <Taiidan at gmx.com>  wrote:
>
>> This is for a server, so that isn't useful.
>>
>> I can't understand why the maintainers of vfio don't do a hard reset if
>> FLR doesn't work.
>>
> You have a graphics card that supports FLR?  An FLR doesn't return a
> status, how do we know it didn't work?  If a device has a known broken
> reset mechanism, report it and maybe we can avoid using it.  There is no
> standard way to remove power from a slot unless you have a hotplug capable
> slot.
>
My mistake, I do not.

I had assumed the "Failed to return from FLR" meant that it had it and 
then I also read the lspci info on the wrong device (oops)


Thanks for the info (and sorry for the kinda snide remark)


This is what happens: (trimmed irrelevant parts)

[  395.927577] vfio_ecap_init: 0000:05:00.0 hiding ecap 0x19 at 0x900
[  395.983252] pciehp 0000:00:02.0:pcie004: Timeout on hotplug command 
0x13f1 (issued 390680 msec ago)
[  397.011388] pciehp 0000:00:02.0:pcie004: Timeout on hotplug command 
0x03f1 (issued 1028 msec ago)
[  398.075480] pciehp 0000:00:02.0:pcie004: Timeout on hotplug command 
0x03f1 (issued 2092 msec ago)
[  398.103494] vfio-pci 0000:00:13.2: enabling device (0000 -> 0002)
[  398.151607] vfio_cap_init: 0000:00:13.2 hiding cap 0xa
[  398.223500] pciehp 0000:00:0b.0:pcie004: Timeout on hotplug command 
0x13f1 (issued 390836 msec ago)
[  399.251615] pciehp 0000:00:0b.0:pcie004: Timeout on hotplug command 
0x03f1 (issued 1028 msec ago
[  399.282188] AMD-Vi: Event logged [
[  399.282195] INVALID_DEVICE_REQUEST device=00:0c.0 
address=0x000000fdf8920000 flags=0x0a00]
[  400.315712] pciehp 0000:00:0b.0:pcie004: Timeout on hotplug command 
0x03f1 (issued 2092 msec ago)
[  400.487723] pciehp 0000:00:02.0:pcie004: Timeout on hotplug command 
0x13f1 (issued 2412 msec ago)
[  401.519839] pciehp 0000:00:02.0:pcie004: Timeout on hotplug command 
0x03f1 (issued 1032 msec ago)
[  402.555919] pciehp 0000:00:02.0:pcie004: Timeout on hotplug command 
0x03f1 (issued 2068 msec ago)
[  402.556241] vfio_bar_restore: 0000:05:00.1 reset recovery - restoring 
bars
[  402.556274] vfio_bar_restore: 0000:05:00.1 reset recovery - restoring 
bars
[  402.556406] vfio_bar_restore: 0000:05:00.0 reset recovery - restoring 
bars
[  402.556440] vfio_bar_restore: 0000:05:00.0 reset recovery - restoring 
bars
[  403.298118] vfio_bar_restore: 0000:05:00.0 reset recovery - restoring 
bars
[  403.329886] vfio_bar_restore: 0000:05:00.1 reset recovery - restoring 
bars
[  415.452779] pciehp 0000:00:02.0:pcie004: Timeout on hotplug command 
0x13f1 (issued 12896 msec ago)
[  416.464842] pciehp 0000:00:02.0:pcie004: Timeout on hotplug command 
0x03f1 (issued 1012 msec ago)
[  417.500876] pciehp 0000:00:02.0:pcie004: Timeout on hotplug command 
0x03f1 (issued 2048 msec ago)
[  417.692888] pci_raw_set_power_state: 114 callbacks suppressed
[  417.692894] vfio-pci 0000:05:00.1: Refused to change power state, 
currently in D3
[  417.700883] pciehp 0000:00:0b.0:pcie004: Timeout on hotplug command 
0x13f1 (issued 17384 msec ago)
[  418.704943] pciehp 0000:00:0b.0:pcie004: Timeout on hotplug command 
0x03f1 (issued 1004 msec ago)
[  419.736960] pciehp 0000:00:0b.0:pcie004: Timeout on hotplug command 
0x03f1 (issued 2036 msec ago)

I then do a virsh destroy as the VM didn't start

----
[  476.475584] vfio_ecap_init: 0000:05:00.0 hiding ecap 0xffff at 0xff
[  478.513411] vfio_cap_init: 0000:05:00.1 hiding cap 0xff
- NOTE: this is spammed many times
----then

[  478.535720] vfio-pci 0000:01:00.0: enabling device (0400 -> 0403)
[  478.543731] pciehp 0000:00:02.0:pcie004: Timeout on hotplug command 
0x13f1 (issued 61044 msec ago)
[  479.571663] pciehp 0000:00:02.0:pcie004: Timeout on hotplug command 
0x03f1 (issued 1028 msec ago)
[  480.635584] pciehp 0000:00:02.0:pcie004: Timeout on hotplug command 
0x03f1 (issued 2092 msec ago)
[  480.663596] vfio-pci 0000:00:13.2: enabling device (0000 -> 0002)
[  480.711690] vfio_cap_init: 0000:00:13.2 hiding cap 0xa
[  481.647531] pciehp 0000:00:02.0:pcie004: Timeout on hotplug command 
0x13f1 (issued 1012 msec ago)
[  482.671457] pciehp 0000:00:02.0:pcie004: Timeout on hotplug command 
0x03f1 (issued 1024 msec ago)
[  483.703377] pciehp 0000:00:02.0:pcie004: Timeout on hotplug command 
0x03f1 (issued 2056 msec ago)
[  483.727378] vfio-pci 0000:05:00.1: Refused to change power state, 
currently in D3
[  484.467337] vfio-pci 0000:05:00.1: timed out waiting for pending 
transaction; performing function level reset anyway
[  485.671249] vfio-pci 0000:05:00.1: Failed to return from FLR
[  485.691235] vfio-pci 0000:05:00.0: Refused to change power state, 
currently in D3
[  486.415199] vfio-pci 0000:05:00.0: timed out waiting for pending 
transaction; performing function level reset anyway

After this I must reboot or I cant assign the device, I will just get 
the "Failed to return from FLR" again and again.

GFX card: geforce 780 gtx
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20170311/9bf90589/attachment.htm>