[vfio-users] Failed to return from FLR on SuperMicro A2SDi-16C-HLN4F

[Maik Broemme]

Hi,

I have a SuperMicro A2SDi-16C-HLN4F which uses recently released
Denverton SoC (Intel Atom C3955). This mainboard has one PCI-E 3.0 x4
slot, but whatever card is included there it doesn't work with VFIO.

1. All tried cards work fine in another mainboard using VT-d and in
   another mainboard using AMD-IOMMU.

2. All tried cards report DPC events (AER fixed them). However using
   them on host seems to work fine (tried it for some time now)

[29136.808030] dpc 0000:00:09.0:pcie010: DPC containment event, status:0x1f00 source:0x0000
[29136.808045] pcieport 0000:00:09.0: AER: Corrected error received: id=0048
[29136.808051] pcieport 0000:00:09.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0048(Transmitter ID)
[29136.809533] pcieport 0000:00:09.0:   device [8086:19a4] error status/mask=00001000/00002000
[29136.811079] pcieport 0000:00:09.0:    [12] Replay Timer Timeout

00:09.0 is the PCI bridge and current device behind it is a Digital
Devices GmbH Octopus DVB Adapter. The above error is what I see on host
if using device there, as soon as I start using it vie VFIO I get the
following:

Nov 17 05:06:13 server.theraso.int kernel: vfio-pci 0000:01:00.0: enabling device (0140 -> 0142)
Nov 17 05:06:14 server.theraso.int kernel: vfio_bar_restore: 0000:01:00.0 reset recovery - restoring bars
Nov 17 05:06:36 server.theraso.int kernel: vfio_bar_restore: 0000:01:00.0 reset recovery - restoring bars
Nov 17 05:06:36 server.theraso.int kernel: vfio_bar_restore: 0000:01:00.0 reset recovery - restoring bars

Inside VM I get immediate after boot:

Nov 17 00:25:18 vdr.theraso.int kernel: Disabling IRQ #11
Nov 17 00:25:18 vdr.theraso.int kernel: [<ffffffffc074d060>] qxl_irq_handler [qxl]
Nov 17 00:25:18 vdr.theraso.int kernel: [<ffffffffc03a4570>] usb_hcd_irq [usbcore]
Nov 17 00:25:18 vdr.theraso.int kernel: handlers:
Nov 17 00:25:18 vdr.theraso.int kernel:  secondary_startup_64+0x9f/0x9f
Nov 17 00:25:18 vdr.theraso.int kernel:  x86_64_start_kernel+0x13e/0x161
Nov 17 00:25:18 vdr.theraso.int kernel:  x86_64_start_reservations+0x24/0x26
Nov 17 00:25:18 vdr.theraso.int kernel:  ? early_idt_handler_array+0x120/0x120
Nov 17 00:25:18 vdr.theraso.int kernel:  start_kernel+0x496/0x4b7
Nov 17 00:25:18 vdr.theraso.int kernel:  rest_init+0xd5/0xe0
Nov 17 00:25:18 vdr.theraso.int kernel:  cpu_startup_entry+0x73/0x80
Nov 17 00:25:18 vdr.theraso.int kernel:  do_idle+0x175/0x1e0
Nov 17 00:25:18 vdr.theraso.int kernel:  default_idle_call+0x23/0x30
Nov 17 00:25:18 vdr.theraso.int kernel:  arch_cpu_idle+0xf/0x20
Nov 17 00:25:18 vdr.theraso.int kernel:  default_idle+0x20/0x130
...
Nov 17 00:25:18 vdr.theraso.int kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.11.0-20171110_100015-anatol 04/01/2014
Nov 17 00:25:18 vdr.theraso.int kernel: CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C      4.13.5-1-ARCH #1
Nov 17 00:25:18 vdr.theraso.int kernel: irq 11: nobody cared (try booting with the "irqpoll" option)

If I shutdown the VM, host puts device in a state which makes it not
working anymore:

Nov 17 05:14:00 server.theraso.int kernel: vfio-pci 0000:01:00.0: Failed to return from FLR
Nov 17 05:13:58 server.theraso.int kernel: vfio-pci 0000:01:00.0: timed out waiting for pending transaction; performing function level reset anyway

Next VM start:

Nov 17 00:28:22 server.theraso.int kernel: vfio-pci 0000:01:00.0: Refused to change power state, currently in D3

Moreover I've tried this all already with a RealTek RTL-8169 NIC. The
issue remains the same. As mentioned in the beginning the devices works
fine on other boards.

Any help would be much appreciated to narrow down the problem. The DPC
events occurs also in case of not using VFIO at all.

--Maik