[vfio-users] VM Boot Hang After Unclean Shutdown w/ ASMedia ASM1042A USB 3.0 PCI-e

[Adam Zegelin]

Hi,

I have a server with a 4-port USB 3.0 PCI-e card, sold as "HighPoint
4-Port USB 3.0 PCI-Express 2.0 x 4 HBA RocketU 1144D" on Amazon.
It has 4 independent ASM1042A USB 3.0  controllers, one per port,
behind a PCI-e switch.

It shows up as 4 devices in `lspci`:

$ lspci | grep "USB 3.0"
05:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller
06:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller
07:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller
08:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller

I have two VMs, each with two of these controllers connected via PCI-e
pass-through:

$ qemu ... -device vfio-pci,host=05:00.0 -device vfio-pci,host=06:00.0 ...
$ qemu ... -device vfio-pci,host=07:00.0 -device vfio-pci,host=08:00.0 ...

Unfortunately, the PCI-e switch on the board doesn't advertise ACS.
I have read the warnings, accepted the risk, and have enabled ACS override:

Kernel command line:
... pcie_acs_override=downstream,multifunction ...

>From a cold start of the host, the VMs boot successfully. The USB 3.0
controllers are recognised by the guest and any attached USB devices
function without issue.
"Clean" restarts of the VMs also boot correctly.
If a VM crashes, or fails to shutdown successfully, or if it is
`kill`ed, the USB 3.0 controller stops working.

After an ungraceful shutdown, subsequent boots of the VM are broken --
OVMF/EDK2/TianoCore (or whatever you call it -- the firmware) hangs
for ~1min on its logo screen before eventually booting the guest OS.
Any USB 3.0 devices connected to either of the passed-through ports
are unavailable during the firmware stage (i.e, a USB keyboard is
unable to interact with any firmware screens), and once the OS has
booted they aren't recognised as connected devices.

I'm unable to fully confirm that the USB 3.0 controllers are
recognised but malfunctioning (e.g, Win10 detects but is unable to
init the device) as I'm unable to interact with the OS without a USB
mouse + keyboard to open the Device Manager or run `lspci`.

I /think/ this is an issue with PCI/device reset. But I'm not sure how
to proceed.

This doesn't affect my day-to-day usability of my dual-workstation
server, but restating the host repeatedly gets old when experimenting
with things at the VM or VM firmware level.

----

$ pacman -Q qemu linux
qemu 2.8.0-1
linux 4.8.13-1

Running OVMF via
edk2.git-ovmf-x64-0-20170111.b2398.g0772737.noarch.rpm from
kraxel.org.

----

Any input would be appreciated.

Kind regards,
Adam