[vfio-users] Radeon passthrough causes host lockups when using vfio

[Arjen]

Hi,

I'm getting ring 0 stalls and GPU lockups errors in the radeon driver 
when starting Xorg. When passing through a AMD Radeon 7750 using VFIO., 
the display looks garbled with random-looking patterns and colors. The 
guest tries to reset the GPU for a few seconds, until the host freezes 
to the point where the reset button does not even work.

Passthrough of other devices such as on-board audio, USB3 controller 
work fine with VFIO and linux guests. I have a VM that runs Fedora 21 
and works fine on a kernel 2.6.32 (RedHat+OpenVZ+Proxmox) using 
pci-assign. The same VM experiences the ring 0 stall, GPU lockup and a 
host freeze with kernel 4.4.21 (Ubuntu+Proxmox) using vfio-pci.

The differences in kvm command-line parameters are:
Linux 2.6.32 + QEMU 2.2.1:  -cpu SandyBridge,+x2apic
  -device 
'pci-assign,host=01:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on' 

  -device 
'pci-assign,host=01:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1'
  -device 'pci-assign,host=00:1b.0,id=hostpci1,bus=pci.0,addr=0x11'
Linux 4.4.21 + QEMU 2.7.0:  -cpu 
'SandyBridge,+kvm_pv_unhalt,+kvm_pv_eoi,enforce,vendor=GenuineIntel'
  -device 
'vfio-pci,host=01:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on'
  -device 
'vfio-pci,host=01:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1'
  -device 'vfio-pci,host=00:1b.0,id=hostpci1,bus=pci.0,addr=0x11'

I have tried to use the same kvm command-line in 4.4, but it complains 
about an invalid argument for hostpci0.0 and I cannot find out which or 
why. (It also does not support +x2apic anymore)

I have tested also SeaBIOS (as secondary VGA) and OVMF (with single 
x-vga=1) using Ubuntu Gnome 16.10, using machine q35 because the radeon 
driver assumes a PCIE root, with the same results. According to lspci, 
MSI-X is enabled. I have tried various kernel module settings for radeon 
and vfio-pci that I found on this mailinglist  to no avail 
(radeon.hard_reset=1, .dpm=1, ..., vfio-pci.disable_idle_d3).

Has someone experienced similar issues and found a fix for it? Can 
someone advise me on things to investigate or try to get my desktop VM 
running with VFIO (or how to fall-back to pci-assign)?

I noticed people having more luck with NVidia, but I replaced mine with 
AMD because I could get it to work with pci-assign some years back, and 
would like to keep using open-source drivers.

Thank you and kind regards,
	Arjen