[vfio-users] Binding Radeon R9 380X 4gb to the vfio driver hangs system on recent kernels.

Sat Jun 25 11:26:31 UTC 2016

Hello,

After upgrading from kernel 4.3.3 to the 4.6.2-1 my system completely 
hangs when I'm trying to bind Radeon R9 380X to the vfio driver. I'm 
using arch linux and all kernels I used were with acs override patch 
installed from aur repo. After fail with the latest kernel I tried to 
use latest lts kernel 4.4.13-1, but got the same issue. Before upgrades 
I managed to make passthrough work (ran furmark on win7 guest with the 
same results as on the bare win10), so I'm confident that it's possible 
with my hardware.

journalctl entries before hang:

  ./vfio-bind 0000:01:00.1
  session opened for user root by werman(uid=0)
  session closed for user root
./vfio-bind 0000:01:00.0
  session opened for user root by werman(uid=0)
  kernel: vgaarb: device changed decodes: 
PCI:0000:01:00.0,olddecodes=io+mem,decodes=io+mem:owns=io+mem
  session closed for user root
-- Reboot --

After hang I can't do anything except reboot.

vfio-bind script:

|for dev in "$@"; do vendor=$(cat /sys/bus/pci/devices/$dev/vendor) 
device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e 
/sys/bus/pci/devices/$dev/driver ]; then echo $dev > 
/sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > 
/sys/bus/pci/drivers/vfio-pci/new_id done|

Hardware:

CPU: AMD FX-8320
Motherboard: M5A99X EVO Rev 1 | Chipset: RD9x0 | Southbridge: SB910/950
GPU1: Radeon R9 380X 4gb
GPU2: GT610

Kernel params:

ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1 amd_iommu_dump=1 
amd_iommu=on pcie_acs_override=downstream vga=normal nofb nomodeset 
video=vesafb:off i915.modeset=0

Before, there was pci_stub, but adding/removing it didn't changed anything.

lspci:

01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. 
[AMD/ATI] Tonga XT / Amethyst XT [Radeon R9 380X / R9 M295X Mac Edition] 
[1002:6938] (rev f1) (prog-if 00 [VGA controller])
     Subsystem: Gigabyte Technology Co., Ltd Device [1458:22c8]
     Kernel modules: amdgpu

01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] 
Tonga HDMI Audio [Radeon R9 285/380] [1002:aad8]
     Subsystem: Gigabyte Technology Co., Ltd Device [1458:aad8]
     Kernel driver in use: snd_hda_intel
     Kernel modules: snd_hda_intel

02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF119 
[GeForce GT 610] [10de:104a] (rev a1) (prog-if 00 [VGA controller])
     Subsystem: Gigabyte Technology Co., Ltd Device [1458:362a]
     Kernel driver in use: nvidia
     Kernel modules: nouveau, nvidia_drm, nvidia

02:00.1 Audio device [0403]: NVIDIA Corporation GF119 HDMI Audio 
Controller [10de:0e08] (rev a1)
     Subsystem: Gigabyte Technology Co., Ltd Device [1458:362a]
     Kernel driver in use: snd_hda_intel
     Kernel modules: snd_hda_intel

iommu:

[    0.000000] Warning: PCIe ACS overrides enabled; This may allow 
non-IOMMU protected peer-to-peer DMA
[    1.253800] vgaarb: setting as boot device: PCI:0000:01:00.0
[    1.253801] vgaarb: device added: 
PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none
[    1.253804] vgaarb: device added: 
PCI:0000:02:00.0,decodes=io+mem,owns=none,locks=none
[    1.253806] vgaarb: loaded
[    1.253807] vgaarb: bridge control possible 0000:02:00.0
[    1.253808] vgaarb: bridge control possible 0000:01:00.0
[    1.715368] iommu: Adding device 0000:00:00.0 to group 0
[    1.715730] iommu: Adding device 0000:00:02.0 to group 1
[    1.716087] iommu: Adding device 0000:00:03.0 to group 2
[    1.716443] iommu: Adding device 0000:00:04.0 to group 3
[    1.716800] iommu: Adding device 0000:00:05.0 to group 4
[    1.717156] iommu: Adding device 0000:00:06.0 to group 5
[    1.717514] iommu: Adding device 0000:00:07.0 to group 6
[    1.717872] iommu: Adding device 0000:00:0a.0 to group 7
[    1.718228] iommu: Adding device 0000:00:11.0 to group 8
[    1.718596] iommu: Adding device 0000:00:12.0 to group 9
[    1.718611] iommu: Adding device 0000:00:12.2 to group 9
[    1.718980] iommu: Adding device 0000:00:13.0 to group 10
[    1.718996] iommu: Adding device 0000:00:13.2 to group 10
[    1.719363] iommu: Adding device 0000:00:14.0 to group 11
[    1.719720] iommu: Adding device 0000:00:14.2 to group 12
[    1.720076] iommu: Adding device 0000:00:14.3 to group 13
[    1.720434] iommu: Adding device 0000:00:14.4 to group 14
[    1.720789] iommu: Adding device 0000:00:14.5 to group 15
[    1.721156] iommu: Adding device 0000:00:16.0 to group 16
[    1.721172] iommu: Adding device 0000:00:16.2 to group 16
[    1.721563] iommu: Adding device 0000:01:00.0 to group 17
[    1.721623] iommu: Adding device 0000:01:00.1 to group 17
[    1.721999] iommu: Adding device 0000:02:00.0 to group 18
[    1.722023] iommu: Adding device 0000:02:00.1 to group 18
[    1.722387] iommu: Adding device 0000:03:00.0 to group 19
[    1.722750] iommu: Adding device 0000:04:00.0 to group 20
[    1.723106] iommu: Adding device 0000:05:00.0 to group 21
[    1.723462] iommu: Adding device 0000:06:00.0 to group 22
[    1.723819] iommu: Adding device 0000:07:00.0 to group 23
[    1.723828] iommu: Adding device 0000:08:06.0 to group 14
[    1.723829] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    1.723830] AMD-Vi: Interrupt remapping enabled
[    1.723910] AMD-Vi: Lazy IO/TLB flushing enabled

lsmod:

kvm_amd                65536  0
kvm                    499712 1 kvm_amd
vfio_pci               36864  0
irqbypass              16384  2 kvm,vfio_pci
vfio_virqfd            16384  1 vfio_pci
vfio_iommu_type1       20480  0
vfio                   24576  2 vfio_iommu_type1,vfio_pci

I don't know how to get more info about this issue and I'll appreciate 
any help with solving it.

werman
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160625/92515c14/attachment.htm>