[vfio-users] Passing through eGPU over thunderbolt 3 partly works, how to debug further?

[Roman Babenko]

Dear all,


I am trying to solve the following problem.

My laptop is Thinkpad P50/XeonE3-1535Mv5/ECC running Ubuntu Xenial. I
need to work with design software like sketch/affinity designer which
is not available on linux. At the moment I use
VirtulaBox/Win10/AffinityDesignerForWin but the performance is not
good so I am looking for a way to provide more GPU power to the VM
where the design software is running.


What I tried.

I looked if I can pass through the internal GPUs from my laptop to the
VM. The laptop has two graphics modes:

1. Optimus: IntelHD530 + nVidia QuadPro M2000M
I tried out GVT-g set up according to https://github.com/01org/gvt-
linux/wiki/GVTg_Setup_Guide. Even though the software inside the VM
can utilize the provided GPU, the lack of the direct output to the
display makes it unusable for me.

2. Discrete: nVidia QuadPro M2000M
I gave a shot to passing through nVidia QuadPro M2000M, could not get
any vital signs out of it. The VBIOS is not UEFI capable, so I used
one from here https://www.techpowerup.com/vgabios/183206/183206 . When
starting up the VM I get something like

Failed to mmap 0000:xx:xx.x BAR 3. Performance may be slow.

In the windows VM one can see the device, but its subsystemId is zero
and the nvidia driver does not recognize and nvidia hardware. Just in
case, I moded the driver by replacing the correct subsystemId by zeros
in xxx.inf files. The driver installer proceeds with installation but
eventually still fails.

The above seems like a dead end, if there is anything else worth
exploring let me know.


Having had no success with internal GPUs I am trying to pass through
an external RX570 GPU to a Windows 10 guest. The GPU connected to the
host over Thunderbolt3 using an Akitio Node enclosure, please see
further details I thought could be useful on the following gist:
https://gist.github.com/sparlampe/5b9dc019a62db8a19941f6526e39dff1


The set up seemed to work so well that it took me some time to notice
there are problems.  The system freezes at random times, sometimes in
2 hours,  sometimes in 5 mins, and  I need to power down both the host
and the eGPU enclosure to be able to start the VM with working
displays again.

There are no errors that I can report either in dmesg output or in top
output before the lockup. The only weird thing I noticed is that
immediately after the VM locks up the mouse cursor on the host still
can be moved 1-2 seconds, then the host locks up and I need to reset
the host and the enclosure.


Any advice on how to proceed diagnosing the problem?


Thanks a lot in advance, Roman