[vfio-users] NVIDIA driver cause to black screen and can't launch desktop environment

[Zhifeng Hu]

Hi all,

I'm able to pass through a NVIDIA graphic card(GTX750 Ti) to RHEL 5.11 successfully,
For performance reasons, I need to install NVIDIA's official driver,
After installing NVIDIA's official driver, the guest system always encountered black screen and can't enter desktop environment.
But I still can log in via SSH, so the guest system didn't crash.
I tested more than 30 times, only once was successful.

I checked the /var/log/Xorg.0.log,
Below is the failed log:
---------------------------------------------------------------------------------------------
(II) Jul 19 19:49:36 NVIDIA: Using 12288.00 MB of virtual memory for indirect memory
(II) Jul 19 19:49:36 NVIDIA:     access.
(WW) Jul 19 19:49:36 NVIDIA(0): ACPI: Error: Unable to find the brightness file path under
(WW) Jul 19 19:49:36 NVIDIA(0):     /proc/acpi/video. The NVIDIA X driver will not be able to
(WW) Jul 19 19:49:36 NVIDIA(0):     respond to ACPI brightness change hotkey events.

Fatal server error:
NVIDIA: A GPU exception occurred during X server initialization
---------------------------------------------------------------------------------------------

And this is the successful log:
---------------------------------------------------------------------------------------------
(II) Jul 19 20:19:00 NVIDIA: Using 12288.00 MB of virtual memory for indirect memory
(II) Jul 19 20:19:00 NVIDIA:     access.
(WW) Jul 19 20:19:00 NVIDIA(0): ACPI: Error: Unable to find the brightness file path under
(WW) Jul 19 20:19:00 NVIDIA(0):     /proc/acpi/video. The NVIDIA X driver will not be able to
(WW) Jul 19 20:19:00 NVIDIA(0):     respond to ACPI brightness change hotkey events.
(II) Jul 19 20:19:00 NVIDIA(0): Setting mode "DFP-0:nvidia-auto-select"
(II) Loading extension NV-GLX
(==) NVIDIA(0): Disabling shared memory pixmaps
(==) NVIDIA(0): Backing store disabled
(==) NVIDIA(0): Silken mouse enabled
(**) Option "dpms"
(**) NVIDIA(0): DPMS enabled
(II) Loading extension NV-CONTROL
(==) RandR enabled
(II) Setting vga for screen 0.
---------------------------------------------------------------------------------------------

For detail log information, please go: https://pastebin.com/U7f0Up9q and https://pastebin.com/QTXMKMuC

Here is my setup:
CPU: Intel i7-6700K
Motherboard: Gigabyte B150-HD3
IGD: Intel® HD Graphics 530 (used by the host)
Graphics Card: GTX750 Ti (used by the Guest)
Host OS: Fedora 25 (Fedora-Workstation-Live-x86_64-25-1.3.iso)
Host Kernel: 4.8.6-300.fc25.x86_64
QEMU version: qemu-2.7.0-7.fc25
Guest OS: RHEL 5.11
Guest Kernel: 2.6.18-398.el5
Guest Driver: NVIDIA-Linux-x86_64-375.66.run

I run QEMU with the following command:
qemu-system-x86_64 -enable-kvm -m 4G -cpu host,kvm=off -smp 4,sockets=1,cores=2,threads=2 -hda ~/rhel511.img -usbdevice host:0000:0538 -usbdevice host:413c:2107 -device vfio-pci,host=01:00.0,x-vga=on -device vfio-pci,host=01:00.1 -vga none -net nic -net tap

What I've tried so far:
1. Install RHEL 5.11 on a physical machine without QEMU (use the same hardware platform),
and install the same driver from NVIDIA, everything is okay.
It shows that both RHEL 5.11 and NVIDIA driver are not the cause of this problem, they are compatible.

2. Tried a few different distros (RHEL 7.3, CentOS 7.3) for host OS, all have the same problem.

3. Tried some newer or older QEMU (qemu 2.9.0, qemu 2.6.2), all have the same problem.

4. Tried different hardware platform (i5-4460 + QEMU + GT 710), and got the same issue.

5. Upgrade the kernel of the guest OS (The origin kernel of RHEL 5.11 is 2.6.18-398.el5), The problem disappeared from 2.6.25

To Be Honest, I am a little confused, I can't figure out what is the root cause.
It seems to be related to the kernel, and it seems not to be the case .
I also suspect that the kernel 2.6.18 and NVIDIA driver are not incompatible with QEMU simulator, since it only happen on QEMU, not on physical machine.

For business reasons, I can't upgrade the kernel to 2.6.25

Any hint is highly appreciated.

Best regards,
Zhifeng