[vfio-users] QEMU VGA passthrough hangs with no error - NVIDIA GTX 580

[Alex Williamson]

On Mon, 8 Feb 2016 22:42:48 -0500
Ben J <btpprograms at gmail.com> wrote:

> Hi, I've been working to get VGA passthrough working for a Windows 7 VM for
> the past few days. My setup is as follows:
> 
> Processor: I7-3960X
> Graphics: 2 NVIDIA GTX 580's and one NVIDIA GT 420
> Monitors: 1 Monitor hooked up to the 420, 1 monitor hooked up to both 580's
> Motherboard: ASUS Rampage IV Extreme
> OS: Arch Linux
> Kernel: Standard, 4.4.1
> 
> My end goal is to bind one 580 to the VM and the other to Arch. The 420 is
> just to allow X to start initially since I haven't found a way to disable
> only one 580(They have the same vendor and model ID) at a time. I was
> initially following this guide
> <https://www.reddit.com/r/pcmasterrace/comments/3lno0t/gpu_passthrough_revisited_an_updated_guide_on_how/>.
> My guest card was in its own IOMMU group and I'm not using integrated
> graphics so from what I understand I don't need the patched kernel. My
> cards bound to vfio-pci fine, but I got stuck when I realized that the GTX
> 580 doesn't support UEFI booting. I changed over to booting with Seabios
> but any time I've enabled "x-vga=on" QEMU just hangs. The QEMU monitor
> window freezes, there is no video output to my other monitor, and the only
> way to get out of the window is to force close it in htop. It uses 0% cpu
> during this. The only log put out into dmesg is "kvm: zapping shadow pages
> for mmio generation wraparound" which looks fairly unrelated from my
> searches.
> 
> I've tested the following things:
> Enabling vga cirrus and x-vga - Opens an empty, black graphical window,
> still hangs
> Enabling vga cirrus and no x-vga - Boots as expected, but obviously no
> passthrough
> Passing in a ROM for the graphics card - Hangs
> Enabling unsafe interrupts - Hangs
> Using linux-vfio kernel from AUR - Hangs
> Lowering RAM (based on this post
> <https://bugzilla.kernel.org/show_bug.cgi?id=107561>) - Hangs
> 
> Here are my boot options, booting using rEFInd:
> "root=UUID=4accaa3c-6d16-40e9-bb95-e78d160962a5 ro intel_iommu=on
> vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=10de:1080,10de:0e09"
> 
> The two pci-stub ID's are for the GTX 580 and its audio card, it disables
> both 580's.
> 
> Here is my QEMU startup script:
> 
> qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off \
> -smp 4,sockets=1,cores=4,threads=1 \
> -bios /usr/share/qemu/bios.bin  -vga none \
> -device vfio-pci,host=02:00.0,multifunction=on,x-vga=on \
> -device vfio-pci,host=02:00.1 \
> -drive file=/mnt/LinuxStorage/VMs/VGAWin7.qcow2,id=disk,format=qcow2,if=virtio \
> -boot menu=on \
> 
> At this point I don't know of any log files to look at so I'm essentially
> taking shots in the dark. Are there any other logs that could be useful?
> Has anyone else experienced issues like this? I've done a lot of searching
> around and haven't come across anyone who has had this kind of hang without
> an error message. Any ideas would be appreciated, thanks.

Possibly a VGA arbiter deadlock with the host graphics not allowing the
the assigned graphics a chance to access VGA space.  What happens if
you pci-stub all the NVIDIA devices (pci-stub.ids=10de:ffffffff)?  X
obviously won't start on the host, but you should be able to login from
another system (VGA text mode on the 420 will get really dicey once the
VM starts, but the goal is just to test if the driver with X running on
the 420 is the problem).  There used to be a problem with this using
the nvidia.ko driver and a workaround was to hack on the wrapper script
to prevent it from permanently grabbing the VGA arbiter resources.
That seems to be fixed for most folks, but perhaps it still exits with
the older driver you might be using with a 4-series card.  Using
nouveau in the host may also avoid it.  5-series cards seem to cause
folks quite a bit of trouble when trying to assign them.  Good luck,

Alex