So I've made some progress. I pci stubbed all of my graphics devices and I was able to succesfully bind my 580 and boot into Seabios. I attempted to bind Nouveau to each of my other cards but it failed and crashes my system. Clearly the drivers were fighting each other there. Oddly enough, the NVIDIA drivers actually worked fine with the VM. I'm currently successfully running the VM on card 2 and Arch on the other 2 cards. I did this by booting with all cards pci stubbed, using a standard vfio bind script to bind card 2, starting the VM, and then binding card 1 and 3 using an nvidia bind script (same script but replaces "vfio-pci" with "nvidia"). Unfortunately this setup doesn't stand up to a restart of the VM, it gives an error that it "Cannot read device rom" when I try to boot the VM again. Clearly the nvidia driver is claiming some part of the guest card and blocking access to its bios. As a next step I'm considering trying this arbiter patch: <a href="https://lkml.org/lkml/2014/5/25/94">https://lkml.org/lkml/2014/5/25/94</a> I noticed it's fairly old though, has that functionality already been implemented into the kernel? Is there a way that I can check if it has on my own? Are there any other nvidia specifc patches? I've seen ones for significantly older drivers but I'm using the most up to date ones currently. Thanks. <div class="gmail_quote">On Feb 8, 2016 11:50 PM, "Ben J" <<a href="mailto:btpprograms@gmail.com">btpprograms@gmail.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Switching over to Nouveau did the trick, at least for this issue. I was able to start the VM without hanging and had a different error which I fixed by passing it a ROM file for my GPU. I'm still not getting an output from the passed card though. Any ideas on how I would fix the arbiter issue then? Obviously I'd prefer to keep using the normal nvidia driver on linux if possible, although I suppose I could pass the other card to a Linux VM if all else fails. Thanks for the help so far, I'll continue working on this tomorrow. <div class="gmail_quote">On Feb 8, 2016 11:19 PM, "Alex Williamson" <<a href="mailto:alex.williamson@redhat.com" target="_blank">alex.williamson@redhat.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Mon, 8 Feb 2016 22:42:48 -0500 Ben J <<a href="mailto:btpprograms@gmail.com" target="_blank">btpprograms@gmail.com</a>> wrote: > Hi, I've been working to get VGA passthrough working for a Windows 7 VM for > the past few days. My setup is as follows: > > Processor: I7-3960X > Graphics: 2 NVIDIA GTX 580's and one NVIDIA GT 420 > Monitors: 1 Monitor hooked up to the 420, 1 monitor hooked up to both 580's > Motherboard: ASUS Rampage IV Extreme > OS: Arch Linux > Kernel: Standard, 4.4.1 > > My end goal is to bind one 580 to the VM and the other to Arch. The 420 is > just to allow X to start initially since I haven't found a way to disable > only one 580(They have the same vendor and model ID) at a time. I was > initially following this guide > <<a href="https://www.reddit.com/r/pcmasterrace/comments/3lno0t/gpu_passthrough_revisited_an_updated_guide_on_how/" rel="noreferrer" target="_blank">https://www.reddit.com/r/pcmasterrace/comments/3lno0t/gpu_passthrough_revisited_an_updated_guide_on_how/</a>>. > My guest card was in its own IOMMU group and I'm not using integrated > graphics so from what I understand I don't need the patched kernel. My > cards bound to vfio-pci fine, but I got stuck when I realized that the GTX > 580 doesn't support UEFI booting. I changed over to booting with Seabios > but any time I've enabled "x-vga=on" QEMU just hangs. The QEMU monitor > window freezes, there is no video output to my other monitor, and the only > way to get out of the window is to force close it in htop. It uses 0% cpu > during this. The only log put out into dmesg is "kvm: zapping shadow pages > for mmio generation wraparound" which looks fairly unrelated from my > searches. > > I've tested the following things: > Enabling vga cirrus and x-vga - Opens an empty, black graphical window, > still hangs > Enabling vga cirrus and no x-vga - Boots as expected, but obviously no > passthrough > Passing in a ROM for the graphics card - Hangs > Enabling unsafe interrupts - Hangs > Using linux-vfio kernel from AUR - Hangs > Lowering RAM (based on this post > <<a href="https://bugzilla.kernel.org/show_bug.cgi?id=107561" rel="noreferrer" target="_blank">https://bugzilla.kernel.org/show_bug.cgi?id=107561</a>>) - Hangs > > Here are my boot options, booting using rEFInd: > "root=UUID=4accaa3c-6d16-40e9-bb95-e78d160962a5 ro intel_iommu=on > vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=10de:1080,10de:0e09" > > The two pci-stub ID's are for the GTX 580 and its audio card, it disables > both 580's. > > Here is my QEMU startup script: > > qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off \ > -smp 4,sockets=1,cores=4,threads=1 \ > -bios /usr/share/qemu/bios.bin -vga none \ > -device vfio-pci,host=02:00.0,multifunction=on,x-vga=on \ > -device vfio-pci,host=02:00.1 \ > -drive file=/mnt/LinuxStorage/VMs/VGAWin7.qcow2,id=disk,format=qcow2,if=virtio \ > -boot menu=on \ > > At this point I don't know of any log files to look at so I'm essentially > taking shots in the dark. Are there any other logs that could be useful? > Has anyone else experienced issues like this? I've done a lot of searching > around and haven't come across anyone who has had this kind of hang without > an error message. Any ideas would be appreciated, thanks. Possibly a VGA arbiter deadlock with the host graphics not allowing the the assigned graphics a chance to access VGA space. What happens if you pci-stub all the NVIDIA devices (pci-stub.ids=10de:ffffffff)? X obviously won't start on the host, but you should be able to login from another system (VGA text mode on the 420 will get really dicey once the VM starts, but the goal is just to test if the driver with X running on the 420 is the problem). There used to be a problem with this using the nvidia.ko driver and a workaround was to hack on the wrapper script to prevent it from permanently grabbing the VGA arbiter resources. That seems to be fixed for most folks, but perhaps it still exits with the older driver you might be using with a 4-series card. Using nouveau in the host may also avoid it. 5-series cards seem to cause folks quite a bit of trouble when trying to assign them. Good luck, Alex </blockquote></div> </blockquote></div>