[vfio-users] Code 43 with Rampage III and GTX660 in Windows 10

Thu Jun 23 21:37:52 UTC 2016

I'm trying this on Proxmox VE, which is Debian 8 based.

I'm awaiting a UEFI bios from EVGA, but for now I don't have it available.

I'm following the "tips and tricks" series, but we differ at part 4, as I'd like this to eventually work with my distro's web frontend. 

With the goal of finding an invocation of the KVM command that will work, I did a ps ax|grep kvm and modified what wasn't working in my distro to come up with something that I think should work, which is:

=================
/usr/bin/kvm -id 102 -chardev socket,id=qmp,path=/var/run/qemu-server/102.qmp,server,nowait -mon chardev=qmp,mode=control -pidfile /var/run/qemu-server/102.pid -daemonize -smbios type=1,uuid=6a51d2b1-c63f-4eef-9c58-30743a0252f6 -name MSW10E-1 -smp 8,sockets=1,cores=8,maxcpus=8 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000 -vga std -vnc unix:/var/run/qemu-server/102.vnc,x509,password -no-hpet -cpu host,kvm=off,hv_vendor_id=Nvidia43FIX,+kvm_pv_unhalt,+kvm_pv_eoi -m 12288 -k en-us -readconfig /usr/share/qemu-server/pve-q35.cfg -device usb-tablet,id=tablet,bus=ehci.0,port=1 -device vfio-pci,host=04:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:3f1e9afe6fdb -drive file=/var/lib/vz/template/iso/MSW106in1.iso,if=none,id=drive-ide2,media=cdrom,aio=threads -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=100 -drive file=/dev/zvol/rpool/data/vm-102-disk-1,if=none,id=drive-virtio0,cache=writeback,format=raw,aio=threads,detect-zeroes=on -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=200 -netdev type=tap,id=net0,ifname=tap102i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=32:36:39:65:33:31,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 -rtc driftfix=slew,base=localtime -machine type=q35 -global kvm-pit.lost_tick_policy=discard
=================

Specifically, I added the ,kvm=off,hv_vendor_id=Nvidia43FIX and removed all of the hypervisor hv_ stuff which might flag the nvidia driver.

My understanding is that I'm still running up against the "problem" of the nvidia driver detecting that it is in a VM, and going into stupid mode.

I did take a shortcut in that I don't load the vfio module in the initrd, but instead I assign driver overrides to GPUs respectively for both the nouveau and vfio-pci, with a slightly modified version of the script from the howto series.  I don't think it's causing a problem in my setup, but I realize it isn't ideal..  Nouveau seems to work well on the boot GPU, though I haven't tested it much.

Here is some additional info which may be helpful:

==========================================

 dmesg |grep -e DMAR -e IOMMU
[    0.000000] ACPI: DMAR 0x000000009F7980C0 000130 (v01 AMI    OEMDMAR  00000001 MSFT 00000097)
[    0.000000] DMAR: IOMMU enabled
[    0.102094] DMAR-IR: This system BIOS has enabled interrupt remapping
[    1.109864] DMAR: Host address width 39
[    1.109915] DMAR: DRHD base: 0x000000fbffe000 flags: 0x1
[    1.109984] DMAR: dmar0: reg_base_addr fbffe000 ver 1:0 cap c90780106f0462 ecap f020fe
[    1.110054] DMAR: RMRR base: 0x000000000ec000 end: 0x000000000effff
[    1.110108] DMAR: RMRR base: 0x0000009f7da000 end: 0x0000009f7d9fff
[    1.110163] DMAR: ATSR flags: 0x0
[    1.110410] DMAR: dmar0: Using Queued invalidation
[    1.110475] DMAR: Setting RMRR:
[    1.110545] DMAR: Setting identity map for device 0000:00:1a.0 [0x9f7da000 - 0x9f7d9fff]
[    1.118284] DMAR: Mapping reserved region failed
[    1.118350] DMAR: Setting identity map for device 0000:00:1a.1 [0x9f7da000 - 0x9f7d9fff]
[    1.120550] DMAR: Mapping reserved region failed
[    1.120611] DMAR: Setting identity map for device 0000:00:1a.2 [0x9f7da000 - 0x9f7d9fff]
[    1.122815] DMAR: Mapping reserved region failed
[    1.122876] DMAR: Setting identity map for device 0000:00:1a.7 [0x9f7da000 - 0x9f7d9fff]
[    1.127756] DMAR: Mapping reserved region failed
[    1.127818] DMAR: Setting identity map for device 0000:00:1d.0 [0x9f7da000 - 0x9f7d9fff]
[    1.130015] DMAR: Mapping reserved region failed
[    1.130076] DMAR: Setting identity map for device 0000:00:1d.1 [0x9f7da000 - 0x9f7d9fff]
[    1.137610] DMAR: Mapping reserved region failed
[    1.137676] DMAR: Setting identity map for device 0000:00:1d.2 [0x9f7da000 - 0x9f7d9fff]
[    1.139892] DMAR: Mapping reserved region failed
[    1.139953] DMAR: Setting identity map for device 0000:00:1d.7 [0x9f7da000 - 0x9f7d9fff]
[    1.142154] DMAR: Mapping reserved region failed
[    1.142215] DMAR: Setting identity map for device 0000:00:1a.0 [0xec000 - 0xeffff]
[    1.142312] DMAR: Setting identity map for device 0000:00:1a.1 [0xec000 - 0xeffff]
[    1.142408] DMAR: Setting identity map for device 0000:00:1a.2 [0xec000 - 0xeffff]
[    1.142503] DMAR: Setting identity map for device 0000:00:1a.7 [0xec000 - 0xeffff]
[    1.142598] DMAR: Setting identity map for device 0000:00:1d.0 [0xec000 - 0xeffff]
[    1.142693] DMAR: Setting identity map for device 0000:00:1d.1 [0xec000 - 0xeffff]
[    1.145483] DMAR: Setting identity map for device 0000:00:1d.2 [0xec000 - 0xeffff]
[    1.145579] DMAR: Setting identity map for device 0000:00:1d.7 [0xec000 - 0xeffff]
[    1.145668] DMAR: Prepare 0-16MiB unity mapping for LPC
[    1.145730] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    1.145958] DMAR: Intel(R) Virtualization Technology for Directed I/O

===================================

# cat /etc/modprobe.d/vfio_pci.conf
options vfio_pci disable_vga=1
install vfio_pci /root/sbin/vfio-pci-override-vga.sh

====================================

# cat /root/sbin/vfio-pci-override-vga.sh
#!/bin/sh

for i in $(find /sys/devices/pci* -name boot_vga); do
        if [ $(cat $i) -eq 0 ]; then
                GPU=$(dirname $i)
                AUDIO=$(echo $GPU | sed -e "s/0$/1/")
                echo "vfio-pci" > $GPU/driver_override
                if [ -d $AUDIO ]; then
                        echo "vfio-pci" > $AUDIO/driver_override
                fi
        fi
        if [ $(cat $i) -eq 1 ]; then
                GPU=$(dirname $i)
                AUDIO=$(echo $GPU | sed -e "s/0$/1/")
                echo "nouveau" > $GPU/driver_override
                if [ -d $AUDIO ]; then
                        echo "nouveau" > $AUDIO/driver_override
                fi
        fi
done

# All rights reserved :p

=======================================

# lspci -v |grep -i VGA
03:00.0 VGA compatible controller: NVIDIA Corporation GK106 [GeForce GTX 660] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: eVga.com. Corp. Device 2662
        Subsystem: eVga.com. Corp. Device 2662
04:00.0 VGA compatible controller: NVIDIA Corporation GK106 [GeForce GTX 660] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: eVga.com. Corp. Device 2662
        Subsystem: eVga.com. Corp. Device 2662
05:00.0 VGA compatible controller: NVIDIA Corporation GK106 [GeForce GTX 660] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: eVga.com. Corp. Device 2662
        Subsystem: eVga.com. Corp. Device 2662

========================================

# cat /sys/devices/pci0000:00/0000:00:03.0/0000:03:00.0/driver_override
nouveau
# cat /sys/devices/pci0000:00/0000:00:07.0/0000:04:00.0/driver_override
vfio-pci

=========================================

Thanks in advance, and I appreciate all the help to date,
Brian