[vfio-users] vfio help needed (nvidia GTX 750 Ti persistent code 43)
Mogliii
mogliii at gmx.net
Wed Sep 16 14:10:28 UTC 2015
Hello,
I have been trying to get vga passthrough working for quite some time
now. I even ordered a new graphics card just for that purpose.
I explain my setup and configuration below.
My current problem: I can install Windows 8.1 with uefi (gpt partition
layout) and using virt-manager. Then, I shutdown the VM, remove the
spice and graphics, and add the pcie MSI GTX 750 Ti Gaming (it has a
legacy and uefi bios; uefi bios selected by physical switch). When I
then boot-up the VM, I see the OVMF bios on the physical screen attached
to the GTX750 Ti (sometimes I get graphics glitches i.e. all white) and
Windows 8.1 boots into 800x600. I see that a Nvidia GTX750Ti is
recognized in the device manager. After installing driver (I tried
335.23, 340.52, 341.44 and most recent) the gpu is shown with yellow
exclamation mark with error code 43 and I have to restart. After
restart, I see again the bios and the boot splash, but then the screen
goes into power save mode because it does not get any signal just when
the login screen is supposed to appear.
I can login via vnc and see that the gpu has error code 43.
De-installing the driver and restarting I will go back to a 800x600
desktop on my physical screen.
Any help with my setup would be immensely appreciated. I have another
spare ASUS HD6570, but it does not have uefi bios. I am not sure if I
would need i915 patch in that case? Does that patch only apply to Intel
CPUs?
Is the 00:02.2 pci device the problem?
I would really appreciate any hint as I currently exhausted all options
I see and I would like to avoid spending more money on video cards that
I'm not sure I can even use.
==========================
SETUP
=========
[[[ 1. Physical setup
Asus A88x-PRO
AMD A10-7850K (on-board GPU used for linux host with radeon driver
(fglrx not installed))
12 GB memory
MSI GTX 750 Ti Gaming (bios can be switched between legacy/uefi by
hardware switch; using uefi)
GPU is installed in middle pcie 16x slot (black, not golden) due to
iommu grouping (see below)
[[[ 2. OS
UBuntu 15.04 64 bit, unpatched
$ uname -a
Linux host 3.19.0-28-generic #30-Ubuntu SMP Mon Aug 31 15:52:51 UTC 2015
x86_64 x86_64 x86_64 GNU/Linux
[[[ 3. Relevant packages (not sure if these are all)
virt-manager 1:1.2.1-1~getdeb1 (get deb ppa)
qemu 1:2.2+dfsg-5expubuntu9.4
libvirt-bin 1.2.12-0ubuntu14.2
qemu-kvm 1:2.2+dfsg-5expubuntu9.4
[[[ 4. Bios settings
Set to boot from on board GPU. AMD-V and IOMMU activated. Legacy boot
for ubuntu host
[[[ 5. iommu grouping (iommu.sh output)
### Group 2 ###
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1424
00:02.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1425
01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce
GTX 750 Ti] (rev a2)
01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)
[[[ 6. pci identifier
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM107
[GeForce GTX 750 Ti] [10de:1380] (rev a2)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3102]
Kernel driver in use: pci-stub
01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbc] (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3102]
Kernel driver in use: pci-stub
[[[ 7. /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.19.0-28-generic root=/dev/mapper/main-root ro
rootflags=subvol=@ quiet splash nomdmonddf nomdmonisw amd_iommu=on
iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1
pci-stub.ids=10de:0fbc,10de:1380 vt.handoff=7
[[[ 8. /etc/initramfs-tools/modules
pci_stub ids=10de:0fbc,10de:1380
[[[ 9. /etc/modules
lp
rtc
pci_stub
vfio
vfio_iommu_type1
vfio_pci
kvm
kvm_amd
[[[ 10. # dmseg | grep pci-stub
[ 1.263269] pci-stub: add 10DE:0FBC sub=FFFFFFFF:FFFFFFFF
cls=00000000/00000000
[ 1.263287] pci-stub 0000:01:00.1: claimed by stub
[ 1.263296] pci-stub: add 10DE:1380 sub=FFFFFFFF:FFFFFFFF
cls=00000000/00000000
[ 1.263304] pci-stub 0000:01:00.0: claimed by stub
[[[ 11. /usr/windows.sh (executed before running vm. Afterwards lspci
-nnk shows "vfio-pci" as kernel driver in use for 01:00.0 and 01:00.1)
#!/bin/bash
configfile=/etc/vfio_pci1.cfg
vfiobind() {
dev="$1"
vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
device=$(cat /sys/bus/pci/devices/$dev/device)
if [ -e /sys/bus/pci/devices/$dev/driver ]; then
echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
fi
echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
}
modprobe vfio-pci
cat $configfile | while read line;do
echo $line | grep ^# >/dev/null 2>&1 && continue
vfiobind $line
done
exit 0
[[[ 12. /etc/vfio_pci1.cfg
0000:01:00.0
0000:01:00.1
[[[ 13. OVMF-bios
Got from sourceforge
(edk2.git-ovmf-x64-0-20150910.b1206.g5abc2a7.noarch.rpm) and extracted.
Then, made individual copy of OVMF-pure-efi.fd (md5
fbb13f2d4be2f9d72472628f885f5cb4 ) for VM.
[[[ 14. Created VM in virt-manager using the following configuration
(configuration shown after installation, i.e. no cdroms, but before
attaching pci)
<domain type='kvm'>
<name>win81</name>
<uuid>redacted</uuid>
<memory unit='KiB'>4194304</memory>
<currentMemory unit='KiB'>4194304</currentMemory>
<vcpu placement='static'>2</vcpu>
<os>
<type arch='x86_64' machine='pc-i440fx-utopic'>hvm</type>
<loader type='rom'>/usr/share/ovmf/OVMF-pure-efi_win81.fd</loader>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
<kvm>
<hidden state='on'/>
</kvm>
</features>
<cpu mode='host-passthrough'>
</cpu>
<clock offset='localtime'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source dev='/dev/main/Win7xen'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08'
function='0x0'/>
</disk>
<controller type='usb' index='0' model='ich9-ehci1'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x7'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci1'>
<master startport='0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x0' multifunction='on'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci2'>
<master startport='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x1'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci3'>
<master startport='4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x0'/>
</controller>
<interface type='bridge'>
<mac address='52:54:00:ab:b6:10'/>
<source bridge='br1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
</interface>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='spice' autoport='yes'/>
<video>
<model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/>
</video>
<hostdev mode='subsystem' type='usb' managed='yes'>
<source>
<vendor id='0x0603'/>
<product id='0x1605'/>
</source>
</hostdev>
<hostdev mode='subsystem' type='usb' managed='yes'>
<source>
<vendor id='0x05ac'/>
<product id='0x1006'/>
</source>
</hostdev>
<hostdev mode='subsystem' type='usb' managed='yes'>
<source>
<vendor id='0x05ac'/>
<product id='0x024f'/>
</source>
</hostdev>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07'
function='0x0'/>
</memballoon>
</devices>
</domain>
[[[ 15. After installation and installation of updates, shut down the VM
[[[ 16. Attached PCI devices and removed video and display spice. Below
is virsh configuration after attaching pcie passthrough.
<domain type='kvm'>
<name>win81</name>
<uuid>redacted</uuid>
<memory unit='KiB'>4194304</memory>
<currentMemory unit='KiB'>4194304</currentMemory>
<vcpu placement='static'>2</vcpu>
<os>
<type arch='x86_64' machine='pc-i440fx-utopic'>hvm</type>
<loader type='rom'>/usr/share/ovmf/OVMF-pure-efi_win81.fd</loader>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
<kvm>
<hidden state='on'/>
</kvm>
</features>
<cpu mode='host-passthrough'>
</cpu>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source dev='/dev/main/Win7xen'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08'
function='0x0'/>
</disk>
<controller type='usb' index='0' model='ich9-ehci1'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x7'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci1'>
<master startport='0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x0' multifunction='on'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci2'>
<master startport='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x1'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci3'>
<master startport='4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x0'/>
</controller>
<interface type='bridge'>
<mac address='52:54:00:ab:b6:10'/>
<source bridge='br1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
</interface>
<hostdev mode='subsystem' type='usb' managed='yes'>
<source>
<vendor id='0x0603'/>
<product id='0x1605'/>
</source>
</hostdev>
<hostdev mode='subsystem' type='usb' managed='yes'>
<source>
<vendor id='0x05ac'/>
<product id='0x1006'/>
</source>
</hostdev>
<hostdev mode='subsystem' type='usb' managed='yes'>
<source>
<vendor id='0x05ac'/>
<product id='0x024f'/>
</source>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09'
function='0x0'/>
</hostdev>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07'
function='0x0'/>
</memballoon>
</devices>
</domain>
[[[ 17. What else did I try
- Different nvidia drivers (see top of email)
- Attaching 00:02.2 as well, but cannot do vfio-pci assignement so failed
# virsh nodedev-list --tree
+- pci_0000_00_02_0
+- pci_0000_00_02_2
| |
| +- pci_0000_01_00_0
| +- pci_0000_01_00_1
|
+- pci_0000_00_03_0
# lspci -nnk (extract)
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device
[1022:1424]
00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device
[1022:1425]
Kernel driver in use: pcieport
- Reinstall the whole VM with seabios and using the GTX 750 Ti with
legacy bios (physical switch on card). But no picture was shown on
screen after attaching, not even the bios.
- Tried to use q35 emulation, but not successful. It does not work with
ovmf? I.e. I cannot use the qemu:cmd or ioh3420.
More information about the vfio-users
mailing list