[vfio-users] GPU Pass-through for Linux guest
Daniel Browne
jairuncaloth at gmail.com
Sun May 1 21:00:05 UTC 2016
I'm working on testing a setup where all of my 'desktops' are VMs, however
I am running into issues getting GPU pass-through functioning correctly on
a Linux guest.
Host OS is Ubuntu Server 16.04
Guest is Xubuntu 16.04
These errors in the guests xorg log show something isn't working correctly,
but I've been unable to identify what it might be.
$ grep EE /var/log/Xorg.0.log
[ 2.339] (EE) [drm] Failed to open DRM device for (null): -22
[ 2.346] (EE) Failed to initialize GLX extension (Compatible NVIDIA X
driver not found)
I installed the guest without pass-through configured using virt-manager.
After installation I assigned the host PCI devices for a GTX970 and
installed the official NVIDIA drivers using Ubuntu's additional drivers
utility.
For now I'm testing with the 970 because I know it works. I've been using
it for months for a Win7 VM on a Fedora22 workstation host in the current
hardware configuration.
GPU is bound to vfio-pci on the host
$ lspci -v
01:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX
970] (rev a1) (prog-if 00 [VGA controller])
Subsystem: Micro-Star International Co., Ltd. [MSI] GM204 [GeForce GTX 970]
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at ee000000 (32-bit, non-prefetchable) [size=16M]
Memory at d0000000 (64-bit, prefetchable) [size=256M]
Memory at e0000000 (64-bit, prefetchable) [size=32M]
I/O ports at e000 [size=128]
Expansion ROM at ef000000 [disabled] [size=512K]
Capabilities: <access denied>
Kernel driver in use: vfio-pci
Kernel modules: nvidiafb, nouveau
01:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio
Controller (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] GM204 High Definition
Audio Controller
Flags: bus master, fast devsel, latency 0, IRQ 17
Memory at ef080000 (32-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
The only errors I see on the host while loading the VM is this in dmesg:
These were showing up before I added the PCI devices.
[51487.402933] kvm [4472]: vcpu2 unhandled rdmsr: 0x34
[51487.403976] kvm [4472]: vcpu2 unhandled rdmsr: 0x606
[51488.144792] kvm [4472]: vcpu0 unhandled rdmsr: 0x611
[51488.145794] kvm [4472]: vcpu0 unhandled rdmsr: 0x639
[51488.146767] kvm [4472]: vcpu0 unhandled rdmsr: 0x641
[51488.147761] kvm [4472]: vcpu0 unhandled rdmsr: 0x619
Here's the XML for the domain
<domain type='kvm' id='10'>
<name>Xubuntu16.04</name>
<uuid>9dc39e2e-f56e-4b0e-b65c-54d778db472c</uuid>
<memory unit='KiB'>16777216</memory>
<currentMemory unit='KiB'>16777216</currentMemory>
<vcpu placement='static'>4</vcpu>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64' machine='pc-i440fx-wily'>hvm</type>
<loader readonly='yes'
type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
<nvram>/var/lib/libvirt/qemu/nvram/Xubuntu16.04_VARS.fd</nvram>
</os>
<features>
<acpi/>
<apic/>
<pae/>
<kvm>
<hidden state='on'/>
</kvm>
<vmport state='off'/>
</features>
<cpu mode='host-model'>
<model fallback='allow'/>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/kvm-spice</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='raw'/>
<source file='/home/dbrowne/storage/ssd/xubuntu1604.img'/>
<backingStore/>
<target dev='vda' bus='virtio'/>
<boot order='1'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07'
function='0x0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<backingStore/>
<target dev='hda' bus='ide'/>
<readonly/>
<alias name='ide0-0-0'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<controller type='usb' index='0' model='ich9-ehci1'>
<alias name='usb'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x7'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci1'>
<alias name='usb'/>
<master startport='0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x0' multifunction='on'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci2'>
<alias name='usb'/>
<master startport='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x1'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci3'>
<alias name='usb'/>
<master startport='4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'>
<alias name='pci.0'/>
</controller>
<controller type='ide' index='0'>
<alias name='ide'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x1'/>
</controller>
<controller type='virtio-serial' index='0'>
<alias name='virtio-serial0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x0'/>
</controller>
<interface type='network'>
<mac address='52:54:00:92:7c:cf'/>
<source network='default' bridge='virbr0'/>
<target dev='vnet0'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
</interface>
<serial type='pty'>
<source path='/dev/pts/3'/>
<target port='0'/>
<alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/3'>
<source path='/dev/pts/3'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<channel type='spicevmc'>
<target type='virtio' name='com.redhat.spice.0' state='disconnected'/>
<alias name='channel0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<input type='tablet' bus='usb'>
<alias name='input0'/>
</input>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='spice' port='5900' autoport='yes' listen='0.0.0.0'
keymap='en-us'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<sound model='ich6'>
<alias name='sound0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
</sound>
<video>
<model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/>
</video>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</source>
<alias name='hostdev0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a'
function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
</source>
<alias name='hostdev1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0b'
function='0x0'/>
</hostdev>
<redirdev bus='usb' type='spicevmc'>
<alias name='redir0'/>
</redirdev>
<redirdev bus='usb' type='spicevmc'>
<alias name='redir1'/>
</redirdev>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08'
function='0x0'/>
</memballoon>
<rng model='virtio'>
<backend model='random'>/dev/random</backend>
<alias name='rng0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09'
function='0x0'/>
</rng>
</devices>
<seclabel type='dynamic' model='apparmor' relabel='yes'>
<label>libvirt-9dc39e2e-f56e-4b0e-b65c-54d778db472c</label>
<imagelabel>libvirt-9dc39e2e-f56e-4b0e-b65c-54d778db472c</imagelabel>
</seclabel>
</domain>
On the guest, the card shows up and nvidia kernel module loads, however the
display is not listed in xrandr or in the xfce gui display configuration.
Errors in the guest xorg log.
$ grep EE /var/log/Xorg.0.log
[ 2.339] (EE) [drm] Failed to open DRM device for (null): -22
[ 2.346] (EE) Failed to initialize GLX extension (Compatible NVIDIA X
driver not found)
full xorg log http://paste.ubuntu.com/16181913/
00:0a.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX
970] (rev a1) (prog-if 00 [VGA controller])
Subsystem: Micro-Star International Co., Ltd. [MSI] GM204 [GeForce GTX 970]
Physical Slot: 10
Flags: bus master, fast devsel, latency 0, IRQ 11
Memory at da000000 (32-bit, non-prefetchable) [size=16M]
Memory at c0000000 (64-bit, prefetchable) [size=256M]
Memory at d8000000 (64-bit, prefetchable) [size=32M]
I/O ports at c000 [size=128]
[virtual] Expansion ROM at db080000 [disabled] [size=512K]
Capabilities: <access denied>
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_361
00:0b.0 Audio device: NVIDIA Corporation GM204 High Definition Audio
Controller (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] GM204 High Definition
Audio Controller
Physical Slot: 11
Flags: bus master, fast devsel, latency 0, IRQ 10
Memory at db000000 (32-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
$ lsmod | grep nvidia
nvidia_uvm 696320 0
nvidia_modeset 745472 0
nvidia 10076160 2 nvidia_modeset,nvidia_uvm
drm 360448 7 qxl,ttm,drm_kms_helper,nvidia
$ xrandr
Screen 0: minimum 320 x 200, current 1280 x 800, maximum 8192 x 8192
Virtual-0 connected 1280x800+0+0 0mm x 0mm
1920x871 59.95 +
1920x1200 59.88
1920x1080 59.96
1600x1200 59.87
1680x1050 59.95
1400x1050 59.98
1280x1024 59.89
1440x900 59.89
1280x960 59.94
1280x854 59.89
1280x800 59.81*
1280x720 59.86
1152x768 59.78
1024x768 59.92
800x600 59.86
848x480 59.66
720x480 59.71
640x480 59.38
Virtual-1 disconnected
Virtual-2 disconnected
Virtual-3 disconnected
I didn't notice anything in here that looked useful - guest dmesg -
http://paste.ubuntu.com/16181879/
Thank you all for your time, any help would be appreciated.
-Daniel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160501/b87c9ff8/attachment.htm>
More information about the vfio-users
mailing list