[vfio-users] Poor performance with nvidia GTX 980
Eddie Yen
missile0407 at gmail.com
Mon Nov 2 01:53:56 UTC 2015
According from AW's blog:
" For this step we again need to run virsh edit on the VM. Within the
<features> section, remove everything between the <hyperv> tags, including
the tags themselves.0"
and
"Additionally, within the <clock> tag, find the timer named hypervclock,
remove the line containing this tag completely. Save and exit the edit
session."
I found that these still exist in your XML file, so try to do this:
1. Remove these tags.
2. Re-compile QEMU and re-install it with this patch
http://www.spinics.net/lists/kvm/msg121742.html
3. Add these tags between </devices> and </domain>
<qemu:commandline>
<qemu:arg value='-cpu'/>
<qemu:arg
value='host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,kvm=off,hv)vendor_id='/>
</qemu:commandline>
I'm using GTX980, too. Before that, I got poor 3D performance in Windows
10, after this patch and edition, I got performance back.
2015-11-02 1:43 GMT+08:00 Georgios Kourachanis <geo.kourachanis at gmail.com>:
> Hello all,
>
> I had been using Xen with some AMD GPUs for almost 2 years till about
> June 2015, when I then found out that KVM and libvirt could do the same
> stuff I was interested in with nvidia GPUs, too. I needed the CUDA cores so
> I did change to an ASUS GTX 980 Strix. But unfortunatelly, I don't get any
> good performance output from it. On native windows 7/10 installation it's a
> beast though.
> I also have an AMD R7 250 which works great with KVM. But let's not mess
> with it.
>
> Let me get to the point:
>
> I have no problems as for the installation of Windows or OVMF or
> passing-through or anything else. The only problem is the GTX980's
> performance.
> The performance had a significant boost when I used the latest qemu branch
> with the hyper-v trick, but still, not getting what many people seem to
> claim in this mailing list "almost-native" (even with nvidia GPUs).
>
>
> Here is my system's specs:
>
> Archlinux with 4.1.6-1-vfio (with the ACS patch ALONE)
> Intel Core i7 3770 ( I use the igpu for the archlinux)
> 24GiB RAM
> ASUS GTX 980 Strix
> Sapphire R7 250
> ------------------------------
> lspci (only pass-through'd stuff):
>
> 01:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX
> 980] (rev a1)
> Subsystem: ASUSTeK Computer Inc. Device 8518
> Kernel driver in use: vfio-pci
> Kernel modules: nouveau
> 01:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio
> Controller (rev a1)
> Subsystem: ASUSTeK Computer Inc. Device 8518
> Kernel driver in use: vfio-pci
> Kernel modules: snd_hda_intel
> 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI]
> Oland PRO [Radeon R7 240/340]
> Subsystem: PC Partner Limited / Sapphire Technology Device e266
> Kernel modules: radeon
> 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape
> Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> Subsystem: PC Partner Limited / Sapphire Technology Device aab0
> Kernel driver in use: snd_hda_intel
> Kernel modules: snd_hda_intel
> 08:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB
> Host Controller
> Subsystem: ASRock Incorporation Motherboard
> Kernel driver in use: vfio-pci
> Kernel modules: xhci_pci
> ------------------------------
> booting lines:
>
> linux /boot/vmlinuz-linux-vfio root=UUID=XXXX rw intel_iommu=on
> pcie_acs_override=downstream isolcpus=2-3,6-7 nohz_full=2-3,6-7
> initrd /boot/intel-ucode.img /boot/initramfs-linux-vfio.img
> ------------------------------
> /etc/fstab:
>
> hugetlbfs /hugepages hugetlbfs defaults 0 0
> ------------------------------
> /etc/sysctl.d/40-hugepage.conf:
>
> vm.nr_hugepages = 8000
> ------------------------------
> /etc/modules-load.d/vfio.conf:
>
> kvm
> kvm-intel
> vfio
> vfio-pci
> vfio_iommu_type1
> vfio_virqfd
> ------------------------------
> /etc/modprobe.d/kvm.conf:
>
> options kvm ignore_msrs=1
> ------------------------------
> /etc/modprobe.d/kvm-intel.conf:
>
> options kvm-intel nested=1
> ------------------------------
> /etc/modprobe.d/vfio_iommu_type1.conf:
>
> options vfio_iommu_type1 allow_unsafe_interrupts=0
> ------------------------------
> /etc/modprobe.d/vfio-pci.conf:
>
> options vfio-pci ids=10de:13c0,10de:0fbb,1002:6613,1002:aab0,1b21:1042
> ------------------------------
>
> And the virsh xml:
>
> <domain type='kvm'>
> <name>windows_10</name>
> <uuid>63045df8-c782-4cfd-abc7-a3598826ae83</uuid>
> <memory unit='KiB'>6553600</memory>
> <currentMemory unit='KiB'>6553600</currentMemory>
> <memoryBacking>
> <hugepages/>
> </memoryBacking>
> <vcpu placement='static'>4</vcpu>
> <cputune>
> <vcpupin vcpu='0' cpuset='2'/>
> <vcpupin vcpu='1' cpuset='3'/>
> <vcpupin vcpu='2' cpuset='6'/>
> <vcpupin vcpu='3' cpuset='7'/>
> </cputune>
> <os>
> <type arch='x86_64' machine='pc-i440fx-2.4'>hvm</type>
> <loader readonly='yes'
> type='pflash'>/usr/local/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
> <nvram>/var/lib/libvirt/qemu/nvram/windows_nvidia_VARS.fd</nvram>
> </os>
> <features>
> <acpi/>
> <apic/>
> <pae/>
> <hyperv>
> <relaxed state='on'/>
> <vapic state='on'/>
> <spinlocks state='on' retries='8191'/>
> </hyperv>
> <kvm>
> <hidden state='on'/>
> </kvm>
> <vmport state='off'/>
> </features>
> <cpu mode='host-passthrough'>
> <topology sockets='1' cores='4' threads='1'/>
> </cpu>
> <clock offset='localtime'>
> <timer name='rtc' tickpolicy='catchup'/>
> <timer name='pit' tickpolicy='delay'/>
> <timer name='hpet' present='no'/>
> <timer name='hypervclock' present='yes'/>
> </clock>
> <on_poweroff>destroy</on_poweroff>
> <on_reboot>restart</on_reboot>
> <on_crash>restart</on_crash>
> <pm>
> <suspend-to-mem enabled='no'/>
> <suspend-to-disk enabled='no'/>
> </pm>
> <devices>
> <emulator>/usr/local/bin/qemu-system-x86_64.hv</emulator>
> <disk type='block' device='disk'>
> <driver name='qemu' type='raw' cache='none'/>
> <source dev='/dev/mapper/vg_ssd-lv_kvm_NVIDIA'/>
> <target dev='sda' bus='scsi'/>
> <boot order='1'/>
> <address type='drive' controller='0' bus='0' target='0' unit='0'/>
> </disk>
> <disk type='block' device='disk'>
> <driver name='qemu' type='raw' cache='none'/>
> <source dev='/dev/mapper/vg_raid5-lv_xen_ntfs_files'/>
> <target dev='sdb' bus='scsi'/>
> <address type='drive' controller='0' bus='0' target='0' unit='1'/>
> </disk>
> <controller type='usb' index='0'>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> function='0x2'/>
> </controller>
> <controller type='pci' index='0' model='pci-root'/>
> <controller type='scsi' index='0' model='virtio-scsi'>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> function='0x0'/>
> </controller>
> <interface type='bridge'>
> <mac address='52:54:00:e9:85:8f'/>
> <source bridge='xenbr0'/>
> <model type='e1000'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
> </interface>
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a'
> function='0x0' multifunction='on'/>
> </hostdev>
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a'
> function='0x1'/>
> </hostdev>
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x08'
> function='0x0'/>
> </hostdev>
> <memballoon model='virtio'>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
> function='0x0'/>
> </memballoon>
> </devices>
> </domain>
> ------------------------------
>
> /usr/local/bin/qemu-system-x86_64.hv:
> #!/bin/sh
> exec /usr/bin/qemu-system-x86_64 `echo "\$@" | \
> sed 's|hv_time|hv_time,hv_vendor_id=GoobyPLS|g'
>
>
>
> And some notes:
>
> 1) Using "<topology sockets='1' cores='4' threads='1'/>" instead of "<topology
> sockets='1' cores='2' threads='2'/>" provided about 2% boost in GPU
> performance. No change in RAM or CPU tests. I've tested with the passmark.
>
> 2) I tried using the emulatorpin method Alex says on a mail here on
> vfio-users, but I didn't notice any changed in GPU performance. I didn't
> test it on the CPU side though.
>
> 3) The main problem of the performance lack is that a specific game that
> I've been playing isn't quite playable. That game has been mentioned before
> here on the list, it's Tera (european version (gameforge), although
> american version(enmasse) has exactly the same performance).
>
> 4) Every other game I managed to play is quite playable, though I haven't
> tested them to see if they run on native speeds.
>
>
> I'd really want some help on this matter, I really want to make my server
> run this VM with the nvidia GPU. I hate dual booting Windows >_>
>
>
> Thanks!
>
> _______________________________________________
> vfio-users mailing list
> vfio-users at redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20151102/b9f6123a/attachment.htm>
More information about the vfio-users
mailing list