[vfio-users] Poor performance with nvidia GTX 980
Georgios Kourachanis
geo.kourachanis at gmail.com
Mon Nov 2 12:04:30 UTC 2015
Hello Eddie,
Thanks for answering, though:
What you suggest me to do, I've already done it with this way:
/usr/local/bin/qemu-system-x86_64.hv:
#!/bin/sh
exec /usr/bin/qemu-system-x86_64 `echo "\$@" | \
sed 's|hv_time|hv_time,hv_vendor_id=GoobyPLS|g'
and by changing the emulator qemu to this line:
<emulator>/usr/local/bin/qemu-system-x86_64.hv</emulator>
I'm just giving the ID "GoobyPLS" to the vendor. I'll try without a
vendor name to see if it changes anything.
Also, I'm using the qemu git version "r41983.g3a958f5" so it already
contains the patch that helps us use the lines above.
On 02/11/2015 03:53 πμ, Eddie Yen wrote:
> According from AW's blog:
> " For this step we again need to run virsh edit on the VM. Within
> the<features> section, remove everything between the <hyperv> tags,
> including the tags themselves.0"
> and
> "Additionally, within the <clock> tag, find the timer named
> hypervclock, remove the line containing this tag completely. Save and
> exit the edit session."
>
> I found that these still exist in your XML file, so try to do this:
>
> 1. Remove these tags.
> 2. Re-compile QEMU and re-install it with this patch
> http://www.spinics.net/lists/kvm/msg121742.html
> 3. Add these tags between </devices> and </domain>
>
> <qemu:commandline>
> <qemu:arg value='-cpu'/>
> <qemu:arg
> value='host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,kvm=off,hv)vendor_id='/>
> </qemu:commandline>
>
> I'm using GTX980, too. Before that, I got poor 3D performance in
> Windows 10, after this patch and edition, I got performance back.
>
> 2015-11-02 1:43 GMT+08:00 Georgios Kourachanis
> <geo.kourachanis at gmail.com <mailto:geo.kourachanis at gmail.com>>:
>
> Hello all,
>
> I had been using Xen with some AMD GPUs for almost 2 years till
> about June 2015, when I then found out that KVM and libvirt could
> do the same stuff I was interested in with nvidia GPUs, too. I
> needed the CUDA cores so I did change to an ASUS GTX 980 Strix.
> But unfortunatelly, I don't get any good performance output from
> it. On native windows 7/10 installation it's a beast though.
> I also have an AMD R7 250 which works great with KVM. But let's
> not mess with it.
>
> Let me get to the point:
>
> I have no problems as for the installation of Windows or OVMF or
> passing-through or anything else. The only problem is the GTX980's
> performance.
> The performance had a significant boost when I used the latest
> qemu branch with the hyper-v trick, but still, not getting what
> many people seem to claim in this mailing list "almost-native"
> (even with nvidia GPUs).
>
>
> Here is my system's specs:
>
> Archlinux with 4.1.6-1-vfio (with the ACS patch ALONE)
> Intel Core i73770 ( I use the igpu for the archlinux)
> 24GiB RAM
> ASUS GTX 980 Strix
> Sapphire R7 250
> ------------------------------------------------------------------------
> lspci (only pass-through'd stuff):
>
> 01:00.0 VGA compatible controller: NVIDIA Corporation GM204
> [GeForce GTX 980] (rev a1)
> Subsystem: ASUSTeK Computer Inc. Device 8518
> Kernel driver in use: vfio-pci
> Kernel modules: nouveau
> 01:00.1 Audio device: NVIDIA Corporation GM204 High Definition
> Audio Controller (rev a1)
> Subsystem: ASUSTeK Computer Inc. Device 8518
> Kernel driver in use: vfio-pci
> Kernel modules: snd_hda_intel
> 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc.
> [AMD/ATI] Oland PRO [Radeon R7 240/340]
> Subsystem: PC Partner Limited / Sapphire Technology Device
> e266
> Kernel modules: radeon
> 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape
> Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> Subsystem: PC Partner Limited / Sapphire Technology Device
> aab0
> Kernel driver in use: snd_hda_intel
> Kernel modules: snd_hda_intel
> 08:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed
> USB Host Controller
> Subsystem: ASRock Incorporation Motherboard
> Kernel driver in use: vfio-pci
> Kernel modules: xhci_pci
> ------------------------------------------------------------------------
> booting lines:
>
> linux /boot/vmlinuz-linux-vfio root=UUID=XXXX rw intel_iommu=on
> pcie_acs_override=downstream isolcpus=2-3,6-7 nohz_full=2-3,6-7
> initrd /boot/intel-ucode.img /boot/initramfs-linux-vfio.img
> ------------------------------------------------------------------------
> /etc/fstab:|
>
> hugetlbfs /hugepages hugetlbfs defaults 0 0|
> ------------------------------------------------------------------------
> /etc/sysctl.d/40-hugepage.conf:
>
> vm.nr_hugepages = 8000
> ------------------------------------------------------------------------
> /etc/modules-load.d/vfio.conf:
>
> kvm
> kvm-intel
> vfio
> vfio-pci
> vfio_iommu_type1
> vfio_virqfd
> ------------------------------------------------------------------------
> /etc/modprobe.d/kvm.conf:
>
> options kvm ignore_msrs=1
> ------------------------------------------------------------------------
> /etc/modprobe.d/kvm-intel.conf:
>
> options kvm-intel nested=1
> ------------------------------------------------------------------------
> /etc/modprobe.d/vfio_iommu_type1.conf:
>
> options vfio_iommu_type1 allow_unsafe_interrupts=0
> ------------------------------------------------------------------------
> /etc/modprobe.d/vfio-pci.conf:
>
> options vfio-pci ids=10de:13c0,10de:0fbb,1002:6613,1002:aab0,1b21:1042
> ------------------------------------------------------------------------
>
> And the virsh xml:
>
> <domain type='kvm'>
> <name>windows_10</name>
> <uuid>63045df8-c782-4cfd-abc7-a3598826ae83</uuid>
> <memory unit='KiB'>6553600</memory>
> <currentMemory unit='KiB'>6553600</currentMemory>
> <memoryBacking>
> <hugepages/>
> </memoryBacking>
> <vcpu placement='static'>4</vcpu>
> <cputune>
> <vcpupin vcpu='0' cpuset='2'/>
> <vcpupin vcpu='1' cpuset='3'/>
> <vcpupin vcpu='2' cpuset='6'/>
> <vcpupin vcpu='3' cpuset='7'/>
> </cputune>
> <os>
> <type arch='x86_64' machine='pc-i440fx-2.4'>hvm</type>
> <loader readonly='yes'
> type='pflash'>/usr/local/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
> <nvram>/var/lib/libvirt/qemu/nvram/windows_nvidia_VARS.fd</nvram>
> </os>
> <features>
> <acpi/>
> <apic/>
> <pae/>
> <hyperv>
> <relaxed state='on'/>
> <vapic state='on'/>
> <spinlocks state='on' retries='8191'/>
> </hyperv>
> <kvm>
> <hidden state='on'/>
> </kvm>
> <vmport state='off'/>
> </features>
> <cpu mode='host-passthrough'>
> <topology sockets='1' cores='4' threads='1'/>
> </cpu>
> <clock offset='localtime'>
> <timer name='rtc' tickpolicy='catchup'/>
> <timer name='pit' tickpolicy='delay'/>
> <timer name='hpet' present='no'/>
> <timer name='hypervclock' present='yes'/>
> </clock>
> <on_poweroff>destroy</on_poweroff>
> <on_reboot>restart</on_reboot>
> <on_crash>restart</on_crash>
> <pm>
> <suspend-to-mem enabled='no'/>
> <suspend-to-disk enabled='no'/>
> </pm>
> <devices>
> <emulator>/usr/local/bin/qemu-system-x86_64.hv</emulator>
> <disk type='block' device='disk'>
> <driver name='qemu' type='raw' cache='none'/>
> <source dev='/dev/mapper/vg_ssd-lv_kvm_NVIDIA'/>
> <target dev='sda' bus='scsi'/>
> <boot order='1'/>
> <address type='drive' controller='0' bus='0' target='0'
> unit='0'/>
> </disk>
> <disk type='block' device='disk'>
> <driver name='qemu' type='raw' cache='none'/>
> <source dev='/dev/mapper/vg_raid5-lv_xen_ntfs_files'/>
> <target dev='sdb' bus='scsi'/>
> <address type='drive' controller='0' bus='0' target='0'
> unit='1'/>
> </disk>
> <controller type='usb' index='0'>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> function='0x2'/>
> </controller>
> <controller type='pci' index='0' model='pci-root'/>
> <controller type='scsi' index='0' model='virtio-scsi'>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> function='0x0'/>
> </controller>
> <interface type='bridge'>
> <mac address='52:54:00:e9:85:8f'/>
> <source bridge='xenbr0'/>
> <model type='e1000'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
> </interface>
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0x01' slot='0x00'
> function='0x0'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a'
> function='0x0' multifunction='on'/>
> </hostdev>
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0x01' slot='0x00'
> function='0x1'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a'
> function='0x1'/>
> </hostdev>
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0x08' slot='0x00'
> function='0x0'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x08'
> function='0x0'/>
> </hostdev>
> <memballoon model='virtio'>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
> function='0x0'/>
> </memballoon>
> </devices>
> </domain>
> ------------------------------------------------------------------------
>
> /usr/local/bin/qemu-system-x86_64.hv:
> #!/bin/sh
> exec /usr/bin/qemu-system-x86_64 `echo "\$@" | \
> sed 's|hv_time|hv_time,hv_vendor_id=GoobyPLS|g'
>
>
>
> And some notes:
>
> 1) Using "<topology sockets='1' cores='4' threads='1'/>" instead
> of "<topology sockets='1' cores='2' threads='2'/>" provided about
> 2% boost in GPU performance. No change in RAM or CPU tests. I've
> tested with the passmark.
>
> 2) I tried using the emulatorpin method Alex says on a mail here
> on vfio-users, but I didn't notice any changed in GPU performance.
> I didn't test it on the CPU side though.
>
> 3) The main problem of the performance lack is that a specific
> game that I've been playing isn't quite playable. That game has
> been mentioned before here on the list, it's Tera (european
> version (gameforge), although american version(enmasse) has
> exactly the same performance).
>
> 4) Every other game I managed to play is quite playable, though I
> haven't tested them to see if they run on native speeds.
>
>
> I'd really want some help on this matter, I really want to make my
> server run this VM with the nvidia GPU. I hate dual booting
> Windows >_>
>
>
> Thanks!
>
> _______________________________________________
> vfio-users mailing list
> vfio-users at redhat.com <mailto:vfio-users at redhat.com>
> https://www.redhat.com/mailman/listinfo/vfio-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20151102/c56ca6f5/attachment.htm>
More information about the vfio-users
mailing list