[vfio-users] Poor performance with nvidia GTX 980
Georgios Kourachanis
geo.kourachanis at gmail.com
Mon Nov 2 12:47:57 UTC 2015
It's the same thing, either by adding them with qemuarguments, or with
the wrapper.
The thing is to use the hyper-v functions. That's what the hyper-v
vendor-id patch has given to us. The ability of hidding the hyper-v
functions from nvidia GPUs so that we can use them!
Also, I've tried with a null name for the vendor-id, I got the same
performance.
The nvidia drivers I'm currently using are 358.50
Moreover, could you suggest me a nice software to test the VM's
performance in general? I don't really like passmark.
On 02/11/2015 02:11 μμ, Eddie Yen wrote:
> OK, but I still suggest that remove Hyper-V function tags in your XML.
> Because we don't know about what new tricks inside the driver that
> NVIDIA wants to "surprise" us.
>
> For me, my GTX980 works well by using upon edits. But I'm using 4820K
> which didn't need ACS patch and wo intel graphic.
> So I'm not sure it may cause by patch or sth.
>
> 2015-11-02 20:04 GMT+08:00 Georgios Kourachanis
> <geo.kourachanis at gmail.com <mailto:geo.kourachanis at gmail.com>>:
>
> Hello Eddie,
>
> Thanks for answering, though:
>
> What you suggest me to do, I've already done it with this way:
>
> /usr/local/bin/qemu-system-x86_64.hv:
> #!/bin/sh
> exec /usr/bin/qemu-system-x86_64 `echo "\$@" | \
> sed 's|hv_time|hv_time,hv_vendor_id=GoobyPLS|g'
>
>
> and by changing the emulator qemu to this line:
>
> <emulator>/usr/local/bin/qemu-system-x86_64.hv</emulator>
>
> I'm just giving the ID "GoobyPLS" to the vendor. I'll try without
> a vendor name to see if it changes anything.
>
> Also, I'm using the qemu git version "r41983.g3a958f5" so it
> already contains the patch that helps us use the lines above.
>
>
>
>
>
> On 02/11/2015 03:53 πμ, Eddie Yen wrote:
>> According from AW's blog:
>> " For this step we again need to run virsh edit on the VM.
>> Within the<features> section, remove everything between the
>> <hyperv> tags, including the tags themselves.0"
>> and
>> "Additionally, within the <clock> tag, find the timer named
>> hypervclock, remove the line containing this tag completely.
>> Save and exit the edit session."
>>
>> I found that these still exist in your XML file, so try to do this:
>>
>> 1. Remove these tags.
>> 2. Re-compile QEMU and re-install it with this patch
>> http://www.spinics.net/lists/kvm/msg121742.html
>> 3. Add these tags between </devices> and </domain>
>>
>> <qemu:commandline>
>> <qemu:arg value='-cpu'/>
>> <qemu:arg
>> value='host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,kvm=off,hv)vendor_id='/>
>> </qemu:commandline>
>>
>> I'm using GTX980, too. Before that, I got poor 3D performance in
>> Windows 10, after this patch and edition, I got performance back.
>>
>> 2015-11-02 1:43 GMT+08:00 Georgios Kourachanis
>> <geo.kourachanis at gmail.com <mailto:geo.kourachanis at gmail.com>>:
>>
>> Hello all,
>>
>> I had been using Xen with some AMD GPUs for almost 2 years
>> till about June 2015, when I then found out that KVM and
>> libvirt could do the same stuff I was interested in with
>> nvidia GPUs, too. I needed the CUDA cores so I did change to
>> an ASUS GTX 980 Strix. But unfortunatelly, I don't get any
>> good performance output from it. On native windows 7/10
>> installation it's a beast though.
>> I also have an AMD R7 250 which works great with KVM. But
>> let's not mess with it.
>>
>> Let me get to the point:
>>
>> I have no problems as for the installation of Windows or OVMF
>> or passing-through or anything else. The only problem is the
>> GTX980's performance.
>> The performance had a significant boost when I used the
>> latest qemu branch with the hyper-v trick, but still, not
>> getting what many people seem to claim in this mailing list
>> "almost-native" (even with nvidia GPUs).
>>
>>
>> Here is my system's specs:
>>
>> Archlinux with 4.1.6-1-vfio (with the ACS patch ALONE)
>> Intel Core i73770 ( I use the igpu for the archlinux)
>> 24GiB RAM
>> ASUS GTX 980 Strix
>> Sapphire R7 250
>> ------------------------------------------------------------------------
>> lspci (only pass-through'd stuff):
>>
>> 01:00.0 VGA compatible controller: NVIDIA Corporation GM204
>> [GeForce GTX 980] (rev a1)
>> Subsystem: ASUSTeK Computer Inc. Device 8518
>> Kernel driver in use: vfio-pci
>> Kernel modules: nouveau
>> 01:00.1 Audio device: NVIDIA Corporation GM204 High
>> Definition Audio Controller (rev a1)
>> Subsystem: ASUSTeK Computer Inc. Device 8518
>> Kernel driver in use: vfio-pci
>> Kernel modules: snd_hda_intel
>> 02:00.0 VGA compatible controller: Advanced Micro Devices,
>> Inc. [AMD/ATI] Oland PRO [Radeon R7 240/340]
>> Subsystem: PC Partner Limited / Sapphire Technology
>> Device e266
>> Kernel modules: radeon
>> 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI]
>> Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
>> Subsystem: PC Partner Limited / Sapphire Technology
>> Device aab0
>> Kernel driver in use: snd_hda_intel
>> Kernel modules: snd_hda_intel
>> 08:00.0 USB controller: ASMedia Technology Inc. ASM1042
>> SuperSpeed USB Host Controller
>> Subsystem: ASRock Incorporation Motherboard
>> Kernel driver in use: vfio-pci
>> Kernel modules: xhci_pci
>> ------------------------------------------------------------------------
>> booting lines:
>>
>> linux /boot/vmlinuz-linux-vfio root=UUID=XXXX rw
>> intel_iommu=on pcie_acs_override=downstream isolcpus=2-3,6-7
>> nohz_full=2-3,6-7
>> initrd /boot/intel-ucode.img /boot/initramfs-linux-vfio.img
>> ------------------------------------------------------------------------
>> /etc/fstab:|
>>
>> hugetlbfs /hugepages hugetlbfs defaults 0 0|
>> ------------------------------------------------------------------------
>> /etc/sysctl.d/40-hugepage.conf:
>>
>> vm.nr_hugepages = 8000
>> ------------------------------------------------------------------------
>> /etc/modules-load.d/vfio.conf:
>>
>> kvm
>> kvm-intel
>> vfio
>> vfio-pci
>> vfio_iommu_type1
>> vfio_virqfd
>> ------------------------------------------------------------------------
>> /etc/modprobe.d/kvm.conf:
>>
>> options kvm ignore_msrs=1
>> ------------------------------------------------------------------------
>> /etc/modprobe.d/kvm-intel.conf:
>>
>> options kvm-intel nested=1
>> ------------------------------------------------------------------------
>> /etc/modprobe.d/vfio_iommu_type1.conf:
>>
>> options vfio_iommu_type1 allow_unsafe_interrupts=0
>> ------------------------------------------------------------------------
>> /etc/modprobe.d/vfio-pci.conf:
>>
>> options vfio-pci
>> ids=10de:13c0,10de:0fbb,1002:6613,1002:aab0,1b21:1042
>> ------------------------------------------------------------------------
>>
>> And the virsh xml:
>>
>> <domain type='kvm'>
>> <name>windows_10</name>
>> <uuid>63045df8-c782-4cfd-abc7-a3598826ae83</uuid>
>> <memory unit='KiB'>6553600</memory>
>> <currentMemory unit='KiB'>6553600</currentMemory>
>> <memoryBacking>
>> <hugepages/>
>> </memoryBacking>
>> <vcpu placement='static'>4</vcpu>
>> <cputune>
>> <vcpupin vcpu='0' cpuset='2'/>
>> <vcpupin vcpu='1' cpuset='3'/>
>> <vcpupin vcpu='2' cpuset='6'/>
>> <vcpupin vcpu='3' cpuset='7'/>
>> </cputune>
>> <os>
>> <type arch='x86_64' machine='pc-i440fx-2.4'>hvm</type>
>> <loader readonly='yes'
>> type='pflash'>/usr/local/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
>> <nvram>/var/lib/libvirt/qemu/nvram/windows_nvidia_VARS.fd</nvram>
>> </os>
>> <features>
>> <acpi/>
>> <apic/>
>> <pae/>
>> <hyperv>
>> <relaxed state='on'/>
>> <vapic state='on'/>
>> <spinlocks state='on' retries='8191'/>
>> </hyperv>
>> <kvm>
>> <hidden state='on'/>
>> </kvm>
>> <vmport state='off'/>
>> </features>
>> <cpu mode='host-passthrough'>
>> <topology sockets='1' cores='4' threads='1'/>
>> </cpu>
>> <clock offset='localtime'>
>> <timer name='rtc' tickpolicy='catchup'/>
>> <timer name='pit' tickpolicy='delay'/>
>> <timer name='hpet' present='no'/>
>> <timer name='hypervclock' present='yes'/>
>> </clock>
>> <on_poweroff>destroy</on_poweroff>
>> <on_reboot>restart</on_reboot>
>> <on_crash>restart</on_crash>
>> <pm>
>> <suspend-to-mem enabled='no'/>
>> <suspend-to-disk enabled='no'/>
>> </pm>
>> <devices>
>> <emulator>/usr/local/bin/qemu-system-x86_64.hv</emulator>
>> <disk type='block' device='disk'>
>> <driver name='qemu' type='raw' cache='none'/>
>> <source dev='/dev/mapper/vg_ssd-lv_kvm_NVIDIA'/>
>> <target dev='sda' bus='scsi'/>
>> <boot order='1'/>
>> <address type='drive' controller='0' bus='0' target='0'
>> unit='0'/>
>> </disk>
>> <disk type='block' device='disk'>
>> <driver name='qemu' type='raw' cache='none'/>
>> <source dev='/dev/mapper/vg_raid5-lv_xen_ntfs_files'/>
>> <target dev='sdb' bus='scsi'/>
>> <address type='drive' controller='0' bus='0' target='0'
>> unit='1'/>
>> </disk>
>> <controller type='usb' index='0'>
>> <address type='pci' domain='0x0000' bus='0x00'
>> slot='0x01' function='0x2'/>
>> </controller>
>> <controller type='pci' index='0' model='pci-root'/>
>> <controller type='scsi' index='0' model='virtio-scsi'>
>> <address type='pci' domain='0x0000' bus='0x00'
>> slot='0x06' function='0x0'/>
>> </controller>
>> <interface type='bridge'>
>> <mac address='52:54:00:e9:85:8f'/>
>> <source bridge='xenbr0'/>
>> <model type='e1000'/>
>> <address type='pci' domain='0x0000' bus='0x00'
>> slot='0x03' function='0x0'/>
>> </interface>
>> <hostdev mode='subsystem' type='pci' managed='yes'>
>> <source>
>> <address domain='0x0000' bus='0x01' slot='0x00'
>> function='0x0'/>
>> </source>
>> <address type='pci' domain='0x0000' bus='0x00'
>> slot='0x0a' function='0x0' multifunction='on'/>
>> </hostdev>
>> <hostdev mode='subsystem' type='pci' managed='yes'>
>> <source>
>> <address domain='0x0000' bus='0x01' slot='0x00'
>> function='0x1'/>
>> </source>
>> <address type='pci' domain='0x0000' bus='0x00'
>> slot='0x0a' function='0x1'/>
>> </hostdev>
>> <hostdev mode='subsystem' type='pci' managed='yes'>
>> <source>
>> <address domain='0x0000' bus='0x08' slot='0x00'
>> function='0x0'/>
>> </source>
>> <address type='pci' domain='0x0000' bus='0x00'
>> slot='0x08' function='0x0'/>
>> </hostdev>
>> <memballoon model='virtio'>
>> <address type='pci' domain='0x0000' bus='0x00'
>> slot='0x05' function='0x0'/>
>> </memballoon>
>> </devices>
>> </domain>
>> ------------------------------------------------------------------------
>>
>> /usr/local/bin/qemu-system-x86_64.hv:
>> #!/bin/sh
>> exec /usr/bin/qemu-system-x86_64 `echo "\$@" | \
>> sed 's|hv_time|hv_time,hv_vendor_id=GoobyPLS|g'
>>
>>
>>
>> And some notes:
>>
>> 1) Using "<topology sockets='1' cores='4' threads='1'/>"
>> instead of "<topology sockets='1' cores='2' threads='2'/>"
>> provided about 2% boost in GPU performance. No change in RAM
>> or CPU tests. I've tested with the passmark.
>>
>> 2) I tried using the emulatorpin method Alex says on a mail
>> here on vfio-users, but I didn't notice any changed in GPU
>> performance. I didn't test it on the CPU side though.
>>
>> 3) The main problem of the performance lack is that a
>> specific game that I've been playing isn't quite playable.
>> That game has been mentioned before here on the list, it's
>> Tera (european version (gameforge), although american
>> version(enmasse) has exactly the same performance).
>>
>> 4) Every other game I managed to play is quite playable,
>> though I haven't tested them to see if they run on native speeds.
>>
>>
>> I'd really want some help on this matter, I really want to
>> make my server run this VM with the nvidia GPU. I hate dual
>> booting Windows >_>
>>
>>
>> Thanks!
>>
>> _______________________________________________
>> vfio-users mailing list
>> vfio-users at redhat.com <mailto:vfio-users at redhat.com>
>> https://www.redhat.com/mailman/listinfo/vfio-users
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20151102/3d8cb80d/attachment.htm>
More information about the vfio-users
mailing list