[vfio-users] Poor performance with nvidia GTX 980
Georgios Kourachanis
geo.kourachanis at gmail.com
Tue Nov 3 15:27:11 UTC 2015
Hello Eddie,
I've tried the way you suggested me, and I got exactly the same qemu
line to be executed.
I suggest you to use my way of using the vendor-id and don't mess with
qemu arguments in the xml file.
Regards,
George
On 02/11/2015 02:54 μμ, Eddie Yen wrote:
> For now the latest driver is 358.50, and my guest using latest driver
> wo any problem.
> But I'm using the method that AW talking about, so maybe give it a try?
>
> 2015-11-02 20:47 GMT+08:00 Georgios Kourachanis
> <geo.kourachanis at gmail.com <mailto:geo.kourachanis at gmail.com>>:
>
> It's the same thing, either by adding them with qemuarguments, or
> with the wrapper.
>
> The thing is to use the hyper-v functions. That's what the hyper-v
> vendor-id patch has given to us. The ability of hidding the
> hyper-v functions from nvidia GPUs so that we can use them!
>
> Also, I've tried with a null name for the vendor-id, I got the
> same performance.
>
> The nvidia drivers I'm currently using are 358.50
>
> Moreover, could you suggest me a nice software to test the VM's
> performance in general? I don't really like passmark.
>
>
>
>
> On 02/11/2015 02:11 μμ, Eddie Yen wrote:
>> OK, but I still suggest that remove Hyper-V function tags in your
>> XML.
>> Because we don't know about what new tricks inside the driver
>> that NVIDIA wants to "surprise" us.
>>
>> For me, my GTX980 works well by using upon edits. But I'm using
>> 4820K which didn't need ACS patch and wo intel graphic.
>> So I'm not sure it may cause by patch or sth.
>>
>> 2015-11-02 20:04 GMT+08:00 Georgios Kourachanis
>> <geo.kourachanis at gmail.com <mailto:geo.kourachanis at gmail.com>>:
>>
>> Hello Eddie,
>>
>> Thanks for answering, though:
>>
>> What you suggest me to do, I've already done it with this way:
>>
>> /usr/local/bin/qemu-system-x86_64.hv:
>> #!/bin/sh
>> exec /usr/bin/qemu-system-x86_64 `echo "\$@" | \
>> sed 's|hv_time|hv_time,hv_vendor_id=GoobyPLS|g'
>>
>>
>> and by changing the emulator qemu to this line:
>>
>> <emulator>/usr/local/bin/qemu-system-x86_64.hv</emulator>
>>
>> I'm just giving the ID "GoobyPLS" to the vendor. I'll try
>> without a vendor name to see if it changes anything.
>>
>> Also, I'm using the qemu git version "r41983.g3a958f5" so it
>> already contains the patch that helps us use the lines above.
>>
>>
>>
>>
>>
>> On 02/11/2015 03:53 πμ, Eddie Yen wrote:
>>> According from AW's blog:
>>> " For this step we again need to run virsh edit on the VM.
>>> Within the<features> section, remove everything between the
>>> <hyperv> tags, including the tags themselves.0"
>>> and
>>> "Additionally, within the <clock> tag, find the timer named
>>> hypervclock, remove the line containing this tag
>>> completely. Save and exit the edit session."
>>>
>>> I found that these still exist in your XML file, so try to
>>> do this:
>>>
>>> 1. Remove these tags.
>>> 2. Re-compile QEMU and re-install it with this patch
>>> http://www.spinics.net/lists/kvm/msg121742.html
>>> 3. Add these tags between </devices> and </domain>
>>>
>>> <qemu:commandline>
>>> <qemu:arg value='-cpu'/>
>>> <qemu:arg
>>> value='host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,kvm=off,hv)vendor_id='/>
>>> </qemu:commandline>
>>>
>>> I'm using GTX980, too. Before that, I got poor 3D
>>> performance in Windows 10, after this patch and edition, I
>>> got performance back.
>>>
>>> 2015-11-02 1:43 GMT+08:00 Georgios Kourachanis
>>> <geo.kourachanis at gmail.com <mailto:geo.kourachanis at gmail.com>>:
>>>
>>> Hello all,
>>>
>>> I had been using Xen with some AMD GPUs for almost 2
>>> years till about June 2015, when I then found out that
>>> KVM and libvirt could do the same stuff I was interested
>>> in with nvidia GPUs, too. I needed the CUDA cores so I
>>> did change to an ASUS GTX 980 Strix. But unfortunatelly,
>>> I don't get any good performance output from it. On
>>> native windows 7/10 installation it's a beast though.
>>> I also have an AMD R7 250 which works great with KVM.
>>> But let's not mess with it.
>>>
>>> Let me get to the point:
>>>
>>> I have no problems as for the installation of Windows or
>>> OVMF or passing-through or anything else. The only
>>> problem is the GTX980's performance.
>>> The performance had a significant boost when I used the
>>> latest qemu branch with the hyper-v trick, but still,
>>> not getting what many people seem to claim in this
>>> mailing list "almost-native" (even with nvidia GPUs).
>>>
>>>
>>> Here is my system's specs:
>>>
>>> Archlinux with 4.1.6-1-vfio (with the ACS patch ALONE)
>>> Intel Core i73770 ( I use the igpu for the archlinux)
>>> 24GiB RAM
>>> ASUS GTX 980 Strix
>>> Sapphire R7 250
>>> ------------------------------------------------------------------------
>>> lspci (only pass-through'd stuff):
>>>
>>> 01:00.0 VGA compatible controller: NVIDIA Corporation
>>> GM204 [GeForce GTX 980] (rev a1)
>>> Subsystem: ASUSTeK Computer Inc. Device 8518
>>> Kernel driver in use: vfio-pci
>>> Kernel modules: nouveau
>>> 01:00.1 Audio device: NVIDIA Corporation GM204 High
>>> Definition Audio Controller (rev a1)
>>> Subsystem: ASUSTeK Computer Inc. Device 8518
>>> Kernel driver in use: vfio-pci
>>> Kernel modules: snd_hda_intel
>>> 02:00.0 VGA compatible controller: Advanced Micro
>>> Devices, Inc. [AMD/ATI] Oland PRO [Radeon R7 240/340]
>>> Subsystem: PC Partner Limited / Sapphire Technology
>>> Device e266
>>> Kernel modules: radeon
>>> 02:00.1 Audio device: Advanced Micro Devices, Inc.
>>> [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD
>>> 7700/7800 Series]
>>> Subsystem: PC Partner Limited / Sapphire Technology
>>> Device aab0
>>> Kernel driver in use: snd_hda_intel
>>> Kernel modules: snd_hda_intel
>>> 08:00.0 USB controller: ASMedia Technology Inc. ASM1042
>>> SuperSpeed USB Host Controller
>>> Subsystem: ASRock Incorporation Motherboard
>>> Kernel driver in use: vfio-pci
>>> Kernel modules: xhci_pci
>>> ------------------------------------------------------------------------
>>> booting lines:
>>>
>>> linux /boot/vmlinuz-linux-vfio root=UUID=XXXX rw
>>> intel_iommu=on pcie_acs_override=downstream
>>> isolcpus=2-3,6-7 nohz_full=2-3,6-7
>>> initrd /boot/intel-ucode.img /boot/initramfs-linux-vfio.img
>>> ------------------------------------------------------------------------
>>> /etc/fstab:|
>>>
>>> hugetlbfs /hugepages hugetlbfs defaults 0 0|
>>> ------------------------------------------------------------------------
>>> /etc/sysctl.d/40-hugepage.conf:
>>>
>>> vm.nr_hugepages = 8000
>>> ------------------------------------------------------------------------
>>> /etc/modules-load.d/vfio.conf:
>>>
>>> kvm
>>> kvm-intel
>>> vfio
>>> vfio-pci
>>> vfio_iommu_type1
>>> vfio_virqfd
>>> ------------------------------------------------------------------------
>>> /etc/modprobe.d/kvm.conf:
>>>
>>> options kvm ignore_msrs=1
>>> ------------------------------------------------------------------------
>>> /etc/modprobe.d/kvm-intel.conf:
>>>
>>> options kvm-intel nested=1
>>> ------------------------------------------------------------------------
>>> /etc/modprobe.d/vfio_iommu_type1.conf:
>>>
>>> options vfio_iommu_type1 allow_unsafe_interrupts=0
>>> ------------------------------------------------------------------------
>>> /etc/modprobe.d/vfio-pci.conf:
>>>
>>> options vfio-pci
>>> ids=10de:13c0,10de:0fbb,1002:6613,1002:aab0,1b21:1042
>>> ------------------------------------------------------------------------
>>>
>>> And the virsh xml:
>>>
>>> <domain type='kvm'>
>>> <name>windows_10</name>
>>> <uuid>63045df8-c782-4cfd-abc7-a3598826ae83</uuid>
>>> <memory unit='KiB'>6553600</memory>
>>> <currentMemory unit='KiB'>6553600</currentMemory>
>>> <memoryBacking>
>>> <hugepages/>
>>> </memoryBacking>
>>> <vcpu placement='static'>4</vcpu>
>>> <cputune>
>>> <vcpupin vcpu='0' cpuset='2'/>
>>> <vcpupin vcpu='1' cpuset='3'/>
>>> <vcpupin vcpu='2' cpuset='6'/>
>>> <vcpupin vcpu='3' cpuset='7'/>
>>> </cputune>
>>> <os>
>>> <type arch='x86_64' machine='pc-i440fx-2.4'>hvm</type>
>>> <loader readonly='yes'
>>> type='pflash'>/usr/local/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
>>> <nvram>/var/lib/libvirt/qemu/nvram/windows_nvidia_VARS.fd</nvram>
>>> </os>
>>> <features>
>>> <acpi/>
>>> <apic/>
>>> <pae/>
>>> <hyperv>
>>> <relaxed state='on'/>
>>> <vapic state='on'/>
>>> <spinlocks state='on' retries='8191'/>
>>> </hyperv>
>>> <kvm>
>>> <hidden state='on'/>
>>> </kvm>
>>> <vmport state='off'/>
>>> </features>
>>> <cpu mode='host-passthrough'>
>>> <topology sockets='1' cores='4' threads='1'/>
>>> </cpu>
>>> <clock offset='localtime'>
>>> <timer name='rtc' tickpolicy='catchup'/>
>>> <timer name='pit' tickpolicy='delay'/>
>>> <timer name='hpet' present='no'/>
>>> <timer name='hypervclock' present='yes'/>
>>> </clock>
>>> <on_poweroff>destroy</on_poweroff>
>>> <on_reboot>restart</on_reboot>
>>> <on_crash>restart</on_crash>
>>> <pm>
>>> <suspend-to-mem enabled='no'/>
>>> <suspend-to-disk enabled='no'/>
>>> </pm>
>>> <devices>
>>> <emulator>/usr/local/bin/qemu-system-x86_64.hv</emulator>
>>> <disk type='block' device='disk'>
>>> <driver name='qemu' type='raw' cache='none'/>
>>> <source dev='/dev/mapper/vg_ssd-lv_kvm_NVIDIA'/>
>>> <target dev='sda' bus='scsi'/>
>>> <boot order='1'/>
>>> <address type='drive' controller='0' bus='0'
>>> target='0' unit='0'/>
>>> </disk>
>>> <disk type='block' device='disk'>
>>> <driver name='qemu' type='raw' cache='none'/>
>>> <source dev='/dev/mapper/vg_raid5-lv_xen_ntfs_files'/>
>>> <target dev='sdb' bus='scsi'/>
>>> <address type='drive' controller='0' bus='0'
>>> target='0' unit='1'/>
>>> </disk>
>>> <controller type='usb' index='0'>
>>> <address type='pci' domain='0x0000' bus='0x00'
>>> slot='0x01' function='0x2'/>
>>> </controller>
>>> <controller type='pci' index='0' model='pci-root'/>
>>> <controller type='scsi' index='0' model='virtio-scsi'>
>>> <address type='pci' domain='0x0000' bus='0x00'
>>> slot='0x06' function='0x0'/>
>>> </controller>
>>> <interface type='bridge'>
>>> <mac address='52:54:00:e9:85:8f'/>
>>> <source bridge='xenbr0'/>
>>> <model type='e1000'/>
>>> <address type='pci' domain='0x0000' bus='0x00'
>>> slot='0x03' function='0x0'/>
>>> </interface>
>>> <hostdev mode='subsystem' type='pci' managed='yes'>
>>> <source>
>>> <address domain='0x0000' bus='0x01' slot='0x00'
>>> function='0x0'/>
>>> </source>
>>> <address type='pci' domain='0x0000' bus='0x00'
>>> slot='0x0a' function='0x0' multifunction='on'/>
>>> </hostdev>
>>> <hostdev mode='subsystem' type='pci' managed='yes'>
>>> <source>
>>> <address domain='0x0000' bus='0x01' slot='0x00'
>>> function='0x1'/>
>>> </source>
>>> <address type='pci' domain='0x0000' bus='0x00'
>>> slot='0x0a' function='0x1'/>
>>> </hostdev>
>>> <hostdev mode='subsystem' type='pci' managed='yes'>
>>> <source>
>>> <address domain='0x0000' bus='0x08' slot='0x00'
>>> function='0x0'/>
>>> </source>
>>> <address type='pci' domain='0x0000' bus='0x00'
>>> slot='0x08' function='0x0'/>
>>> </hostdev>
>>> <memballoon model='virtio'>
>>> <address type='pci' domain='0x0000' bus='0x00'
>>> slot='0x05' function='0x0'/>
>>> </memballoon>
>>> </devices>
>>> </domain>
>>> ------------------------------------------------------------------------
>>>
>>> /usr/local/bin/qemu-system-x86_64.hv:
>>> #!/bin/sh
>>> exec /usr/bin/qemu-system-x86_64 `echo "\$@" | \
>>> sed 's|hv_time|hv_time,hv_vendor_id=GoobyPLS|g'
>>>
>>>
>>>
>>> And some notes:
>>>
>>> 1) Using "<topology sockets='1' cores='4' threads='1'/>"
>>> instead of "<topology sockets='1' cores='2'
>>> threads='2'/>" provided about 2% boost in GPU
>>> performance. No change in RAM or CPU tests. I've tested
>>> with the passmark.
>>>
>>> 2) I tried using the emulatorpin method Alex says on a
>>> mail here on vfio-users, but I didn't notice any changed
>>> in GPU performance. I didn't test it on the CPU side though.
>>>
>>> 3) The main problem of the performance lack is that a
>>> specific game that I've been playing isn't quite
>>> playable. That game has been mentioned before here on
>>> the list, it's Tera (european version (gameforge),
>>> although american version(enmasse) has exactly the same
>>> performance).
>>>
>>> 4) Every other game I managed to play is quite playable,
>>> though I haven't tested them to see if they run on
>>> native speeds.
>>>
>>>
>>> I'd really want some help on this matter, I really want
>>> to make my server run this VM with the nvidia GPU. I
>>> hate dual booting Windows >_>
>>>
>>>
>>> Thanks!
>>>
>>> _______________________________________________
>>> vfio-users mailing list
>>> vfio-users at redhat.com <mailto:vfio-users at redhat.com>
>>> https://www.redhat.com/mailman/listinfo/vfio-users
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20151103/043cdc38/attachment.htm>
More information about the vfio-users
mailing list