[vfio-users] vfio-users Digest, Vol 4, Issue 13

Bob Newell newellrp1 at yahoo.com
Mon Nov 2 17:29:02 UTC 2015 

If I want to change my subscription to this mailing list, how do I do it? I want to change to one email per day.
Thanks!
      From: "vfio-users-request at redhat.com" <vfio-users-request at redhat.com>
 To: vfio-users at redhat.com 
 Sent: Monday, November 2, 2015 9:21 AM
 Subject: vfio-users Digest, Vol 4, Issue 13
   
Send vfio-users mailing list submissions to
    vfio-users at redhat.com

To subscribe or unsubscribe via the World Wide Web, visit
    https://www.redhat.com/mailman/listinfo/vfio-users
or, via email, send a message with subject or body 'help' to
    vfio-users-request at redhat.com

You can reach the person managing the list at
    vfio-users-owner at redhat.com

When replying, please edit your Subject line so it is more specific
than "Re: Contents of vfio-users digest..."


Today's Topics:

  1. Re: Poor performance with nvidia GTX 980 (Okky Hendriansyah)


----------------------------------------------------------------------

Message: 1
Date: Mon, 2 Nov 2015 21:21:38 +0700
From: Okky Hendriansyah <okky at nostratech.com>
To: Georgios Kourachanis <geo.kourachanis at gmail.com>
Cc: vfio-users <vfio-users at redhat.com>
Subject: Re: [vfio-users] Poor performance with nvidia GTX 980
Message-ID: <BBA5DF1C-752C-46B1-BA82-7E54CE75EE22 at nostratech.com>
Content-Type: text/plain; charset="utf-8"

Sorry, it should be kernel 4.1.12-lts. :)

Best regards,
Okky Hendriansyah

> On Nov 2, 2015, at 21:16, Okky Hendriansyah <okky at nostratech.com> wrote:
> 
> I think the best benchmark would be the in-game ones. Since I cannot try Tera (region restricted), what are the other games that you play? Hopefully I can try to benchmark that on my VM and start from there. You can also try the Unigiene Valley benchmark.
> 
> My host has an ASRock Z87 Extreme6 with Intel Core i7-4770, patched my kernel with ACS Override patch, currently at kernel 4.1.2-lts (patched from ABS PKGBUILD). I do not isolate any cores on the host and give all the cores the VM (exposed as 8 vcpus with topology 1 socket, 4 cores, each core 2 threads)
> 
> Best regards,
> Okky Hendriansyah
> 
>> On Nov 2, 2015, at 19:54, Eddie Yen <missile0407 at gmail.com> wrote:
>> 
>> For now the latest driver is 358.50, and my guest using latest driver wo any problem.
>> But I'm using the method that AW talking about, so maybe give it a try?
>> 
>> 2015-11-02 20:47 GMT+08:00 Georgios Kourachanis <geo.kourachanis at gmail.com>:
>>> It's the same thing, either by adding them with qemuarguments, or with the wrapper.
>>> 
>>> The thing is to use the hyper-v functions. That's what the hyper-v vendor-id patch has given to us. The ability of hidding the hyper-v functions from nvidia GPUs so that we can use them!
>>> 
>>> Also, I've tried with a null name for the vendor-id, I got the same performance.
>>> 
>>> The nvidia drivers I'm currently using are 358.50
>>> 
>>> Moreover, could you suggest me a nice software to test the VM's performance in general? I don't really like passmark.
>>> 
>>> 
>>> 
>>> 
>>>> On 02/11/2015 02:11 ??, Eddie Yen wrote:
>>>> OK, but I still suggest that remove Hyper-V function tags in your XML.
>>>> Because we don't know about what new tricks inside the driver that NVIDIA wants to "surprise" us.
>>>> 
>>>> For me, my GTX980 works well by using upon edits. But I'm using 4820K which didn't need ACS patch and wo intel graphic.
>>>> So I'm not sure it may cause by patch or sth.
>>>> 
>>>> 2015-11-02 20:04 GMT+08:00 Georgios Kourachanis <geo.kourachanis at gmail.com>:
>>>>> Hello Eddie,
>>>>> 
>>>>> Thanks for answering, though:
>>>>> 
>>>>> What you suggest me to do, I've already done it with this way:
>>>>> 
>>>>> /usr/local/bin/qemu-system-x86_64.hv:
>>>>> #!/bin/sh
>>>>> exec /usr/bin/qemu-system-x86_64 `echo "\$@" | \
>>>>> sed 's|hv_time|hv_time,hv_vendor_id=GoobyPLS|g'
>>>>> 
>>>>> 
>>>>> and by changing the emulator qemu to this line:
>>>>> 
>>>>> <emulator>/usr/local/bin/qemu-system-x86_64.hv</emulator>
>>>>> 
>>>>> I'm just giving the ID "GoobyPLS" to the vendor. I'll try without a vendor name to see if it changes anything.
>>>>> 
>>>>> Also, I'm using the qemu git version "r41983.g3a958f5" so it already contains the patch that helps us use the lines above.
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On 02/11/2015 03:53 ??, Eddie Yen wrote:
>>>>>> According from AW's blog:
>>>>>> " For this step we again need to run virsh edit on the VM.  Within the<features> section, remove everything between the <hyperv> tags, including the tags themselves.0"
>>>>>> and
>>>>>> "Additionally, within the <clock> tag, find the timer named hypervclock, remove the line containing this tag completely.  Save and exit the edit session."
>>>>>> 
>>>>>> I found that these still exist in your XML file, so try to do this:
>>>>>> 
>>>>>> 1. Remove these tags.
>>>>>> 2. Re-compile QEMU and re-install it with this patch
>>>>>> http://www.spinics.net/lists/kvm/msg121742.html
>>>>>> 3. Add these tags between </devices> and </domain>
>>>>>> 
>>>>>> <qemu:commandline>
>>>>>>    <qemu:arg value='-cpu'/>
>>>>>>    <qemu:arg value='host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,kvm=off,hv)vendor_id='/>
>>>>>> </qemu:commandline>
>>>>>> 
>>>>>> I'm using GTX980, too. Before that, I got poor 3D performance in Windows 10, after this patch and edition, I got performance back.
>>>>>> 
>>>>>> 2015-11-02 1:43 GMT+08:00 Georgios Kourachanis <geo.kourachanis at gmail.com>:
>>>>>>> Hello all,
>>>>>>> 
>>>>>>> I had been using Xen with some AMD GPUs for almost 2 years till about June 2015, when I then found out that KVM and libvirt could do the same stuff I was interested in with nvidia GPUs, too. I needed the CUDA cores so I did change to an ASUS GTX 980 Strix. But unfortunatelly, I don't get any good performance output from it. On native windows 7/10 installation it's a beast though.
>>>>>>> I also have an AMD R7 250 which works great with KVM. But let's not                                  mess with it.
>>>>>>> 
>>>>>>> Let me get to the point:
>>>>>>> 
>>>>>>> I have no problems as for the installation of Windows or OVMF or passing-through or anything else. The only problem is the GTX980's performance.
>>>>>>> The performance had a significant boost when I used the latest qemu branch with the hyper-v trick, but still, not getting what many people seem to claim in this mailing list "almost-native" (even with nvidia GPUs).
>>>>>>> 
>>>>>>> 
>>>>>>> Here is my system's specs:
>>>>>>> 
>>>>>>> Archlinux with 4.1.6-1-vfio (with the ACS patch ALONE)
>>>>>>> Intel Core i7 3770 ( I use the igpu for the archlinux)
>>>>>>> 24GiB RAM
>>>>>>> ASUS GTX 980 Strix
>>>>>>> Sapphire R7 250
>>>>>>> lspci (only pass-through'd stuff):
>>>>>>> 
>>>>>>> 01:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 980] (rev a1)
>>>>>>>        Subsystem: ASUSTeK Computer Inc. Device 8518
>>>>>>>        Kernel driver in use: vfio-pci
>>>>>>>        Kernel modules: nouveau
>>>>>>> 01:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1)
>>>>>>>        Subsystem: ASUSTeK Computer Inc. Device 8518
>>>>>>>        Kernel driver in use: vfio-pci
>>>>>>>        Kernel modules: snd_hda_intel
>>>>>>> 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Oland PRO [Radeon R7 240/340]
>>>>>>>        Subsystem: PC Partner Limited / Sapphire Technology Device e266
>>>>>>>        Kernel modules: radeon
>>>>>>> 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
>>>>>>>        Subsystem: PC Partner Limited / Sapphire Technology Device aab0
>>>>>>>        Kernel driver in use: snd_hda_intel
>>>>>>>        Kernel modules: snd_hda_intel
>>>>>>> 08:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
>>>>>>>        Subsystem: ASRock Incorporation Motherboard
>>>>>>>        Kernel driver in use: vfio-pci
>>>>>>>        Kernel modules: xhci_pci
>>>>>>> booting lines:
>>>>>>> 
>>>>>>> linux    /boot/vmlinuz-linux-vfio root=UUID=XXXX rw intel_iommu=on pcie_acs_override=downstream isolcpus=2-3,6-7 nohz_full=2-3,6-7
>>>>>>> initrd    /boot/intel-ucode.img /boot/initramfs-linux-vfio.img
>>>>>>> /etc/fstab:
>>>>>>> 
>>>>>>> hugetlbfs /hugepages hugetlbfs defaults 0 0
>>>>>>> /etc/sysctl.d/40-hugepage.conf:
>>>>>>> 
>>>>>>> vm.nr_hugepages = 8000
>>>>>>> /etc/modules-load.d/vfio.conf:
>>>>>>> 
>>>>>>> kvm
>>>>>>> kvm-intel
>>>>>>> vfio
>>>>>>> vfio-pci
>>>>>>> vfio_iommu_type1
>>>>>>> vfio_virqfd
>>>>>>> /etc/modprobe.d/kvm.conf:
>>>>>>> 
>>>>>>> options kvm ignore_msrs=1
>>>>>>> /etc/modprobe.d/kvm-intel.conf:
>>>>>>> 
>>>>>>> options kvm-intel nested=1
>>>>>>> /etc/modprobe.d/vfio_iommu_type1.conf:
>>>>>>> 
>>>>>>> options vfio_iommu_type1 allow_unsafe_interrupts=0
>>>>>>> /etc/modprobe.d/vfio-pci.conf:
>>>>>>> 
>>>>>>> options vfio-pci ids=10de:13c0,10de:0fbb,1002:6613,1002:aab0,1b21:1042
>>>>>>> 
>>>>>>> And the virsh xml:
>>>>>>> 
>>>>>>> <domain type='kvm'>
>>>>>>>  <name>windows_10</name>
>>>>>>>  <uuid>63045df8-c782-4cfd-abc7-a3598826ae83</uuid>
>>>>>>>  <memory unit='KiB'>6553600</memory>
>>>>>>>  <currentMemory unit='KiB'>6553600</currentMemory>
>>>>>>>  <memoryBacking>
>>>>>>>    <hugepages/>
>>>>>>>  </memoryBacking>
>>>>>>>  <vcpu placement='static'>4</vcpu>
>>>>>>>  <cputune>
>>>>>>>    <vcpupin vcpu='0' cpuset='2'/>
>>>>>>>    <vcpupin vcpu='1' cpuset='3'/>
>>>>>>>    <vcpupin vcpu='2' cpuset='6'/>
>>>>>>>    <vcpupin vcpu='3' cpuset='7'/>
>>>>>>>  </cputune>
>>>>>>>  <os>
>>>>>>>    <type arch='x86_64' machine='pc-i440fx-2.4'>hvm</type>
>>>>>>>    <loader readonly='yes' type='pflash'>/usr/local/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
>>>>>>>    <nvram>/var/lib/libvirt/qemu/nvram/windows_nvidia_VARS.fd</nvram>
>>>>>>>  </os>
>>>>>>>  <features>
>>>>>>>    <acpi/>
>>>>>>>    <apic/>
>>>>>>>    <pae/>
>>>>>>>    <hyperv>
>>>>>>>      <relaxed state='on'/>
>>>>>>>      <vapic state='on'/>
>>>>>>>      <spinlocks state='on' retries='8191'/>
>>>>>>>    </hyperv>
>>>>>>>    <kvm>
>>>>>>>      <hidden state='on'/>
>>>>>>>    </kvm>
>>>>>>>    <vmport state='off'/>
>>>>>>>  </features>
>>>>>>>  <cpu mode='host-passthrough'>
>>>>>>>    <topology sockets='1' cores='4' threads='1'/>
>>>>>>>  </cpu>
>>>>>>>  <clock offset='localtime'>
>>>>>>>    <timer name='rtc' tickpolicy='catchup'/>
>>>>>>>    <timer name='pit' tickpolicy='delay'/>
>>>>>>>    <timer name='hpet' present='no'/>
>>>>>>>    <timer name='hypervclock' present='yes'/>
>>>>>>>  </clock>
>>>>>>>  <on_poweroff>destroy</on_poweroff>
>>>>>>>  <on_reboot>restart</on_reboot>
>>>>>>>  <on_crash>restart</on_crash>
>>>>>>>  <pm>
>>>>>>>    <suspend-to-mem enabled='no'/>
>>>>>>>    <suspend-to-disk enabled='no'/>
>>>>>>>  </pm>
>>>>>>>  <devices>
>>>>>>>    <emulator>/usr/local/bin/qemu-system-x86_64.hv</emulator>
>>>>>>>    <disk type='block' device='disk'>
>>>>>>>      <driver name='qemu' type='raw' cache='none'/>
>>>>>>>      <source dev='/dev/mapper/vg_ssd-lv_kvm_NVIDIA'/>
>>>>>>>      <target dev='sda' bus='scsi'/>
>>>>>>>      <boot order='1'/>
>>>>>>>      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>>>>>>>    </disk>
>>>>>>>    <disk type='block' device='disk'>
>>>>>>>      <driver name='qemu' type='raw' cache='none'/>
>>>>>>>      <source dev='/dev/mapper/vg_raid5-lv_xen_ntfs_files'/>
>>>>>>>      <target dev='sdb' bus='scsi'/>
>>>>>>>      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
>>>>>>>    </disk>
>>>>>>>    <controller type='usb' index='0'>
>>>>>>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
>>>>>>>    </controller>
>>>>>>>    <controller type='pci' index='0' model='pci-root'/>
>>>>>>>    <controller type='scsi' index='0' model='virtio-scsi'>
>>>>>>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
>>>>>>>    </controller>
>>>>>>>    <interface type='bridge'>
>>>>>>>      <mac address='52:54:00:e9:85:8f'/>
>>>>>>>      <source bridge='xenbr0'/>
>>>>>>>      <model type='e1000'/>
>>>>>>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
>>>>>>>    </interface>
>>>>>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>>>>>      <source>
>>>>>>>        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
>>>>>>>      </source>
>>>>>>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0' multifunction='on'/>
>>>>>>>    </hostdev>
>>>>>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>>>>>      <source>
>>>>>>>        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
>>>>>>>      </source>
>>>>>>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x1'/>
>>>>>>>    </hostdev>
>>>>>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>>>>>      <source>
>>>>>>>        <address domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
>>>>>>>      </source>
>>>>>>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
>>>>>>>    </hostdev>
>>>>>>>    <memballoon model='virtio'>
>>>>>>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
>>>>>>>    </memballoon>
>>>>>>>  </devices>
>>>>>>> </domain>
>>>>>>> 
>>>>>>> /usr/local/bin/qemu-system-x86_64.hv:
>>>>>>> #!/bin/sh
>>>>>>> exec /usr/bin/qemu-system-x86_64 `echo "\$@" | \
>>>>>>> sed 's|hv_time|hv_time,hv_vendor_id=GoobyPLS|g'
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> And some notes:
>>>>>>> 
>>>>>>> 1) Using "<topology sockets='1' cores='4' threads='1'/>" instead of "<topology sockets='1' cores='2' threads='2'/>" provided about 2% boost in GPU performance. No change in RAM or CPU tests. I've tested with the passmark.
>>>>>>> 
>>>>>>> 2) I tried using the emulatorpin method Alex says on a mail here on vfio-users, but I didn't notice any changed in GPU performance. I didn't test it on the CPU side though.
>>>>>>> 
>>>>>>> 3) The main problem of the performance lack is that a specific                                  game that I've been playing isn't quite playable. That game has been mentioned before here on the list, it's Tera (european version (gameforge), although american version(enmasse) has exactly the same performance).
>>>>>>> 
>>>>>>> 4) Every other game I managed to play is quite playable, though I haven't tested them to see if they run on native speeds.
>>>>>>> 
>>>>>>> 
>>>>>>> I'd really want some help on this matter, I really want to make my server run this VM with the nvidia GPU. I hate dual booting Windows >_>
>>>>>>> 
>>>>>>> 
>>>>>>> Thanks!
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> vfio-users mailing list
>>>>>>> vfio-users at redhat.com
>>>>>>> https://www.redhat.com/mailman/listinfo/vfio-users
>> 
>> _______________________________________________
>> vfio-users mailing list
>> vfio-users at redhat.com
>> https://www.redhat.com/mailman/listinfo/vfio-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.redhat.com/archives/vfio-users/attachments/20151102/74b1ae54/attachment.html>

------------------------------

_______________________________________________
vfio-users mailing list
vfio-users at redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users


End of vfio-users Digest, Vol 4, Issue 13
*****************************************


  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20151102/e85fc575/attachment.htm>

More information about the vfio-users mailing list