[vfio-users] Looking for performance tips, qemu machine type and pci/pcie-root questionning?

[Okky Hendriansyah]

Hi Thibaut,

On Fri, Apr 15, 2016 at 10:46 AM, thibaut noah <thibaut.noah at gmail.com>
wrote:

> Hello everyone, so i'm still in my endless pursuit of performance.
>
> Currently crawling qemu options a bit, running the command "$
> qemu-system-x86_64 -machine help" seems that the main machine for us are
> q35 and pci-i440FX.
> From what i read in my online research the main advantage of q35 seems to
> be the support of pcie controller instead of just pci, so it might give
> more performance?
>

I think Alex had mentioned about this, and if I recall correctly using
pc-i440fx is preferrable since it is simpler and going to pc-q35 won't have
any performance benefit. Currently I only use pc-q35 specifically just for
my Hackintosh guest. I never done any benchmark between these two types
recently though, so the result might change.


> However replacing pc by q35 and replace pci-root by pcie-root in the
> libvirt xml file seems to break nvidia driver, even with removing all
> hyperv features, the vendor id being masqued i have to admit i don't
> understand why the nvidia drivers are not loading.
> My vm is running just fine with the other machine type but if i can gain a
> little bit more i would like to.
>
> Also on the performance side, i would like to sum everything that is
> possible to do in order to boost the guest system, currently i have this
> list :
>
> - hyperv features on libvirt (require to hide vendor_id)
>

According to one of the reddit users at /r/vfio [1], avoiding to use
hv_vapic and hv_syncic in newer Intel CPUs starting Ivy Bridge-E onwards
which has built-in Intel APICv will generally improve performance by
reducing VM exits. Currently I'm using these options:

*-cpu
host,kvm=off,hv_time,hv_relaxed,hv_spinlocks=0x1fff,hv_vpindex,hv_reset,hv_runtime,hv_crash,hv_vendor_id=freyja*

> - machine type? (need more specific answers on this)
> - 1000 MHz clock resolution on host kernel
> - Voluntary preemption scheduling on host kernel
>

Those two kernel configurations (1000 MHz and Voluntary) made my stuttery
Garret to a butter smooth Garret ;). Other plus point is that ZFS, which I
use extensively for the OS guest images prefers Voluntary also. [2]


> -  Madvise hugepages and hugelbfs on host, also kvm_hugepages need to be
> activated (don't forget the xml file)
>

I think MADVISE hugepages doesn't directly hit the guest performance.
Though I find that using this option could help eliminating uneeded
hugepage requests on applications that do not gain benefit from hugepages.
So this option is more to have an efficient memory usage on the host,
rather than guest performance since the guest is already using a dedicated
hugepages (hugetlbfs).


> - cpu configuration in libvirt xml (need to make more bench to find the
> perfect config)
> - set cpupower to performance mode
> - use cset to move all host processes to specific cores reserved for the
> system and untouch by the vm (although this one seems to be tricky since
> libvirt fail to grab the cores after the use of cset, need some work on
> this)
> Also, reserving cores for the host system seems pointless atm. With all
> cores pinned and use by the vm i crushed my previous setup on benchmark, as
> long as one doesn't do anything with the host it seems to be the best
> option for performance. (I personnaly made two xml for this very reason,
> one to use both systems at the same time, for example in order to do some
> programming, and the other for performance only where i don't touch the
> host at all as long as the guest os is running)
>
and that is about everything i found atm. If anyone on the mailing list has
> more tweaks, feel free to share.
> Please note i try to have the best vm possible for windows gaming, i
> assume most people using vfio are gamers anyway.
> Have a good day guys and may the mighty power of the loot be with you !
>

Don't forget to still enable Windows paging if your guest memory is below
the requirement. I've had low memory warning on Witcher 3 (I set the guest
memory to 8 GB, and it is still has 50%+ free memory) before I reenabled
back the Windows paging on C again. The other alternative is to increase
the guest memory. When I set it to 16 GB without Windows paging, Witcher 3
didn't complain anymore.

[1]
https://www.reddit.com/r/VFIO/comments/479xnx/guests_with_nvidia_gpus_can_enable_hyperv/
[2] https://github.com/zfsonlinux/zfs/issues/1620

Best regards,
Okky Hendriansyah
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160415/b9f30f45/attachment.htm>