[vfio-users] GPU performance degradation using KVM and Nvidia

[Tobias B.]

Am 2015.08.29 18:01, schrieb Alex Williamson:
> On Sat, Aug 29, 2015 at 9:43 AM, Tobias B.
> <honeypot.carbage at aechelon.de> wrote:
> 
>> Hello,
>> 
>> when I use KVM in combination with a Nvidia GPU I get far lower fps
>> values than expected. Sometimes even 60% lower than native
>> performance. I don't use Hugepages and I pin my vCPUs according to
>> the real pCPUs, like 1-3,5-7. Physical core #0 is assigned to the
>> host and #1,#2,#3 entirely to guest. But sometimes the guest
>> stutters, when the host is busy doing cpu intensive tasks on it's
>> cores
> 
> What else would you expect to happen?  If the host needs CPU and you
> haven't prioritized the QEMU threads or used isolcpus to completely
> remove host processes from those cores, then the host is going to
> share those cores and the guest performance will suffer.  If you want
> to sacrifice host performance for the guest, boot with
> isolcpus=1-3,5-7 nohz_full=1-3,5-7

Never found these kernel options on the internet. Sorry about that.

>> <vcpu placement='static' cpuset='1-3,5-7'>6</vcpu>
> 
> You should be pinning each vcpu, ex:
> 
>   <vcpu placement='static'>6</vcpu>
>   <cputune>
>     <vcpupin vcpu='0' cpuset='1'/>
>     <vcpupin vcpu='1' cpuset='2'/>
>     <vcpupin vcpu='2' cpuset='3'/>
>     <vcpupin vcpu='3' cpuset='5'/>
> 
>     <vcpupin vcpu='4' cpuset='6'/>
>     <vcpupin vcpu='5' cpuset='7'/>
>   </cputune>

I really have to thank you. Nearly native performance now.
That did the trick.

>> <disk type='file' device='disk'>
>> <driver name='qemu' type='qcow2' cache='none' io='native'/>
>> <source file='/home/tobias/Dokumente/test.qcow2'/>
>> <target dev='vda' bus='virtio'/>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
>> function='0x0'/>
>> </disk>
> 
> qcow for your disk could also be contributing to VM performance
> issues.

The qcow2 image is stored on a SSD drive. It's still blazing fast.

>> <qemu:commandline>
>> <qemu:arg value='-device'/>
>> <qemu:arg
>> value='vfio-pci,host=02:00.0,multifunction=on,x-vga=on'/>
>> <qemu:arg value='-device'/>
>> <qemu:arg value='vfio-pci,host=02:00.1'/>
>> </qemu:commandline>
> 
> *gag*, I explain in part 5 of my guide how to apply x-vga=on without
> these awful <qemu:arg> options that hide the assigned device from
> libvirt.  BTW, you're multifunction=on option isn't doing much here
> since you're not giving the device an explicit VM address.

I know, I know. But I had to remove and change these arguments A LOT and
it's easier to change them here as arguments than whole libvirtd 
sections.
Just for testing purposes. I got the 'multifunction=on' part from the 
Arch Linux
wiki. I never questioned that part, so thank you for pointing that out.