[vfio-users] Stuttering in VM and "ignored rdmsr: 0x122" on new kernels?

[Bronek Kozicki]

I recently tried upgrading from vanilla LTS kernel 5.4 to vanilla 5.8 and found that my virtual machines "stutter" when they did not previously. The hypervisor is configured with tickless kernel (NO_HZ_FULL) and physical cores are exclusively reserved for vcpus by means of isolcpus , nohz_full and rcu_nocbs. The user experience under kernel 5.4 is good, it's only 5.8 when stuttering happens. This could be either just a brief "pop" when playing music, every few seconds, or actual short screen-freeze when playing games under Windows VM (I use GPU pass-through with Nvidia Quadro cards).

This seem to correspond with messages appearing on hypervisor console which look like:

[41426.188902] kvm [176116]: vcpu10, guest rIP: 0x7ff87a6a2aba ignored rdmsr: 0x122
[41426.202067] kvm [176116]: vcpu10, guest rIP: 0x7ff87a6a2aba ignored rdmsr: 0x122
[41426.218588] kvm [176116]: vcpu10, guest rIP: 0x7ff87b6d1d3a ignored rdmsr: 0x122
[41451.754213] kvm [1644425]: vcpu3, guest rIP: 0x7f5d4b442335 ignored rdmsr: 0x122
[41482.036754] kvm [176116]: vcpu2, guest rIP: 0x76f320ea ignored rdmsr: 0x122
[41482.181285] kvm [176116]: vcpu2, guest rIP: 0x76f320ea ignored rdmsr: 0x122
[41482.188947] kvm [176116]: vcpu2, guest rIP: 0x76d27f55 ignored rdmsr: 0x122

There is quite a lot of it, but only logged by kernel 5.8 and not by 5.4 (I skipped 5.5 , 5.6 and 5.7).

I build my own kernel roughly based on ArchLinux, but taking unaltered sources directly from kernel.org and my own configuration tweaks (such as CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y , MCORE2=y , PREEMPT_VOLUNTARY=y , NO_HZ_FULL=y , HZ_100=y , disable RCU_FAST_NO_HZ , disable X86_IOPL_IOPERM , disable USERFAULTFD , disable XEN, disable PAGE_POISONING , disable RAID6_PQ_BENCHMARK)

The hypervisor box has 2x Xeon 2697v2 (12 cores per CPU, two sockets, 48 physical threads) so there is plenty of CPU power left for I/O etc, despite 12 cores (24 threads) reserved for vcpus only: nohz_full=6-17,30-41 rcu_nocbs=6-17,30-41 isolcpus=6-17,30-41 . There are two VMs using these reserved cores:

  <vcpu placement='static'>12</vcpu>
  <iothreads>4</iothreads>
  <cputune>
    <vcpupin vcpu='0' cpuset='6'/>
    <vcpupin vcpu='1' cpuset='30'/>
    <vcpupin vcpu='2' cpuset='7'/>
    <vcpupin vcpu='3' cpuset='31'/>
    <vcpupin vcpu='4' cpuset='8'/>
    <vcpupin vcpu='5' cpuset='32'/>
    <vcpupin vcpu='6' cpuset='9'/>
    <vcpupin vcpu='7' cpuset='33'/>
    <vcpupin vcpu='8' cpuset='10'/>
    <vcpupin vcpu='9' cpuset='34'/>
    <vcpupin vcpu='10' cpuset='11'/>
    <vcpupin vcpu='11' cpuset='35'/>
    <emulatorpin cpuset='4-5,28-29'/>
    <iothreadpin iothread='1' cpuset='0,24'/>
    <iothreadpin iothread='2' cpuset='1,25'/>
    <iothreadpin iothread='3' cpuset='2,26'/>
    <iothreadpin iothread='4' cpuset='3,27'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
  </numatune>

... and:

  <vcpu placement='static'>12</vcpu>
  <iothreads>4</iothreads>
  <cputune>
    <vcpupin vcpu='0' cpuset='12'/>
    <vcpupin vcpu='1' cpuset='36'/>
    <vcpupin vcpu='2' cpuset='13'/>
    <vcpupin vcpu='3' cpuset='37'/>
    <vcpupin vcpu='4' cpuset='14'/>
    <vcpupin vcpu='5' cpuset='38'/>
    <vcpupin vcpu='6' cpuset='15'/>
    <vcpupin vcpu='7' cpuset='39'/>
    <vcpupin vcpu='8' cpuset='16'/>
    <vcpupin vcpu='9' cpuset='40'/>
    <vcpupin vcpu='10' cpuset='17'/>
    <vcpupin vcpu='11' cpuset='41'/>
    <emulatorpin cpuset='18-19,42-43'/>
    <iothreadpin iothread='1' cpuset='20,44'/>
    <iothreadpin iothread='2' cpuset='21,45'/>
    <iothreadpin iothread='3' cpuset='22,46'/>
    <iothreadpin iothread='4' cpuset='23,47'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='1'/>
  </numatune


There is no swap at all, the box has almost 400GB RAM.


B.

-- 
  Bronek Kozicki
  brok at incorrekt.com