[vfio-users] BSOD occuring with 1 game in VM

Fri May 13 05:07:27 UTC 2016

So I did a little bit more digging. It seems the MSRs are related to LBR
(Last Branch Records). KVM does not currently support LBR.
I did find a kvm LBR patch submitted last Nov at
https://www.mail-archive.com/kvm@vger.kernel.org/msg122887.html
I tried the patch, and the performance got even worse (down to 5 FPS).

I think the only way to resolve this problem is to somehow get proper LBR
support in KVM.

On Thu, May 12, 2016 at 8:36 PM Ivan Volosyuk <ivan.volosyuk at gmail.com>
wrote:

> My 2c: on Win8.1 in Starcraft 2 (nvidia 970 GT, i7 4790k CPU host) I have
> zero unknown msrs and have excellent sc2 performance. Several time I
> switched to Win10 I got the msrs spam in logs, so I decided to revert back
> to Win8.1 for now.
>
> On Fri, May 13, 2016, 8:02 AM Quentin Deldycke <quentindeldycke at gmail.com>
> wrote:
>
>> Aw:
>>
>> You can have my xml here:
>> https://github.com/qdel/scripts/blob/master/vfio/win10.xml
>>
>> Note that i have 6 cores allocated and on 7 there is also my emulator.
>> This or leaving it alone change nothing. So i let it like this.
>> My dpc latency is quite good actually.
>>
>> And as you can see, i virtualize "nothing", but only pass devices:
>>
>> - nvme ssd on pcie
>> - sata controller
>> - usb3 controller
>> - gpu and i also use it's sound output for windows sound.
>>
>> Sadly, my sata controller cannot use msi, same as sound controller of
>> gpu. But none seems to suffer of this. My sound is perfect and disk work
>> perfectly (and in any cases, hots is on the ssd)
>>
>> Jan:
>>
>> I saw also some cases where my vm is faster than native. Weird stuff.
>> For my case, it works nearly perfectly. I can even hard stress the host
>> without any impact on the vm. (but using cgroups, i can still temporary
>> make a make -j9 :P)
>> This is typically limiting cpu execution as the game asks to reduce
>> physics or stuff like this...
>>
>> Abdulla:
>>
>> Yep, sc2 make same mess, exactly same...
>> Good thing is that Overwatch beta was not affected I had a steady 60fps
>> on 4k with it. =)
>> Diablo 2 is not affected, And Wow on basics area aren't also (no game
>> time since years), same, much fps, and not a drop on gpu usage.
>>
>>
>> Following, msr storm, mainly always same msr: 1c9 680 6c0! On intel it
>> seems to be the "Performance Monitoring Unit". This snippet is on game
>> menus. during party i can easly reach 1 000 000 suppression. (and it pop
>> every 4-5 seconds)
>>
>> May 12 23:39:59 lindesk kernel: [10442.173587] kvm [12608]: vcpu5, guest
>> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
>> May 12 23:40:04 lindesk kernel: [10447.174529] kvm_set_msr_common: 241580
>> callbacks suppressed
>> May 12 23:40:04 lindesk kernel: [10447.174537] kvm [12608]: vcpu3, guest
>> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
>> May 12 23:40:04 lindesk kernel: [10447.174543] kvm_get_msr_common: 724760
>> callbacks suppressed
>> May 12 23:40:04 lindesk kernel: [10447.174545] kvm [12608]: vcpu3, guest
>> rIP: 0xfffff80185f4c9fc ignored rdmsr: 0x1c9
>> May 12 23:40:04 lindesk kernel: [10447.174547] kvm [12608]: vcpu3, guest
>> rIP: 0xfffff80185f4ca0a ignored rdmsr: 0x680
>> May 12 23:40:04 lindesk kernel: [10447.174549] kvm [12608]: vcpu3, guest
>> rIP: 0xfffff80185f4ca21 ignored rdmsr: 0x6c0
>> May 12 23:40:04 lindesk kernel: [10447.174556] kvm [12608]: vcpu3, guest
>> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
>> May 12 23:40:04 lindesk kernel: [10447.174561] kvm [12608]: vcpu3, guest
>> rIP: 0xfffff80185f4c9fc ignored rdmsr: 0x1c9
>> May 12 23:40:04 lindesk kernel: [10447.174564] kvm [12608]: vcpu3, guest
>> rIP: 0xfffff80185f4ca0a ignored rdmsr: 0x680
>> May 12 23:40:04 lindesk kernel: [10447.174567] kvm [12608]: vcpu3, guest
>> rIP: 0xfffff80185f4ca21 ignored rdmsr: 0x6c0
>> May 12 23:40:04 lindesk kernel: [10447.174575] kvm [12608]: vcpu3, guest
>> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
>> May 12 23:40:04 lindesk kernel: [10447.174592] kvm [12608]: vcpu3, guest
>> rIP: 0xfffff80185f4c9fc ignored rdmsr: 0x1c9
>> May 12 23:40:04 lindesk kernel: [10447.174594] kvm [12608]: vcpu3, guest
>> rIP: 0xfffff80185f4ca0a ignored rdmsr: 0x680
>> May 12 23:40:04 lindesk kernel: [10447.174596] kvm [12608]: vcpu3, guest
>> rIP: 0xfffff80185f4ca21 ignored rdmsr: 0x6c0
>> May 12 23:40:04 lindesk kernel: [10447.174603] kvm [12608]: vcpu3, guest
>> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
>> May 12 23:40:04 lindesk kernel: [10447.174611] kvm [12608]: vcpu3, guest
>> rIP: 0xfffff80185f4c9fc ignored rdmsr: 0x1c9
>> May 12 23:40:04 lindesk kernel: [10447.174619] kvm [12608]: vcpu3, guest
>> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
>> May 12 23:40:04 lindesk kernel: [10447.174642] kvm [12608]: vcpu3, guest
>> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
>> May 12 23:40:04 lindesk kernel: [10447.174735] kvm [12608]: vcpu3, guest
>> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
>> May 12 23:40:04 lindesk kernel: [10447.174772] kvm [12608]: vcpu3, guest
>> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
>> May 12 23:40:04 lindesk kernel: [10447.174797] kvm [12608]: vcpu3, guest
>> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
>> May 12 23:40:04 lindesk kernel: [10447.174815] kvm [12608]: vcpu3, guest
>> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
>> May 12 23:40:09 lindesk kernel: [10452.175786] kvm_get_msr_common: 156326
>> callbacks suppressed
>> May 12 23:40:09 lindesk kernel: [10452.175795] kvm [12608]: vcpu2, guest
>> rIP: 0xfffff80185f4c9fc ignored rdmsr: 0x1c9
>> May 12 23:40:09 lindesk kernel: [10452.175798] kvm [12608]: vcpu2, guest
>> rIP: 0xfffff80185f4ca0a ignored rdmsr: 0x680
>> May 12 23:40:09 lindesk kernel: [10452.175802] kvm [12608]: vcpu2, guest
>> rIP: 0xfffff80185f4ca21 ignored rdmsr: 0x6c0
>> May 12 23:40:09 lindesk kernel: [10452.175810] kvm_set_msr_common: 52103
>> callbacks suppressed
>> May 12 23:40:09 lindesk kernel: [10452.175812] kvm [12608]: vcpu2, guest
>> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
>> May 12 23:40:09 lindesk kernel: [10452.176286] kvm [12608]: vcpu2, guest
>> rIP: 0xfffff80185f4c9fc ignored rdmsr: 0x1c9
>> May 12 23:40:09 lindesk kernel: [10452.176292] kvm [12608]: vcpu2, guest
>> rIP: 0xfffff80185f4ca0a ignored rdmsr: 0x680
>> May 12 23:40:09 lindesk kernel: [10452.176295] kvm [12608]: vcpu2, guest
>> rIP: 0xfffff80185f4ca21 ignored rdmsr: 0x6c0
>> May 12 23:40:09 lindesk kernel: [10452.176304] kvm [12608]: vcpu2, guest
>> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
>> May 12 23:40:09 lindesk kernel: [10452.176324] kvm [12608]: vcpu2, guest
>> rIP: 0xfffff80185f4c9fc ignored rdmsr: 0x1c9
>> May 12 23:40:09 lindesk kernel: [10452.176326] kvm [12608]: vcpu2, guest
>> rIP: 0xfffff80185f4ca0a ignored rdmsr: 0x680
>> May 12 23:40:09 lindesk kernel: [10452.176328] kvm [12608]: vcpu2, guest
>> rIP: 0xfffff80185f4ca21 ignored rdmsr: 0x6c0
>> May 12 23:40:09 lindesk kernel: [10452.176335] kvm [12608]: vcpu2, guest
>> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
>> May 12 23:40:09 lindesk kernel: [10452.176785] kvm [12608]: vcpu2, guest
>> rIP: 0xfffff80185f4c9fc ignored rdmsr: 0x1c9
>> May 12 23:40:09 lindesk kernel: [10452.176799] kvm [12608]: vcpu2, guest
>> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
>> May 12 23:40:09 lindesk kernel: [10452.177201] kvm [12608]: vcpu2, guest
>> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
>> May 12 23:40:09 lindesk kernel: [10452.177291] kvm [12608]: vcpu2, guest
>> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
>> May 12 23:40:09 lindesk kernel: [10452.177318] kvm [12608]: vcpu2, guest
>> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
>> May 12 23:40:09 lindesk kernel: [10452.177530] kvm [12608]: vcpu2, guest
>> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
>> May 12 23:40:09 lindesk kernel: [10452.177793] kvm [12608]: vcpu2, guest
>> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
>> May 12 23:40:09 lindesk kernel: [10452.178291] kvm [12608]: vcpu2, guest
>> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
>>
>>
>>
>> On 12 May 2016 11:12 pm, "Jan Wiele" <jan at wiele.org> wrote:
>>
>>> You are not alone. I have had the exact same symptoms. Wanted to build a
>>> "gaming" machine with two seats for playing Heroes of the Storm.
>>> Performance was bad. Very low fps especially in fights. No difference
>>> between low and high graphic settings.
>>>
>>> Nevertheless, graphic benchmarks, like 3D mark, certified a very good
>>> GPU performance: [1] VM, [2] Native (Yes, the VM performs  slightly
>>> better). There, you can also see my system specs. I suppose this is some
>>> kind of CPU scaling problem in HotS. :-/
>>>
>>> I'm very interested if you can actually find a solution :) Good luck!
>>>
>>> Regards,
>>> Jan
>>>
>>> [1] http://www.3dmark.com/fs/7493163
>>> [2] http://www.3dmark.com/fs/7492760
>>>
>>> Am 12.05.2016 um 22:09 schrieb Quentin Deldycke:
>>>
>>> I will try later with sc2.
>>>
>>> I do play in ultra, as changing from this to low makes no differences =)
>>> The game works, no visual glitches. But it's performance is much more
>>> lower than it should be...
>>>
>>> --
>>> Deldycke Quentin
>>>
>>>
>>> On 12 May 2016 at 22:04, Abdulla Bubshait <darkstego at gmail.com> wrote:
>>>
>>>> Honestly, with the msrs ignore I got the game to work fine. Running on
>>>> ultra settings with no issue. This might not be an msrs issue. Try
>>>> StarCraft 2, it uses the same engine and also produces msrs faults.
>>>>
>>>> On Thu, May 12, 2016, 15:52 Quentin Deldycke <quentindeldycke at gmail.com>
>>>> wrote:
>>>>
>>>>> Same, does not work, even with the multiples binaries of the game
>>>>> (launched threw battle.net)
>>>>>
>>>>> Just for people to know it makes the card have this kind of herratic
>>>>> gpu usage during game... Which is quite a pain to play...
>>>>>
>>>>> [image: Inline images 1]
>>>>>
>>>>> Note that it became more quiet while entering menus after game end
>>>>> (after the line with the hour...)
>>>>>
>>>>> --
>>>>> Deldycke Quentin
>>>>>
>>>>>
>>>>> On 11 May 2016 at 18:04, Abdulla Bubshait < <darkstego at gmail.com>
>>>>> darkstego at gmail.com> wrote:
>>>>>
>>>>>> I have tried running it in Compatibility mode, but it didn't work.
>>>>>>
>>>>>> On Wed, May 11, 2016 at 10:04 AM Brett Peckinpaugh <
>>>>>> <bp10 at erylflynn.com>bp10 at erylflynn.com> wrote:
>>>>>>
>>>>>>> Have you tried running the game in compatibility mode set to 8.1?
>>>>>>>
>>>>>>>
>>>>>>> On May 11, 2016 5:26:19 AM PDT, Abdulla Bubshait <
>>>>>>> <darkstego at gmail.com>darkstego at gmail.com> wrote:
>>>>>>>>
>>>>>>>> StarCraft and heroes are the two games that have this problem, they
>>>>>>>> are both based on the same engine. Unfortunately I cannot test the cpu
>>>>>>>> because if I were to change my cpu to anything but "host" my machine will
>>>>>>>> not boot (I believe this is due to a problem with the AMD drivers).
>>>>>>>>
>>>>>>>> I believe the MSRs are called by Windows 10, since they are
>>>>>>>> privileged registers. Likely for debugging purposes here.
>>>>>>>>
>>>>>>>> I would suggest trying something more related tou your  cpu
>>>>>>>> ("Haswell" in my case) instead of "core2duo". I remember when I first tried
>>>>>>>> to install Win10 under virt-manager I got a KVM error about unsupported cpu
>>>>>>>> functions when I tried to clone the host cpu. Setting it to Haswell cleared
>>>>>>>> those errors so this might have something to do with it. Unfortunately that
>>>>>>>> didn't play nice with my video card so I ended up with a command line setup
>>>>>>>> as virt-manager setup refused to load AMD drivers.
>>>>>>>> Hope this helps.
>>>>>>>>
>>>>>>>> Abdulla
>>>>>>>>
>>>>>>>> On Wed, May 11, 2016, 04:03 Ivan Volosyuk <
>>>>>>>> <ivan.volosyuk at gmail.com>ivan.volosyuk at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> My 2c: I had the same issues with StarCraft crashing on Win10 due
>>>>>>>>> to the unsupported msrs (or producing lots of logs in dmesg). On Win8.1 I
>>>>>>>>> don't have this problem. My best guess is that nvidia drivers for Win10
>>>>>>>>> started to use msrs unsupported by qemu. Can you change your CPU to
>>>>>>>>> emulated core2duo and check if it actually works faster and doesn't produce
>>>>>>>>> this logging spam?
>>>>>>>>>
>>>>>>>>> On Wed, May 11, 2016 at 4:50 PM Quentin Deldycke <
>>>>>>>>> <quentindeldycke at gmail.com>quentindeldycke at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> I also play quite much this game. Adding this option makes the
>>>>>>>>>> game "Works".
>>>>>>>>>>
>>>>>>>>>> But for me, it is also the game with worst performance. As there
>>>>>>>>>> is a storm of unsupported msr (not 1 or 2 but hundreds of thousands...)
>>>>>>>>>>
>>>>>>>>>> Do you have correct performance?  I go between 120 at begging to
>>>>>>>>>> 15 during fights.
>>>>>>>>>>
>>>>>>>>>> Note that this is the only game making such mess with msr. Other
>>>>>>>>>> blizzard games works perfectly...
>>>>>>>>>> On 11 May 2016 1:04 am, "Abdulla Bubshait" <
>>>>>>>>>> <darkstego at gmail.com>darkstego at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Just put that in and it solved the problem.
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>>
>>>>>>>>>>> On Tue, May 10, 2016 at 6:54 PM Alex Williamson <
>>>>>>>>>>> <alex.l.williamson at gmail.com>alex.l.williamson at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> On Tue, May 10, 2016 at 4:40 PM, Abdulla Bubshait <
>>>>>>>>>>>> <darkstego at gmail.com>darkstego at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I have a pretty stable VFIO setup running for a while, but I
>>>>>>>>>>>>> am stuck with this
>>>>>>>>>>>>> odd problem where 1 game (heroes of the storm) keeps giving me
>>>>>>>>>>>>> a BSOD
>>>>>>>>>>>>> whenever I try to run it in the VM under Windows 10.
>>>>>>>>>>>>>
>>>>>>>>>>>>>  All other games are running fine. If I install Windows 8 in
>>>>>>>>>>>>> the VM the game runs fine.
>>>>>>>>>>>>>  If I boot the machine into the Windows 10 HDD directly the
>>>>>>>>>>>>> game runs fine.
>>>>>>>>>>>>> This crash occurs with both Nvidia GTX 770 and AMD Fury X.
>>>>>>>>>>>>> It only crashes when in VM and Windows 10.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The BSOD is some form of exception. Examples that occur
>>>>>>>>>>>>> SYSTEM_SERVICE_EXCEPTION
>>>>>>>>>>>>> KMODE_EXCEPTION_NOT_HANDLED
>>>>>>>>>>>>> SYSTEM_THREAD_EXCEPTION_NOT_HANDLED_M
>>>>>>>>>>>>> The dump files seem to suggest a windows8 driver issue,
>>>>>>>>>>>>> but I can't pinpoint any faulty driver.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am running netrunner (manjaro) kernel 4.4.9, qemu 2.5.1.
>>>>>>>>>>>>>
>>>>>>>>>>>>> My config is:
>>>>>>>>>>>>> <http://pastebin.com/W6cPyMEB>http://pastebin.com/W6cPyMEB
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sample BSOD dumps:
>>>>>>>>>>>>> <http://pastebin.com/bgh2uEhf>http://pastebin.com/bgh2uEhf
>>>>>>>>>>>>> <http://pastebin.com/SLPTVUwn>http://pastebin.com/SLPTVUwn
>>>>>>>>>>>>> <http://pastebin.com/zdjTzKuV>http://pastebin.com/zdjTzKuV
>>>>>>>>>>>>> <http://pastebin.com/8Lt5VfLg>http://pastebin.com/8Lt5VfLg
>>>>>>>>>>>>>
>>>>>>>>>>>>> Welcome any ideas to fix this problem. Thanks,
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Do you have the following set in a modprobe.d conf file?
>>>>>>>>>>>>
>>>>>>>>>>>> options kvm ignore_msrs=1
>>>>>>>>>>>>
>>>>>>>>>>>> Windows BSODs are often the result of calling an unsupported
>>>>>>>>>>>> MSR and not handling the exception.  There's some risk to this option
>>>>>>>>>>>> because zero isn't guaranteed to be a valid return for an unknown MSR, but
>>>>>>>>>>>> it seems to solve a lot of problems.  YMMV.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> vfio-users mailing list
>>>>>>>>>>> <vfio-users at redhat.com>vfio-users at redhat.com
>>>>>>>>>>> <https://www.redhat.com/mailman/listinfo/vfio-users>
>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/vfio-users
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>> vfio-users mailing list
>>>>>>>>>> <vfio-users at redhat.com>vfio-users at redhat.com
>>>>>>>>>> <https://www.redhat.com/mailman/listinfo/vfio-users>
>>>>>>>>>> https://www.redhat.com/mailman/listinfo/vfio-users
>>>>>>>>>>
>>>>>>>>> ------------------------------
>>>>>>>>
>>>>>>>> vfio-users mailing listvfio-users at redhat.comhttps://www.redhat.com/mailman/listinfo/vfio-users
>>>>>>>>
>>>>>>>> _______________________________________________
>>> vfio-users mailing listvfio-users at redhat.comhttps://www.redhat.com/mailman/listinfo/vfio-users
>>>
>>>
>>> _______________________________________________
>>> vfio-users mailing list
>>> vfio-users at redhat.com
>>> https://www.redhat.com/mailman/listinfo/vfio-users
>>>
>>> _______________________________________________
>> vfio-users mailing list
>> vfio-users at redhat.com
>> https://www.redhat.com/mailman/listinfo/vfio-users
>>
> _______________________________________________
> vfio-users mailing list
> vfio-users at redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160513/340d468d/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 6922 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160513/340d468d/attachment.png>