[vfio-users] BSOD occuring with 1 game in VM

Fri May 13 00:30:24 UTC 2016

So I tested heroes again and I am getting around 30 fps. Which is lower
than I remember it being on Windows 8.1. I tried it in native and I am
getting 80 to 90 fps which means that the game does indeed seem to suffer
from these errors.

I wonder if it is possible to actually implement these msrs in KVM. Or at
least get them to pass something that doesn't cause Windows to hammer the
system.

On Thu, May 12, 2016, 18:02 Quentin Deldycke <quentindeldycke at gmail.com>
wrote:

> Aw:
>
> You can have my xml here:
> https://github.com/qdel/scripts/blob/master/vfio/win10.xml
>
> Note that i have 6 cores allocated and on 7 there is also my emulator.
> This or leaving it alone change nothing. So i let it like this.
> My dpc latency is quite good actually.
>
> And as you can see, i virtualize "nothing", but only pass devices:
>
> - nvme ssd on pcie
> - sata controller
> - usb3 controller
> - gpu and i also use it's sound output for windows sound.
>
> Sadly, my sata controller cannot use msi, same as sound controller of gpu.
> But none seems to suffer of this. My sound is perfect and disk work
> perfectly (and in any cases, hots is on the ssd)
>
> Jan:
>
> I saw also some cases where my vm is faster than native. Weird stuff.
> For my case, it works nearly perfectly. I can even hard stress the host
> without any impact on the vm. (but using cgroups, i can still temporary
> make a make -j9 :P)
> This is typically limiting cpu execution as the game asks to reduce
> physics or stuff like this...
>
> Abdulla:
>
> Yep, sc2 make same mess, exactly same...
> Good thing is that Overwatch beta was not affected I had a steady 60fps on
> 4k with it. =)
> Diablo 2 is not affected, And Wow on basics area aren't also (no game time
> since years), same, much fps, and not a drop on gpu usage.
>
>
> Following, msr storm, mainly always same msr: 1c9 680 6c0! On intel it
> seems to be the "Performance Monitoring Unit". This snippet is on game
> menus. during party i can easly reach 1 000 000 suppression. (and it pop
> every 4-5 seconds)
>
> May 12 23:39:59 lindesk kernel: [10442.173587] kvm [12608]: vcpu5, guest
> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
> May 12 23:40:04 lindesk kernel: [10447.174529] kvm_set_msr_common: 241580
> callbacks suppressed
> May 12 23:40:04 lindesk kernel: [10447.174537] kvm [12608]: vcpu3, guest
> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
> May 12 23:40:04 lindesk kernel: [10447.174543] kvm_get_msr_common: 724760
> callbacks suppressed
> May 12 23:40:04 lindesk kernel: [10447.174545] kvm [12608]: vcpu3, guest
> rIP: 0xfffff80185f4c9fc ignored rdmsr: 0x1c9
> May 12 23:40:04 lindesk kernel: [10447.174547] kvm [12608]: vcpu3, guest
> rIP: 0xfffff80185f4ca0a ignored rdmsr: 0x680
> May 12 23:40:04 lindesk kernel: [10447.174549] kvm [12608]: vcpu3, guest
> rIP: 0xfffff80185f4ca21 ignored rdmsr: 0x6c0
> May 12 23:40:04 lindesk kernel: [10447.174556] kvm [12608]: vcpu3, guest
> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
> May 12 23:40:04 lindesk kernel: [10447.174561] kvm [12608]: vcpu3, guest
> rIP: 0xfffff80185f4c9fc ignored rdmsr: 0x1c9
> May 12 23:40:04 lindesk kernel: [10447.174564] kvm [12608]: vcpu3, guest
> rIP: 0xfffff80185f4ca0a ignored rdmsr: 0x680
> May 12 23:40:04 lindesk kernel: [10447.174567] kvm [12608]: vcpu3, guest
> rIP: 0xfffff80185f4ca21 ignored rdmsr: 0x6c0
> May 12 23:40:04 lindesk kernel: [10447.174575] kvm [12608]: vcpu3, guest
> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
> May 12 23:40:04 lindesk kernel: [10447.174592] kvm [12608]: vcpu3, guest
> rIP: 0xfffff80185f4c9fc ignored rdmsr: 0x1c9
> May 12 23:40:04 lindesk kernel: [10447.174594] kvm [12608]: vcpu3, guest
> rIP: 0xfffff80185f4ca0a ignored rdmsr: 0x680
> May 12 23:40:04 lindesk kernel: [10447.174596] kvm [12608]: vcpu3, guest
> rIP: 0xfffff80185f4ca21 ignored rdmsr: 0x6c0
> May 12 23:40:04 lindesk kernel: [10447.174603] kvm [12608]: vcpu3, guest
> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
> May 12 23:40:04 lindesk kernel: [10447.174611] kvm [12608]: vcpu3, guest
> rIP: 0xfffff80185f4c9fc ignored rdmsr: 0x1c9
> May 12 23:40:04 lindesk kernel: [10447.174619] kvm [12608]: vcpu3, guest
> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
> May 12 23:40:04 lindesk kernel: [10447.174642] kvm [12608]: vcpu3, guest
> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
> May 12 23:40:04 lindesk kernel: [10447.174735] kvm [12608]: vcpu3, guest
> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
> May 12 23:40:04 lindesk kernel: [10447.174772] kvm [12608]: vcpu3, guest
> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
> May 12 23:40:04 lindesk kernel: [10447.174797] kvm [12608]: vcpu3, guest
> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
> May 12 23:40:04 lindesk kernel: [10447.174815] kvm [12608]: vcpu3, guest
> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
> May 12 23:40:09 lindesk kernel: [10452.175786] kvm_get_msr_common: 156326
> callbacks suppressed
> May 12 23:40:09 lindesk kernel: [10452.175795] kvm [12608]: vcpu2, guest
> rIP: 0xfffff80185f4c9fc ignored rdmsr: 0x1c9
> May 12 23:40:09 lindesk kernel: [10452.175798] kvm [12608]: vcpu2, guest
> rIP: 0xfffff80185f4ca0a ignored rdmsr: 0x680
> May 12 23:40:09 lindesk kernel: [10452.175802] kvm [12608]: vcpu2, guest
> rIP: 0xfffff80185f4ca21 ignored rdmsr: 0x6c0
> May 12 23:40:09 lindesk kernel: [10452.175810] kvm_set_msr_common: 52103
> callbacks suppressed
> May 12 23:40:09 lindesk kernel: [10452.175812] kvm [12608]: vcpu2, guest
> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
> May 12 23:40:09 lindesk kernel: [10452.176286] kvm [12608]: vcpu2, guest
> rIP: 0xfffff80185f4c9fc ignored rdmsr: 0x1c9
> May 12 23:40:09 lindesk kernel: [10452.176292] kvm [12608]: vcpu2, guest
> rIP: 0xfffff80185f4ca0a ignored rdmsr: 0x680
> May 12 23:40:09 lindesk kernel: [10452.176295] kvm [12608]: vcpu2, guest
> rIP: 0xfffff80185f4ca21 ignored rdmsr: 0x6c0
> May 12 23:40:09 lindesk kernel: [10452.176304] kvm [12608]: vcpu2, guest
> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
> May 12 23:40:09 lindesk kernel: [10452.176324] kvm [12608]: vcpu2, guest
> rIP: 0xfffff80185f4c9fc ignored rdmsr: 0x1c9
> May 12 23:40:09 lindesk kernel: [10452.176326] kvm [12608]: vcpu2, guest
> rIP: 0xfffff80185f4ca0a ignored rdmsr: 0x680
> May 12 23:40:09 lindesk kernel: [10452.176328] kvm [12608]: vcpu2, guest
> rIP: 0xfffff80185f4ca21 ignored rdmsr: 0x6c0
> May 12 23:40:09 lindesk kernel: [10452.176335] kvm [12608]: vcpu2, guest
> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
> May 12 23:40:09 lindesk kernel: [10452.176785] kvm [12608]: vcpu2, guest
> rIP: 0xfffff80185f4c9fc ignored rdmsr: 0x1c9
> May 12 23:40:09 lindesk kernel: [10452.176799] kvm [12608]: vcpu2, guest
> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
> May 12 23:40:09 lindesk kernel: [10452.177201] kvm [12608]: vcpu2, guest
> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
> May 12 23:40:09 lindesk kernel: [10452.177291] kvm [12608]: vcpu2, guest
> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
> May 12 23:40:09 lindesk kernel: [10452.177318] kvm [12608]: vcpu2, guest
> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
> May 12 23:40:09 lindesk kernel: [10452.177530] kvm [12608]: vcpu2, guest
> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
> May 12 23:40:09 lindesk kernel: [10452.177793] kvm [12608]: vcpu2, guest
> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
> May 12 23:40:09 lindesk kernel: [10452.178291] kvm [12608]: vcpu2, guest
> rIP: 0xfffff80185f4c993 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop
>
>
>
> On 12 May 2016 11:12 pm, "Jan Wiele" <jan at wiele.org> wrote:
>
>> You are not alone. I have had the exact same symptoms. Wanted to build a
>> "gaming" machine with two seats for playing Heroes of the Storm.
>> Performance was bad. Very low fps especially in fights. No difference
>> between low and high graphic settings.
>>
>> Nevertheless, graphic benchmarks, like 3D mark, certified a very good GPU
>> performance: [1] VM, [2] Native (Yes, the VM performs  slightly better).
>> There, you can also see my system specs. I suppose this is some kind of CPU
>> scaling problem in HotS. :-/
>>
>> I'm very interested if you can actually find a solution :) Good luck!
>>
>> Regards,
>> Jan
>>
>> [1] http://www.3dmark.com/fs/7493163
>> [2] http://www.3dmark.com/fs/7492760
>>
>> Am 12.05.2016 um 22:09 schrieb Quentin Deldycke:
>>
>> I will try later with sc2.
>>
>> I do play in ultra, as changing from this to low makes no differences =)
>> The game works, no visual glitches. But it's performance is much more
>> lower than it should be...
>>
>> --
>> Deldycke Quentin
>>
>>
>> On 12 May 2016 at 22:04, Abdulla Bubshait <darkstego at gmail.com> wrote:
>>
>>> Honestly, with the msrs ignore I got the game to work fine. Running on
>>> ultra settings with no issue. This might not be an msrs issue. Try
>>> StarCraft 2, it uses the same engine and also produces msrs faults.
>>>
>>> On Thu, May 12, 2016, 15:52 Quentin Deldycke <quentindeldycke at gmail.com>
>>> wrote:
>>>
>>>> Same, does not work, even with the multiples binaries of the game
>>>> (launched threw battle.net)
>>>>
>>>> Just for people to know it makes the card have this kind of herratic
>>>> gpu usage during game... Which is quite a pain to play...
>>>>
>>>> [image: Inline images 1]
>>>>
>>>> Note that it became more quiet while entering menus after game end
>>>> (after the line with the hour...)
>>>>
>>>> --
>>>> Deldycke Quentin
>>>>
>>>>
>>>> On 11 May 2016 at 18:04, Abdulla Bubshait < <darkstego at gmail.com>
>>>> darkstego at gmail.com> wrote:
>>>>
>>>>> I have tried running it in Compatibility mode, but it didn't work.
>>>>>
>>>>> On Wed, May 11, 2016 at 10:04 AM Brett Peckinpaugh <
>>>>> <bp10 at erylflynn.com>bp10 at erylflynn.com> wrote:
>>>>>
>>>>>> Have you tried running the game in compatibility mode set to 8.1?
>>>>>>
>>>>>>
>>>>>> On May 11, 2016 5:26:19 AM PDT, Abdulla Bubshait <
>>>>>> <darkstego at gmail.com>darkstego at gmail.com> wrote:
>>>>>>>
>>>>>>> StarCraft and heroes are the two games that have this problem, they
>>>>>>> are both based on the same engine. Unfortunately I cannot test the cpu
>>>>>>> because if I were to change my cpu to anything but "host" my machine will
>>>>>>> not boot (I believe this is due to a problem with the AMD drivers).
>>>>>>>
>>>>>>> I believe the MSRs are called by Windows 10, since they are
>>>>>>> privileged registers. Likely for debugging purposes here.
>>>>>>>
>>>>>>> I would suggest trying something more related tou your  cpu
>>>>>>> ("Haswell" in my case) instead of "core2duo". I remember when I first tried
>>>>>>> to install Win10 under virt-manager I got a KVM error about unsupported cpu
>>>>>>> functions when I tried to clone the host cpu. Setting it to Haswell cleared
>>>>>>> those errors so this might have something to do with it. Unfortunately that
>>>>>>> didn't play nice with my video card so I ended up with a command line setup
>>>>>>> as virt-manager setup refused to load AMD drivers.
>>>>>>> Hope this helps.
>>>>>>>
>>>>>>> Abdulla
>>>>>>>
>>>>>>> On Wed, May 11, 2016, 04:03 Ivan Volosyuk <
>>>>>>> <ivan.volosyuk at gmail.com>ivan.volosyuk at gmail.com> wrote:
>>>>>>>
>>>>>>>> My 2c: I had the same issues with StarCraft crashing on Win10 due
>>>>>>>> to the unsupported msrs (or producing lots of logs in dmesg). On Win8.1 I
>>>>>>>> don't have this problem. My best guess is that nvidia drivers for Win10
>>>>>>>> started to use msrs unsupported by qemu. Can you change your CPU to
>>>>>>>> emulated core2duo and check if it actually works faster and doesn't produce
>>>>>>>> this logging spam?
>>>>>>>>
>>>>>>>> On Wed, May 11, 2016 at 4:50 PM Quentin Deldycke <
>>>>>>>> <quentindeldycke at gmail.com>quentindeldycke at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I also play quite much this game. Adding this option makes the
>>>>>>>>> game "Works".
>>>>>>>>>
>>>>>>>>> But for me, it is also the game with worst performance. As there
>>>>>>>>> is a storm of unsupported msr (not 1 or 2 but hundreds of thousands...)
>>>>>>>>>
>>>>>>>>> Do you have correct performance?  I go between 120 at begging to
>>>>>>>>> 15 during fights.
>>>>>>>>>
>>>>>>>>> Note that this is the only game making such mess with msr. Other
>>>>>>>>> blizzard games works perfectly...
>>>>>>>>> On 11 May 2016 1:04 am, "Abdulla Bubshait" < <darkstego at gmail.com>
>>>>>>>>> darkstego at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Just put that in and it solved the problem.
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>> On Tue, May 10, 2016 at 6:54 PM Alex Williamson <
>>>>>>>>>> <alex.l.williamson at gmail.com>alex.l.williamson at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> On Tue, May 10, 2016 at 4:40 PM, Abdulla Bubshait <
>>>>>>>>>>> <darkstego at gmail.com>darkstego at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I have a pretty stable VFIO setup running for a while, but I am
>>>>>>>>>>>> stuck with this
>>>>>>>>>>>> odd problem where 1 game (heroes of the storm) keeps giving me
>>>>>>>>>>>> a BSOD
>>>>>>>>>>>> whenever I try to run it in the VM under Windows 10.
>>>>>>>>>>>>
>>>>>>>>>>>>  All other games are running fine. If I install Windows 8 in
>>>>>>>>>>>> the VM the game runs fine.
>>>>>>>>>>>>  If I boot the machine into the Windows 10 HDD directly the
>>>>>>>>>>>> game runs fine.
>>>>>>>>>>>> This crash occurs with both Nvidia GTX 770 and AMD Fury X.
>>>>>>>>>>>> It only crashes when in VM and Windows 10.
>>>>>>>>>>>>
>>>>>>>>>>>> The BSOD is some form of exception. Examples that occur
>>>>>>>>>>>> SYSTEM_SERVICE_EXCEPTION
>>>>>>>>>>>> KMODE_EXCEPTION_NOT_HANDLED
>>>>>>>>>>>> SYSTEM_THREAD_EXCEPTION_NOT_HANDLED_M
>>>>>>>>>>>> The dump files seem to suggest a windows8 driver issue,
>>>>>>>>>>>> but I can't pinpoint any faulty driver.
>>>>>>>>>>>>
>>>>>>>>>>>> I am running netrunner (manjaro) kernel 4.4.9, qemu 2.5.1.
>>>>>>>>>>>>
>>>>>>>>>>>> My config is:
>>>>>>>>>>>> <http://pastebin.com/W6cPyMEB>http://pastebin.com/W6cPyMEB
>>>>>>>>>>>>
>>>>>>>>>>>> Sample BSOD dumps:
>>>>>>>>>>>> <http://pastebin.com/bgh2uEhf>http://pastebin.com/bgh2uEhf
>>>>>>>>>>>> <http://pastebin.com/SLPTVUwn>http://pastebin.com/SLPTVUwn
>>>>>>>>>>>> <http://pastebin.com/zdjTzKuV>http://pastebin.com/zdjTzKuV
>>>>>>>>>>>> <http://pastebin.com/8Lt5VfLg>http://pastebin.com/8Lt5VfLg
>>>>>>>>>>>>
>>>>>>>>>>>> Welcome any ideas to fix this problem. Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Do you have the following set in a modprobe.d conf file?
>>>>>>>>>>>
>>>>>>>>>>> options kvm ignore_msrs=1
>>>>>>>>>>>
>>>>>>>>>>> Windows BSODs are often the result of calling an unsupported MSR
>>>>>>>>>>> and not handling the exception.  There's some risk to this option because
>>>>>>>>>>> zero isn't guaranteed to be a valid return for an unknown MSR, but it seems
>>>>>>>>>>> to solve a lot of problems.  YMMV.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> vfio-users mailing list
>>>>>>>>>> <vfio-users at redhat.com>vfio-users at redhat.com
>>>>>>>>>> <https://www.redhat.com/mailman/listinfo/vfio-users>
>>>>>>>>>> https://www.redhat.com/mailman/listinfo/vfio-users
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>> vfio-users mailing list
>>>>>>>>> <vfio-users at redhat.com>vfio-users at redhat.com
>>>>>>>>> <https://www.redhat.com/mailman/listinfo/vfio-users>
>>>>>>>>> https://www.redhat.com/mailman/listinfo/vfio-users
>>>>>>>>>
>>>>>>>> ------------------------------
>>>>>>>
>>>>>>> vfio-users mailing listvfio-users at redhat.comhttps://www.redhat.com/mailman/listinfo/vfio-users
>>>>>>>
>>>>>>> _______________________________________________
>> vfio-users mailing listvfio-users at redhat.comhttps://www.redhat.com/mailman/listinfo/vfio-users
>>
>>
>> _______________________________________________
>> vfio-users mailing list
>> vfio-users at redhat.com
>> https://www.redhat.com/mailman/listinfo/vfio-users
>>
>> _______________________________________________
> vfio-users mailing list
> vfio-users at redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160513/40ac1e4c/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 6922 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160513/40ac1e4c/attachment.png>