[vfio-users] Windows 10 guest refuses to boot when passed my NVIDIA graphics card

[Alex Williamson]

On Tue, Sep 15, 2015 at 1:57 PM, Jasen Borisov <tajjada at gmail.com> wrote:

>
> On Tue, Sep 15, 2015 at 10:34 PM, Alex Williamson <
> alex.l.williamson at gmail.com> wrote:
>
>> On Tue, Sep 15, 2015 at 9:32 AM, Jasen Borisov <tajjada at gmail.com> wrote:
>>
>>> Hello everyone,
>>>
>>> I have been following the guides on the vfio blogspot for setting up a
>>> Windows guest for gaming by passing it my NVIDIA GeForce GTX 980 graphics
>>> card. I have an AMD Radeon R7 250 graphics card for my Linux host. I have
>>> ran into a strange problem and have not managed to find a solution to it
>>> online, so I decided to come here and see if anyone knows anything about it.
>>>
>>> Here is the problem I am facing:
>>>
>>> I installed the Windows guest (Windows 10 Pro) in a kvm virtual machine
>>> made with virt-manager, using the default qxl/spice setup during the
>>> installation, as instructed here
>>> <http://vfio.blogspot.bg/2015/05/vfio-gpu-how-to-series-part-4-our-first.html>.
>>> After the installation completed, I removed any extra virtual devices and
>>> added the PCI host/vfio device for my NVIDIA graphics card. I also edited
>>> the XML to hide the kvm virtualization, in preparation for installing the
>>> NVIDIA drivers on Windows, when I boot with my actual graphics card. I have
>>> made no attempt to configure hugepages yet.
>>>
>>> However, when starting up the VM, Windows no longer booted successfully.
>>> I saw the Tianocore splash on my physical monitor, followed by the Windows
>>> logo. The Windows logo stayed frozen (without the spinning loading
>>> indicator under it) for a few minutes, and then the spinning-dots loading
>>> indicator appeared for a few seconds and the machine *reset*. It kept
>>> resetting again and again, never getting past the Windows boot screen.
>>>
>>> I would assume that my PCI passthrough worked successfully, since my
>>> physical monitor turned on and I saw the video output from the virtual
>>> machine on it. However, I can't figure out why Windows cannot finish
>>> booting successfully. Any help with this issue would be greatly appreciated.
>>>
>>> The same copy of Windows (or rather, installed from the same
>>> installation ISO) works fine with my NVIDIA card on my actual hardware /
>>> when not in a virtual machine, so I am sure this is not a hardware problem.
>>>
>>>
>>> Here is some relevant information about my system:
>>>
>>> /proc/cmdline:
>>> ... intel_iommu=on iommu=pt vfio-pci.ids=10de:13c0,10de:0fbb,8086:8d26
>>> vfio.disable_vga=1 ...
>>>
>>
>> disable_vga is a vfio-pci option, vfio-pci.disable_vga=1
>>
>
> Ok, thanks, will change that. Not sure if it matters though, since both
> the host and guest are booting with UEFI.
>

I think you still want it, but it's probably not going to change anything
here.


> (I omitted my rootfs and other irrelevant kernel options)
>>> The PCI IDs listed in my kernel commandline are my GPU, its audio, and
>>> an EHCI controller on my system for USB, in that order.
>>>
>>> I did not need to use any "hacks" like the ACS override patch or the
>>> enable_unsafe_interrupts option, since I did not experience the relevant
>>> issues that they were made to fix. Each one of the PCI IDs above is in its
>>> own IOMMU group.
>>>
>>
>> Are you sure about that?  The GPU and audio always share a group as far
>> as I've seen.
>>
>>
>
> Ah yes, that is true, the gpu and audio are the same group, but I am
> passing them both to the VM, so I don't need to split them, hence no need
> for ACS. What I meant is that my gpu/audio is a separate group from
> everything else, and so is the EHCI controller I wanted to pass.
>

Ok, the grouping would be suspect if GPU and audio were actually separate.


> My system has no integrated graphics (Core i7 Extreme 5960X CPU).
>>>
>>
>> The XML indicates you're running at least QEMU 2.4, but what kernel
>> version are you using?  I'll guess at least v4.1 based on vfio commandline
>> options, but it'd be nice to know.
>>
>>
>
> Kernel 4.2.0, custom build/config.
>
>
>> These lines appeared in dmesg when the VM started:
>>>
>>> [  582.886435] kvm: SMP vm created on host with unstable TSC; guest TSC
>>> will not be reliable
>>>
>>
>> I'd be a little concerned by this if I were you.
>>
>> Why? What problems could it lead to / why is it concerning? Is there
> anything I can do about it? I googled that message when I first saw it, and
> I was left with the impression that it is a hardware issue that I can do
> nothing about, so I just have to deal with it and hope everything will
> work. I thought the worst thing it can cause is jitter/stuttering in the
> VM. Perhaps I am wrong?
>
>
>> Are you using "options kvm ignore_msrs=1" in modprobe.d?
>>
>
> No, I am not. Should I be using that?
>
> (also, kvm is builtin on my kernel, so, if yes, will have to add that to
> kernel commandline instead of modprobe.d)
>

I don't know if it will make a difference here, but I use it.  I think it
was Passmark PerformanceTest that won't work without it.  I'd be surprised
to see it affect Linux though.

You say that this boot problem happens even if you only assign the EHCI
controller, could assigning the EHCI controller be the problem?  Have you
tried only assigning the GPU?  Where does Linux hang?  Does it get past the
TianoCore screen?  I've had problems assigning an ASMedia USB3 controller
where it seemed like it hung in OVMF, but attributed that to the generally
crappy nature of ASMedia hardware (and an NEC controller worked fine).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20150915/57e40284/attachment.htm>