[vfio-users] VM doesn't boot if I use GPU passthrough

Mon Jan 25 17:12:10 UTC 2016

Thanks Will. Here is my info with the guest that crashes.

Host OS Info
 ubuntu - 14.04.03
 kernel - 3.19.0-47

virsh version
 Compiled against library: libvirt 1.2.18
 Using library: libvirt 1.2.18
 Using API: QEMU 1.2.18
 Running hypervisor: QEMU 2.5.0

patches
 I did not manually apply any patches to Qemu. Built directly from source.

Guest Info
 Windows 10
 nVidia Graphics Driver 361.43

Guest Event Viewer Entry On Driver Crash
 Source - nvlddmkm
 Event ID - 14
 Info - \Device\Video3  CMDre 00000004 0000011c bad0011f 00000000 00d0011f

Guest XML - Attached

On Mon, Jan 25, 2016 at 10:18 AM Will Marler <will at wmarler.com> wrote:

> On Mon, Jan 25, 2016 at 9:07 AM, Ryan Flagler <ryan.flagler at gmail.com>
> wrote:
>
>> Will, could you tell us the following?
>>
>> What Linux distribution on host?
>>
> Arch
>
>> What kernel are you using on host?
>> What libvirt version on host?
>> What qemu version on host?
>>
> Will have to check when I'm home from work & the kids are asnooze, but
> it's whatever's latest (and I'm not using the linux-vfio-lts kernel)
>
>> What OS on guest?
>>
> Windows 10.
>
>> What nvidia graphics driver version on guest?
>>
> Again, I'll have to check. But the latest or nearly latest.
>
>> My machines gpu driver crashes constantly and I'm trying to narrow down
>> why. Thanks!
>>
> How frustrating : (. I'll also get a pastebin of my XML for you, in case
> that will help. I've been running "stable" since mid 2015. I use the quotes
> because some things tripped me up (guest machine can't "sleep," can only
> power on & power off; when host machine goes to sleep with guest running,
> on host wake-up the guest is non-responsive and 100% CPU).
>
> Will
>
>
>>
>> On Mon, Jan 25, 2016, 10:02 AM Will Marler <will at wmarler.com> wrote:
>>
>>> This is discussed in
>>> http://vfio.blogspot.com/2015/05/vfio-gpu-how-to-series-part-4-our-first.html.
>>> You have to do more than <kvm><hidden state='on'/></kvm>:
>>>
>>> "The GeForce card is nearly as easy, but we first need to work around
>>> some of the roadblocks Nvidia has put in place to prevent you from using
>>> the hardware you've purchased in the way that you desire (and by my reading
>>> conforms to the EULA for their software, but IANAL).  For this step we
>>> again need to run virsh edit on the VM.  Within the <features> section,
>>> remove everything between the <hyperv> tags, including the tags
>>> themselves.  In their place add the following tags:
>>>
>>>     <kvm>
>>>       <hidden state='on'/>
>>>     </kvm>
>>>
>>> Additionally, within the <clock> tag, find the timer named hypervclock,
>>> remove the line containing this tag completely.  Save and exit the edit
>>> session."
>>>
>>> I can confirm it works, I've been getting a lot of mileage from my
>>> passed-through 750Ti lately since getting a Steam Link :-D.
>>>
>>> On Sun, Jan 24, 2016 at 7:32 AM, Ruben Felgenhauer <
>>> 4felgenh at informatik.uni-hamburg.de> wrote:
>>>
>>>> Hi,
>>>>
>>>> finally I had time to this again. I tried out virt-manager and after a
>>>> bit of playing around with it, it /somewhat/ worked:
>>>>
>>>> The machine is at least booting. I still have a standard vga card
>>>> enabled in the virt-manager config window.
>>>> After the machine has booted, I can see that the device gets recognized
>>>> as 750ti.
>>>> However, the gpu doesn't get used, because of 'Code 43'.
>>>> Code 43 is a generic error, so any idea what it could mean in this case?
>>>>
>>>> Of course I added the <kvm><hidden state='on'/></kvm> lines at the
>>>> associated position.
>>>>
>>>> Best regards,
>>>> Ruben
>>>>
>>>>
>>>> Am 18.01.2016 um 22:27 schrieb Will Marler:
>>>>
>>>> I'm not sure what correct command-line syntax is. Have you tried using
>>>> libvirt and VirtManager to handle your VM rather than command line, and
>>>> modifying the XML rather than the command line? I think that's generally
>>>> the preferred method these days (it's certainly easier from my point of
>>>> view, and the way I got my 750 Ti to pass through).
>>>>
>>>> On Mon, Jan 18, 2016 at 11:04 AM, Ruben Felgenhauer <
>>>> 4felgenh at informatik.uni-hamburg.de> wrote:
>>>>
>>>>> Hi, Alex!
>>>>>
>>>>> Thanks for your reply!
>>>>> My GPU indeed has a seperate audio device located at 01:00.1.
>>>>>
>>>>> However, just adding -device vfio-pci,host=01:00.1 doesn't seem to do
>>>>> the trick.
>>>>> Of course the corresponding device is already blacklisted and bound to
>>>>> vfio.
>>>>>
>>>>> The Debian Wiki entry about VGA passthrough (
>>>>> <https://wiki.debian.org/VGAPassthrough>
>>>>> https://wiki.debian.org/VGAPassthrough) mentions QEMU arguments like
>>>>> "-device
>>>>> vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=...
>>>>> -device vfio-pci,host=01:00.1,bus=pcie.0" which seems to address GPUs with
>>>>> audio devices, but if I try to do something similar, the buses 'root' and
>>>>> 'pcie' couldn't be found. Maybe I missed something very important?
>>>>>
>>>>> On the same article, it says that the "HDMI soundcard [...] needs to
>>>>> be unbound from its driver":
>>>>> # echo '0000:01:00.1' | sudo tee
>>>>> /sys/bus/pci/devices/0000:01:00.1/driver/unbind
>>>>> I figured the vfio-bind script from the Arch Linux Forum thread (
>>>>> https://bbs.archlinux.org/viewtopic.php?id=162768) would do exactly
>>>>> this thing, so I didn't explicitly do so for the audio device. Is that okay?
>>>>>
>>>>> Best regards,
>>>>> Ruben
>>>>>
>>>>>
>>>>> Am 18.01.2016 um 08:31 schrieb Alexander Petrenz:
>>>>>
>>>>> Hi Ruben,
>>>>>
>>>>> I guess your 750ti also has some audio device. You should pass through
>>>>> this too. It should be something like 01:00.1. There are many command line
>>>>> examples you can find about that.
>>>>> Also I´m not quite sure, if you should remove the x-vga=on.
>>>>>
>>>>> Regards
>>>>> Alex
>>>>>
>>>>> On Sun, Jan 17, 2016 at 11:12 PM, Ruben Felgenhauer <
>>>>> <4felgenh at informatik.uni-hamburg.de>4felgenh at informatik.uni-hamburg.de
>>>>> > wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am trying to pass my nVidia GTX 750ti to my QEMU guest.
>>>>>>
>>>>>> Problem is: After the QEMU monitor pops up, nothing happens. The
>>>>>> GPU's output is dead, and the vm won't be accessible via SSH anymore, so
>>>>>> it's very likely that the VM isn't booting up at all. Also, there are no
>>>>>> error messages from QEMU on the console whatsoever which makes debugging it
>>>>>> especially hard.
>>>>>>
>>>>>> This is how I start the vm with normal vga emulation:
>>>>>> qemu-system-x86_64 -hda vm.ovl -boot c -enable-kvm -m 1024 -cpu
>>>>>> host,kvm=off -smp cores=4,threads=2 -redir tcp:5022::22
>>>>>> Everything runs fine in this case. To do the passthrough, I add this:
>>>>>> -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on -vga none
>>>>>> This brings said problems with it. I also tried out multiple
>>>>>> different combinations of -device's arguments or even adding a romfile for
>>>>>> the GPU, but none of these steps changed anything at all.
>>>>>>
>>>>>> Obviously, I am using a BIOS installation and I'm well-aware with
>>>>>> this bug: <https://bugzilla.kernel.org/show_bug.cgi?id=107561>
>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=107561, but neither
>>>>>> using less RAM (as you can see I am using 1GB now) nor switching to an
>>>>>> older Kernel changed anything about the problem. I have tried Kernel 4.1.0
>>>>>> and 4.3.0.
>>>>>>
>>>>>> Host is Debian testing with QEMU 2.5.0.
>>>>>> I tried both Debian and Windows 7 as a guest, but both are showing
>>>>>> exactly the same behaviour.
>>>>>> Mainboard is an ASUS Z87-PLUS. The 750ti is produced by ASUS aswell.
>>>>>>
>>>>>> Any idea how I could get passthrough running?
>>>>>>
>>>>>> _______________________________________________
>>>>>> vfio-users mailing list
>>>>>> vfio-users at redhat.com
>>>>>> https://www.redhat.com/mailman/listinfo/vfio-users
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> vfio-users mailing list
>>>>> vfio-users at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/vfio-users
>>>>>
>>>>>
>>>>
>>>>
>>> _______________________________________________
>>> vfio-users mailing list
>>> vfio-users at redhat.com
>>> https://www.redhat.com/mailman/listinfo/vfio-users
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160125/5da35dbd/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: win10_crashing.xml
Type: text/xml
Size: 4954 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160125/5da35dbd/attachment.xml>