[vfio-users] VM doesn't boot if I use GPU passthrough

Tue Jan 26 03:19:55 UTC 2016

Hey Ryan,

Here are the answers to your questions:

20:06:27 will ~% uname -a
Linux haze 4.3.3-2-ARCH #1 SMP PREEMPT Wed Dec 23 20:09:18 CET 2015 x86_64
GNU/Linux
20:07:01 will ~% pacman -Q | egrep '^linux|^libvirt|^qemu'
libvirt 1.3.1-1
libvirt-glib 0.2.2-1
libvirt-python 1.3.1-1
linux 4.3.3-2
linux-api-headers 4.1.4-1
linux-firmware 20151207.bbe4917-1
qemu 2.4.1-2

And here is the pastebin to my XML file: http://pastebin.com/nB3DPkEr

As far as the guest drivers are concerned, they're the "GeForce Game Ready
Driver" version 361.43.

HTH!

On Mon, Jan 25, 2016 at 10:12 AM, Ryan Flagler <ryan.flagler at gmail.com>
wrote:

> Thanks Will. Here is my info with the guest that crashes.
>
> Host OS Info
>  ubuntu - 14.04.03
>  kernel - 3.19.0-47
>
> virsh version
>  Compiled against library: libvirt 1.2.18
>  Using library: libvirt 1.2.18
>  Using API: QEMU 1.2.18
>  Running hypervisor: QEMU 2.5.0
>
> patches
>  I did not manually apply any patches to Qemu. Built directly from source.
>
> Guest Info
>  Windows 10
>  nVidia Graphics Driver 361.43
>
> Guest Event Viewer Entry On Driver Crash
>  Source - nvlddmkm
>  Event ID - 14
>  Info - \Device\Video3  CMDre 00000004 0000011c bad0011f 00000000 00d0011f
>
> Guest XML - Attached
>
>
> On Mon, Jan 25, 2016 at 10:18 AM Will Marler <will at wmarler.com> wrote:
>
>> On Mon, Jan 25, 2016 at 9:07 AM, Ryan Flagler <ryan.flagler at gmail.com>
>> wrote:
>>
>>> Will, could you tell us the following?
>>>
>>> What Linux distribution on host?
>>>
>> Arch
>>
>>> What kernel are you using on host?
>>> What libvirt version on host?
>>> What qemu version on host?
>>>
>> Will have to check when I'm home from work & the kids are asnooze, but
>> it's whatever's latest (and I'm not using the linux-vfio-lts kernel)
>>
>>> What OS on guest?
>>>
>> Windows 10.
>>
>>> What nvidia graphics driver version on guest?
>>>
>> Again, I'll have to check. But the latest or nearly latest.
>>
>>> My machines gpu driver crashes constantly and I'm trying to narrow down
>>> why. Thanks!
>>>
>> How frustrating : (. I'll also get a pastebin of my XML for you, in case
>> that will help. I've been running "stable" since mid 2015. I use the quotes
>> because some things tripped me up (guest machine can't "sleep," can only
>> power on & power off; when host machine goes to sleep with guest running,
>> on host wake-up the guest is non-responsive and 100% CPU).
>>
>> Will
>>
>>
>>>
>>> On Mon, Jan 25, 2016, 10:02 AM Will Marler <will at wmarler.com> wrote:
>>>
>>>> This is discussed in
>>>> http://vfio.blogspot.com/2015/05/vfio-gpu-how-to-series-part-4-our-first.html.
>>>> You have to do more than <kvm><hidden state='on'/></kvm>:
>>>>
>>>> "The GeForce card is nearly as easy, but we first need to work around
>>>> some of the roadblocks Nvidia has put in place to prevent you from using
>>>> the hardware you've purchased in the way that you desire (and by my reading
>>>> conforms to the EULA for their software, but IANAL).  For this step we
>>>> again need to run virsh edit on the VM.  Within the <features> section,
>>>> remove everything between the <hyperv> tags, including the tags
>>>> themselves.  In their place add the following tags:
>>>>
>>>>     <kvm>
>>>>       <hidden state='on'/>
>>>>     </kvm>
>>>>
>>>> Additionally, within the <clock> tag, find the timer named hypervclock,
>>>> remove the line containing this tag completely.  Save and exit the edit
>>>> session."
>>>>
>>>> I can confirm it works, I've been getting a lot of mileage from my
>>>> passed-through 750Ti lately since getting a Steam Link :-D.
>>>>
>>>> On Sun, Jan 24, 2016 at 7:32 AM, Ruben Felgenhauer <
>>>> 4felgenh at informatik.uni-hamburg.de> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> finally I had time to this again. I tried out virt-manager and after a
>>>>> bit of playing around with it, it /somewhat/ worked:
>>>>>
>>>>> The machine is at least booting. I still have a standard vga card
>>>>> enabled in the virt-manager config window.
>>>>> After the machine has booted, I can see that the device gets
>>>>> recognized as 750ti.
>>>>> However, the gpu doesn't get used, because of 'Code 43'.
>>>>> Code 43 is a generic error, so any idea what it could mean in this
>>>>> case?
>>>>>
>>>>> Of course I added the <kvm><hidden state='on'/></kvm> lines at the
>>>>> associated position.
>>>>>
>>>>> Best regards,
>>>>> Ruben
>>>>>
>>>>>
>>>>> Am 18.01.2016 um 22:27 schrieb Will Marler:
>>>>>
>>>>> I'm not sure what correct command-line syntax is. Have you tried using
>>>>> libvirt and VirtManager to handle your VM rather than command line, and
>>>>> modifying the XML rather than the command line? I think that's generally
>>>>> the preferred method these days (it's certainly easier from my point of
>>>>> view, and the way I got my 750 Ti to pass through).
>>>>>
>>>>> On Mon, Jan 18, 2016 at 11:04 AM, Ruben Felgenhauer <
>>>>> 4felgenh at informatik.uni-hamburg.de> wrote:
>>>>>
>>>>>> Hi, Alex!
>>>>>>
>>>>>> Thanks for your reply!
>>>>>> My GPU indeed has a seperate audio device located at 01:00.1.
>>>>>>
>>>>>> However, just adding -device vfio-pci,host=01:00.1 doesn't seem to do
>>>>>> the trick.
>>>>>> Of course the corresponding device is already blacklisted and bound
>>>>>> to vfio.
>>>>>>
>>>>>> The Debian Wiki entry about VGA passthrough (
>>>>>> <https://wiki.debian.org/VGAPassthrough>
>>>>>> https://wiki.debian.org/VGAPassthrough) mentions QEMU arguments like
>>>>>> "-device
>>>>>> vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=...
>>>>>> -device vfio-pci,host=01:00.1,bus=pcie.0" which seems to address GPUs with
>>>>>> audio devices, but if I try to do something similar, the buses 'root' and
>>>>>> 'pcie' couldn't be found. Maybe I missed something very important?
>>>>>>
>>>>>> On the same article, it says that the "HDMI soundcard [...] needs to
>>>>>> be unbound from its driver":
>>>>>> # echo '0000:01:00.1' | sudo tee
>>>>>> /sys/bus/pci/devices/0000:01:00.1/driver/unbind
>>>>>> I figured the vfio-bind script from the Arch Linux Forum thread (
>>>>>> https://bbs.archlinux.org/viewtopic.php?id=162768) would do exactly
>>>>>> this thing, so I didn't explicitly do so for the audio device. Is that okay?
>>>>>>
>>>>>> Best regards,
>>>>>> Ruben
>>>>>>
>>>>>>
>>>>>> Am 18.01.2016 um 08:31 schrieb Alexander Petrenz:
>>>>>>
>>>>>> Hi Ruben,
>>>>>>
>>>>>> I guess your 750ti also has some audio device. You should pass
>>>>>> through this too. It should be something like 01:00.1. There are many
>>>>>> command line examples you can find about that.
>>>>>> Also I´m not quite sure, if you should remove the x-vga=on.
>>>>>>
>>>>>> Regards
>>>>>> Alex
>>>>>>
>>>>>> On Sun, Jan 17, 2016 at 11:12 PM, Ruben Felgenhauer <
>>>>>> <4felgenh at informatik.uni-hamburg.de>
>>>>>> 4felgenh at informatik.uni-hamburg.de> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am trying to pass my nVidia GTX 750ti to my QEMU guest.
>>>>>>>
>>>>>>> Problem is: After the QEMU monitor pops up, nothing happens. The
>>>>>>> GPU's output is dead, and the vm won't be accessible via SSH anymore, so
>>>>>>> it's very likely that the VM isn't booting up at all. Also, there are no
>>>>>>> error messages from QEMU on the console whatsoever which makes debugging it
>>>>>>> especially hard.
>>>>>>>
>>>>>>> This is how I start the vm with normal vga emulation:
>>>>>>> qemu-system-x86_64 -hda vm.ovl -boot c -enable-kvm -m 1024 -cpu
>>>>>>> host,kvm=off -smp cores=4,threads=2 -redir tcp:5022::22
>>>>>>> Everything runs fine in this case. To do the passthrough, I add this:
>>>>>>> -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on -vga none
>>>>>>> This brings said problems with it. I also tried out multiple
>>>>>>> different combinations of -device's arguments or even adding a romfile for
>>>>>>> the GPU, but none of these steps changed anything at all.
>>>>>>>
>>>>>>> Obviously, I am using a BIOS installation and I'm well-aware with
>>>>>>> this bug: <https://bugzilla.kernel.org/show_bug.cgi?id=107561>
>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=107561, but neither
>>>>>>> using less RAM (as you can see I am using 1GB now) nor switching to an
>>>>>>> older Kernel changed anything about the problem. I have tried Kernel 4.1.0
>>>>>>> and 4.3.0.
>>>>>>>
>>>>>>> Host is Debian testing with QEMU 2.5.0.
>>>>>>> I tried both Debian and Windows 7 as a guest, but both are showing
>>>>>>> exactly the same behaviour.
>>>>>>> Mainboard is an ASUS Z87-PLUS. The 750ti is produced by ASUS aswell.
>>>>>>>
>>>>>>> Any idea how I could get passthrough running?
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> vfio-users mailing list
>>>>>>> vfio-users at redhat.com
>>>>>>> https://www.redhat.com/mailman/listinfo/vfio-users
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> vfio-users mailing list
>>>>>> vfio-users at redhat.com
>>>>>> https://www.redhat.com/mailman/listinfo/vfio-users
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>> _______________________________________________
>>>> vfio-users mailing list
>>>> vfio-users at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/vfio-users
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160125/02a6e813/attachment.htm>