[vfio-users] VM doesn't boot if I use GPU passthrough

Tue Jan 26 15:27:25 UTC 2016

Thanks for this info Will. Tried matching your qemu/libvirt versions and I
still get the driver crashes. I'm not sure what else to try.

On Mon, Jan 25, 2016 at 9:20 PM Will Marler <will at wmarler.com> wrote:

> Hey Ryan,
>
> Here are the answers to your questions:
>
> 20:06:27 will ~% uname -a
> Linux haze 4.3.3-2-ARCH #1 SMP PREEMPT Wed Dec 23 20:09:18 CET 2015 x86_64
> GNU/Linux
> 20:07:01 will ~% pacman -Q | egrep '^linux|^libvirt|^qemu'
> libvirt 1.3.1-1
> libvirt-glib 0.2.2-1
> libvirt-python 1.3.1-1
> linux 4.3.3-2
> linux-api-headers 4.1.4-1
> linux-firmware 20151207.bbe4917-1
> qemu 2.4.1-2
>
> And here is the pastebin to my XML file: http://pastebin.com/nB3DPkEr
>
> As far as the guest drivers are concerned, they're the "GeForce Game Ready
> Driver" version 361.43.
>
> HTH!
>
> On Mon, Jan 25, 2016 at 10:12 AM, Ryan Flagler <ryan.flagler at gmail.com>
> wrote:
>
>> Thanks Will. Here is my info with the guest that crashes.
>>
>> Host OS Info
>>  ubuntu - 14.04.03
>>  kernel - 3.19.0-47
>>
>> virsh version
>>  Compiled against library: libvirt 1.2.18
>>  Using library: libvirt 1.2.18
>>  Using API: QEMU 1.2.18
>>  Running hypervisor: QEMU 2.5.0
>>
>> patches
>>  I did not manually apply any patches to Qemu. Built directly from source.
>>
>> Guest Info
>>  Windows 10
>>  nVidia Graphics Driver 361.43
>>
>> Guest Event Viewer Entry On Driver Crash
>>  Source - nvlddmkm
>>  Event ID - 14
>>  Info - \Device\Video3  CMDre 00000004 0000011c bad0011f 00000000 00d0011f
>>
>> Guest XML - Attached
>>
>>
>> On Mon, Jan 25, 2016 at 10:18 AM Will Marler <will at wmarler.com> wrote:
>>
>>> On Mon, Jan 25, 2016 at 9:07 AM, Ryan Flagler <ryan.flagler at gmail.com>
>>> wrote:
>>>
>>>> Will, could you tell us the following?
>>>>
>>>> What Linux distribution on host?
>>>>
>>> Arch
>>>
>>>> What kernel are you using on host?
>>>> What libvirt version on host?
>>>> What qemu version on host?
>>>>
>>> Will have to check when I'm home from work & the kids are asnooze, but
>>> it's whatever's latest (and I'm not using the linux-vfio-lts kernel)
>>>
>>>> What OS on guest?
>>>>
>>> Windows 10.
>>>
>>>> What nvidia graphics driver version on guest?
>>>>
>>> Again, I'll have to check. But the latest or nearly latest.
>>>
>>>> My machines gpu driver crashes constantly and I'm trying to narrow down
>>>> why. Thanks!
>>>>
>>> How frustrating : (. I'll also get a pastebin of my XML for you, in case
>>> that will help. I've been running "stable" since mid 2015. I use the quotes
>>> because some things tripped me up (guest machine can't "sleep," can only
>>> power on & power off; when host machine goes to sleep with guest running,
>>> on host wake-up the guest is non-responsive and 100% CPU).
>>>
>>> Will
>>>
>>>
>>>>
>>>> On Mon, Jan 25, 2016, 10:02 AM Will Marler <will at wmarler.com> wrote:
>>>>
>>>>> This is discussed in
>>>>> http://vfio.blogspot.com/2015/05/vfio-gpu-how-to-series-part-4-our-first.html.
>>>>> You have to do more than <kvm><hidden state='on'/></kvm>:
>>>>>
>>>>> "The GeForce card is nearly as easy, but we first need to work around
>>>>> some of the roadblocks Nvidia has put in place to prevent you from using
>>>>> the hardware you've purchased in the way that you desire (and by my reading
>>>>> conforms to the EULA for their software, but IANAL).  For this step we
>>>>> again need to run virsh edit on the VM.  Within the <features> section,
>>>>> remove everything between the <hyperv> tags, including the tags
>>>>> themselves.  In their place add the following tags:
>>>>>
>>>>>     <kvm>
>>>>>       <hidden state='on'/>
>>>>>     </kvm>
>>>>>
>>>>> Additionally, within the <clock> tag, find the timer named
>>>>> hypervclock, remove the line containing this tag completely.  Save and exit
>>>>> the edit session."
>>>>>
>>>>> I can confirm it works, I've been getting a lot of mileage from my
>>>>> passed-through 750Ti lately since getting a Steam Link :-D.
>>>>>
>>>>> On Sun, Jan 24, 2016 at 7:32 AM, Ruben Felgenhauer <
>>>>> 4felgenh at informatik.uni-hamburg.de> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> finally I had time to this again. I tried out virt-manager and after
>>>>>> a bit of playing around with it, it /somewhat/ worked:
>>>>>>
>>>>>> The machine is at least booting. I still have a standard vga card
>>>>>> enabled in the virt-manager config window.
>>>>>> After the machine has booted, I can see that the device gets
>>>>>> recognized as 750ti.
>>>>>> However, the gpu doesn't get used, because of 'Code 43'.
>>>>>> Code 43 is a generic error, so any idea what it could mean in this
>>>>>> case?
>>>>>>
>>>>>> Of course I added the <kvm><hidden state='on'/></kvm> lines at the
>>>>>> associated position.
>>>>>>
>>>>>> Best regards,
>>>>>> Ruben
>>>>>>
>>>>>>
>>>>>> Am 18.01.2016 um 22:27 schrieb Will Marler:
>>>>>>
>>>>>> I'm not sure what correct command-line syntax is. Have you tried
>>>>>> using libvirt and VirtManager to handle your VM rather than command line,
>>>>>> and modifying the XML rather than the command line? I think that's
>>>>>> generally the preferred method these days (it's certainly easier from my
>>>>>> point of view, and the way I got my 750 Ti to pass through).
>>>>>>
>>>>>> On Mon, Jan 18, 2016 at 11:04 AM, Ruben Felgenhauer <
>>>>>> 4felgenh at informatik.uni-hamburg.de> wrote:
>>>>>>
>>>>>>> Hi, Alex!
>>>>>>>
>>>>>>> Thanks for your reply!
>>>>>>> My GPU indeed has a seperate audio device located at 01:00.1.
>>>>>>>
>>>>>>> However, just adding -device vfio-pci,host=01:00.1 doesn't seem to
>>>>>>> do the trick.
>>>>>>> Of course the corresponding device is already blacklisted and bound
>>>>>>> to vfio.
>>>>>>>
>>>>>>> The Debian Wiki entry about VGA passthrough (
>>>>>>> <https://wiki.debian.org/VGAPassthrough>
>>>>>>> https://wiki.debian.org/VGAPassthrough) mentions QEMU arguments
>>>>>>> like "-device
>>>>>>> vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=...
>>>>>>> -device vfio-pci,host=01:00.1,bus=pcie.0" which seems to address GPUs with
>>>>>>> audio devices, but if I try to do something similar, the buses 'root' and
>>>>>>> 'pcie' couldn't be found. Maybe I missed something very important?
>>>>>>>
>>>>>>> On the same article, it says that the "HDMI soundcard [...] needs to
>>>>>>> be unbound from its driver":
>>>>>>> # echo '0000:01:00.1' | sudo tee
>>>>>>> /sys/bus/pci/devices/0000:01:00.1/driver/unbind
>>>>>>> I figured the vfio-bind script from the Arch Linux Forum thread (
>>>>>>> https://bbs.archlinux.org/viewtopic.php?id=162768) would do exactly
>>>>>>> this thing, so I didn't explicitly do so for the audio device. Is that okay?
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Ruben
>>>>>>>
>>>>>>>
>>>>>>> Am 18.01.2016 um 08:31 schrieb Alexander Petrenz:
>>>>>>>
>>>>>>> Hi Ruben,
>>>>>>>
>>>>>>> I guess your 750ti also has some audio device. You should pass
>>>>>>> through this too. It should be something like 01:00.1. There are many
>>>>>>> command line examples you can find about that.
>>>>>>> Also I´m not quite sure, if you should remove the x-vga=on.
>>>>>>>
>>>>>>> Regards
>>>>>>> Alex
>>>>>>>
>>>>>>> On Sun, Jan 17, 2016 at 11:12 PM, Ruben Felgenhauer <
>>>>>>> <4felgenh at informatik.uni-hamburg.de>
>>>>>>> 4felgenh at informatik.uni-hamburg.de> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I am trying to pass my nVidia GTX 750ti to my QEMU guest.
>>>>>>>>
>>>>>>>> Problem is: After the QEMU monitor pops up, nothing happens. The
>>>>>>>> GPU's output is dead, and the vm won't be accessible via SSH anymore, so
>>>>>>>> it's very likely that the VM isn't booting up at all. Also, there are no
>>>>>>>> error messages from QEMU on the console whatsoever which makes debugging it
>>>>>>>> especially hard.
>>>>>>>>
>>>>>>>> This is how I start the vm with normal vga emulation:
>>>>>>>> qemu-system-x86_64 -hda vm.ovl -boot c -enable-kvm -m 1024 -cpu
>>>>>>>> host,kvm=off -smp cores=4,threads=2 -redir tcp:5022::22
>>>>>>>> Everything runs fine in this case. To do the passthrough, I add
>>>>>>>> this:
>>>>>>>> -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on -vga none
>>>>>>>> This brings said problems with it. I also tried out multiple
>>>>>>>> different combinations of -device's arguments or even adding a romfile for
>>>>>>>> the GPU, but none of these steps changed anything at all.
>>>>>>>>
>>>>>>>> Obviously, I am using a BIOS installation and I'm well-aware with
>>>>>>>> this bug: <https://bugzilla.kernel.org/show_bug.cgi?id=107561>
>>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=107561, but neither
>>>>>>>> using less RAM (as you can see I am using 1GB now) nor switching to an
>>>>>>>> older Kernel changed anything about the problem. I have tried Kernel 4.1.0
>>>>>>>> and 4.3.0.
>>>>>>>>
>>>>>>>> Host is Debian testing with QEMU 2.5.0.
>>>>>>>> I tried both Debian and Windows 7 as a guest, but both are showing
>>>>>>>> exactly the same behaviour.
>>>>>>>> Mainboard is an ASUS Z87-PLUS. The 750ti is produced by ASUS aswell.
>>>>>>>>
>>>>>>>> Any idea how I could get passthrough running?
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> vfio-users mailing list
>>>>>>>> vfio-users at redhat.com
>>>>>>>> https://www.redhat.com/mailman/listinfo/vfio-users
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> vfio-users mailing list
>>>>>>> vfio-users at redhat.com
>>>>>>> https://www.redhat.com/mailman/listinfo/vfio-users
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> vfio-users mailing list
>>>>> vfio-users at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/vfio-users
>>>>>
>>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160126/c77fcc93/attachment.htm>