[vfio-users] VM doesn't boot if I use GPU passthrough

Tue Jan 26 17:11:05 UTC 2016

Well, you run Linux and you're experimenting with VGA passthrough ...
you're resourceful! What about picking up a 16GB SSD for $15
<http://www.amazon.com/Samsung-16GB-Solid-State-Drive/dp/B003YMJPE8/ref=sr_1_3?ie=UTF8&qid=1453827934&sr=8-3&keywords=16GB+SSD>
and
installing Arch (or Fedora, or Gentoo... whatever suits) side by side with
Ubuntu? Presumably your VM can be launched either way without any
configuration changes ... when you get tired/frustrated of the
Arch/Fedora/Gentoo way you reboot back. If it works, you've found the
answer, if it doesn't, you've improved your Linux-fu for not much
(monetary) cost.

On Tue, Jan 26, 2016 at 10:03 AM, Ryan Flagler <ryan.flagler at gmail.com>
wrote:

> Yea, that's just a major jump. Wish I had a dedicated test system to try
> more things. ;)
>
> On Tue, Jan 26, 2016 at 10:34 AM Will Marler <will at wmarler.com> wrote:
>
>> Next up would be Kernel, it sounds like...
>>
>> On Tue, Jan 26, 2016 at 8:27 AM, Ryan Flagler <ryan.flagler at gmail.com>
>> wrote:
>>
>>> Thanks for this info Will. Tried matching your qemu/libvirt versions and
>>> I still get the driver crashes. I'm not sure what else to try.
>>>
>>> On Mon, Jan 25, 2016 at 9:20 PM Will Marler <will at wmarler.com> wrote:
>>>
>>>> Hey Ryan,
>>>>
>>>> Here are the answers to your questions:
>>>>
>>>> 20:06:27 will ~% uname -a
>>>> Linux haze 4.3.3-2-ARCH #1 SMP PREEMPT Wed Dec 23 20:09:18 CET 2015
>>>> x86_64 GNU/Linux
>>>> 20:07:01 will ~% pacman -Q | egrep '^linux|^libvirt|^qemu'
>>>> libvirt 1.3.1-1
>>>> libvirt-glib 0.2.2-1
>>>> libvirt-python 1.3.1-1
>>>> linux 4.3.3-2
>>>> linux-api-headers 4.1.4-1
>>>> linux-firmware 20151207.bbe4917-1
>>>> qemu 2.4.1-2
>>>>
>>>> And here is the pastebin to my XML file: http://pastebin.com/nB3DPkEr
>>>>
>>>> As far as the guest drivers are concerned, they're the "GeForce Game
>>>> Ready Driver" version 361.43.
>>>>
>>>> HTH!
>>>>
>>>> On Mon, Jan 25, 2016 at 10:12 AM, Ryan Flagler <ryan.flagler at gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks Will. Here is my info with the guest that crashes.
>>>>>
>>>>> Host OS Info
>>>>>  ubuntu - 14.04.03
>>>>>  kernel - 3.19.0-47
>>>>>
>>>>> virsh version
>>>>>  Compiled against library: libvirt 1.2.18
>>>>>  Using library: libvirt 1.2.18
>>>>>  Using API: QEMU 1.2.18
>>>>>  Running hypervisor: QEMU 2.5.0
>>>>>
>>>>> patches
>>>>>  I did not manually apply any patches to Qemu. Built directly from
>>>>> source.
>>>>>
>>>>> Guest Info
>>>>>  Windows 10
>>>>>  nVidia Graphics Driver 361.43
>>>>>
>>>>> Guest Event Viewer Entry On Driver Crash
>>>>>  Source - nvlddmkm
>>>>>  Event ID - 14
>>>>>  Info - \Device\Video3  CMDre 00000004 0000011c bad0011f 00000000
>>>>> 00d0011f
>>>>>
>>>>> Guest XML - Attached
>>>>>
>>>>>
>>>>> On Mon, Jan 25, 2016 at 10:18 AM Will Marler <will at wmarler.com> wrote:
>>>>>
>>>>>> On Mon, Jan 25, 2016 at 9:07 AM, Ryan Flagler <ryan.flagler at gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Will, could you tell us the following?
>>>>>>>
>>>>>>> What Linux distribution on host?
>>>>>>>
>>>>>> Arch
>>>>>>
>>>>>>> What kernel are you using on host?
>>>>>>> What libvirt version on host?
>>>>>>> What qemu version on host?
>>>>>>>
>>>>>> Will have to check when I'm home from work & the kids are asnooze,
>>>>>> but it's whatever's latest (and I'm not using the linux-vfio-lts kernel)
>>>>>>
>>>>>>> What OS on guest?
>>>>>>>
>>>>>> Windows 10.
>>>>>>
>>>>>>> What nvidia graphics driver version on guest?
>>>>>>>
>>>>>> Again, I'll have to check. But the latest or nearly latest.
>>>>>>
>>>>>>> My machines gpu driver crashes constantly and I'm trying to narrow
>>>>>>> down why. Thanks!
>>>>>>>
>>>>>> How frustrating : (. I'll also get a pastebin of my XML for you, in
>>>>>> case that will help. I've been running "stable" since mid 2015. I use the
>>>>>> quotes because some things tripped me up (guest machine can't "sleep," can
>>>>>> only power on & power off; when host machine goes to sleep with guest
>>>>>> running, on host wake-up the guest is non-responsive and 100% CPU).
>>>>>>
>>>>>> Will
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> On Mon, Jan 25, 2016, 10:02 AM Will Marler <will at wmarler.com> wrote:
>>>>>>>
>>>>>>>> This is discussed in
>>>>>>>> http://vfio.blogspot.com/2015/05/vfio-gpu-how-to-series-part-4-our-first.html.
>>>>>>>> You have to do more than <kvm><hidden state='on'/></kvm>:
>>>>>>>>
>>>>>>>> "The GeForce card is nearly as easy, but we first need to work
>>>>>>>> around some of the roadblocks Nvidia has put in place to prevent you from
>>>>>>>> using the hardware you've purchased in the way that you desire (and by my
>>>>>>>> reading conforms to the EULA for their software, but IANAL).  For this step
>>>>>>>> we again need to run virsh edit on the VM.  Within the <features> section,
>>>>>>>> remove everything between the <hyperv> tags, including the tags
>>>>>>>> themselves.  In their place add the following tags:
>>>>>>>>
>>>>>>>>     <kvm>
>>>>>>>>       <hidden state='on'/>
>>>>>>>>     </kvm>
>>>>>>>>
>>>>>>>> Additionally, within the <clock> tag, find the timer named
>>>>>>>> hypervclock, remove the line containing this tag completely.  Save and exit
>>>>>>>> the edit session."
>>>>>>>>
>>>>>>>> I can confirm it works, I've been getting a lot of mileage from my
>>>>>>>> passed-through 750Ti lately since getting a Steam Link :-D.
>>>>>>>>
>>>>>>>> On Sun, Jan 24, 2016 at 7:32 AM, Ruben Felgenhauer <
>>>>>>>> 4felgenh at informatik.uni-hamburg.de> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> finally I had time to this again. I tried out virt-manager and
>>>>>>>>> after a bit of playing around with it, it /somewhat/ worked:
>>>>>>>>>
>>>>>>>>> The machine is at least booting. I still have a standard vga card
>>>>>>>>> enabled in the virt-manager config window.
>>>>>>>>> After the machine has booted, I can see that the device gets
>>>>>>>>> recognized as 750ti.
>>>>>>>>> However, the gpu doesn't get used, because of 'Code 43'.
>>>>>>>>> Code 43 is a generic error, so any idea what it could mean in this
>>>>>>>>> case?
>>>>>>>>>
>>>>>>>>> Of course I added the <kvm><hidden state='on'/></kvm> lines at the
>>>>>>>>> associated position.
>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>> Ruben
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Am 18.01.2016 um 22:27 schrieb Will Marler:
>>>>>>>>>
>>>>>>>>> I'm not sure what correct command-line syntax is. Have you tried
>>>>>>>>> using libvirt and VirtManager to handle your VM rather than command line,
>>>>>>>>> and modifying the XML rather than the command line? I think that's
>>>>>>>>> generally the preferred method these days (it's certainly easier from my
>>>>>>>>> point of view, and the way I got my 750 Ti to pass through).
>>>>>>>>>
>>>>>>>>> On Mon, Jan 18, 2016 at 11:04 AM, Ruben Felgenhauer <
>>>>>>>>> 4felgenh at informatik.uni-hamburg.de> wrote:
>>>>>>>>>
>>>>>>>>>> Hi, Alex!
>>>>>>>>>>
>>>>>>>>>> Thanks for your reply!
>>>>>>>>>> My GPU indeed has a seperate audio device located at 01:00.1.
>>>>>>>>>>
>>>>>>>>>> However, just adding -device vfio-pci,host=01:00.1 doesn't seem
>>>>>>>>>> to do the trick.
>>>>>>>>>> Of course the corresponding device is already blacklisted and
>>>>>>>>>> bound to vfio.
>>>>>>>>>>
>>>>>>>>>> The Debian Wiki entry about VGA passthrough (
>>>>>>>>>> <https://wiki.debian.org/VGAPassthrough>
>>>>>>>>>> https://wiki.debian.org/VGAPassthrough) mentions QEMU arguments
>>>>>>>>>> like "-device
>>>>>>>>>> vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=...
>>>>>>>>>> -device vfio-pci,host=01:00.1,bus=pcie.0" which seems to address GPUs with
>>>>>>>>>> audio devices, but if I try to do something similar, the buses 'root' and
>>>>>>>>>> 'pcie' couldn't be found. Maybe I missed something very important?
>>>>>>>>>>
>>>>>>>>>> On the same article, it says that the "HDMI soundcard [...] needs
>>>>>>>>>> to be unbound from its driver":
>>>>>>>>>> # echo '0000:01:00.1' | sudo tee
>>>>>>>>>> /sys/bus/pci/devices/0000:01:00.1/driver/unbind
>>>>>>>>>> I figured the vfio-bind script from the Arch Linux Forum thread (
>>>>>>>>>> https://bbs.archlinux.org/viewtopic.php?id=162768) would do
>>>>>>>>>> exactly this thing, so I didn't explicitly do so for the audio device. Is
>>>>>>>>>> that okay?
>>>>>>>>>>
>>>>>>>>>> Best regards,
>>>>>>>>>> Ruben
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Am 18.01.2016 um 08:31 schrieb Alexander Petrenz:
>>>>>>>>>>
>>>>>>>>>> Hi Ruben,
>>>>>>>>>>
>>>>>>>>>> I guess your 750ti also has some audio device. You should pass
>>>>>>>>>> through this too. It should be something like 01:00.1. There are many
>>>>>>>>>> command line examples you can find about that.
>>>>>>>>>> Also I´m not quite sure, if you should remove the x-vga=on.
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>> Alex
>>>>>>>>>>
>>>>>>>>>> On Sun, Jan 17, 2016 at 11:12 PM, Ruben Felgenhauer <
>>>>>>>>>> <4felgenh at informatik.uni-hamburg.de>
>>>>>>>>>> 4felgenh at informatik.uni-hamburg.de> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I am trying to pass my nVidia GTX 750ti to my QEMU guest.
>>>>>>>>>>>
>>>>>>>>>>> Problem is: After the QEMU monitor pops up, nothing happens. The
>>>>>>>>>>> GPU's output is dead, and the vm won't be accessible via SSH anymore, so
>>>>>>>>>>> it's very likely that the VM isn't booting up at all. Also, there are no
>>>>>>>>>>> error messages from QEMU on the console whatsoever which makes debugging it
>>>>>>>>>>> especially hard.
>>>>>>>>>>>
>>>>>>>>>>> This is how I start the vm with normal vga emulation:
>>>>>>>>>>> qemu-system-x86_64 -hda vm.ovl -boot c -enable-kvm -m 1024 -cpu
>>>>>>>>>>> host,kvm=off -smp cores=4,threads=2 -redir tcp:5022::22
>>>>>>>>>>> Everything runs fine in this case. To do the passthrough, I add
>>>>>>>>>>> this:
>>>>>>>>>>> -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on -vga none
>>>>>>>>>>> This brings said problems with it. I also tried out multiple
>>>>>>>>>>> different combinations of -device's arguments or even adding a romfile for
>>>>>>>>>>> the GPU, but none of these steps changed anything at all.
>>>>>>>>>>>
>>>>>>>>>>> Obviously, I am using a BIOS installation and I'm well-aware
>>>>>>>>>>> with this bug:
>>>>>>>>>>> <https://bugzilla.kernel.org/show_bug.cgi?id=107561>
>>>>>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=107561, but neither
>>>>>>>>>>> using less RAM (as you can see I am using 1GB now) nor switching to an
>>>>>>>>>>> older Kernel changed anything about the problem. I have tried Kernel 4.1.0
>>>>>>>>>>> and 4.3.0.
>>>>>>>>>>>
>>>>>>>>>>> Host is Debian testing with QEMU 2.5.0.
>>>>>>>>>>> I tried both Debian and Windows 7 as a guest, but both are
>>>>>>>>>>> showing exactly the same behaviour.
>>>>>>>>>>> Mainboard is an ASUS Z87-PLUS. The 750ti is produced by ASUS
>>>>>>>>>>> aswell.
>>>>>>>>>>>
>>>>>>>>>>> Any idea how I could get passthrough running?
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> vfio-users mailing list
>>>>>>>>>>> vfio-users at redhat.com
>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/vfio-users
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> vfio-users mailing list
>>>>>>>>>> vfio-users at redhat.com
>>>>>>>>>> https://www.redhat.com/mailman/listinfo/vfio-users
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> vfio-users mailing list
>>>>>>>> vfio-users at redhat.com
>>>>>>>> https://www.redhat.com/mailman/listinfo/vfio-users
>>>>>>>>
>>>>>>>
>>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160126/2717a7d4/attachment.htm>