[vfio-users] Guest does not boot OS with GPU passed through, stuck at TianoCore boot splash

[Laszlo Ersek]

On 04/12/16 20:42, Stefan Seil wrote:
> As far as I can tell, the other PCI device is the HDMI audio controller
> of the graphics card. It is listed as a seperate PCI device and I
> assigned it to the VM, too.

I think that makes sense. I have the same situation with my GTX750. I
believe those two functions belong to the same IOMMU domain, so they
have to be assigned together anyway.

> I haven't explicitly added any USB devices, although it seems like there
> are some per default. I have linked the respective domain XMLs below.

Hmm no, as far as device assignment goes, you have only two <hostdev>
elements.

Regarding USB devices, you have a few emulated controllers. As far as
actual devices go, you seem to be using Spice USB redirection (see
<redirdev>). I had to read up on that:
<http://blog.wikichoon.com/2014/04/spice-usb-redirection-in-virt-manager.html>.

I can't tell if you actually redirected some host-side USB devices. I
think it's unlikely; Cole's blog post says "When connected to a VM's
graphical display", and in your first domain XML, your only display is
the assigned GPU (no <graphics> element).

> I have played around with the configuration a bit and removed the rom
> file for the HDMI audio controller of the graphics card (previously I
> had given it the same rom file as the one I passed to the GPU)

Aha! So that is why I saw two instances of AmdGop.efi.

(I think there may be a bug in AmdGop.efi. A UEFI_DRIVER must cope with
the situation when the PciIo protocol interface on the device handle is
already open by another (or the same) driver.)

> and now
> the VM actually does not stop at the TianoCore logo anymore. The monitor
> now displays the GRUB menu

Great!

> and turns black after choosing the first boot
> entry. This is the new log now: http://pastebin.com/2dxBqP6h

Yes, this one dispatches AmdGop.efi only once. I can also see
shimx64.efi and grubx64.efi in the log.

> I have also configured the VM with emulated graphics using QXL (just the
> standard configuration you get with the setup wizard). If I try to boot
> with the GPU passed through in addition to having the emulated graphics,
> I can see it progressing further on the emulated display. It looks like
> it starts booting the OS and then stops at this line:
> 
> intel_rapl: no valid rapl domains found in package 0
> 
> I don't see any of that on the monitor connected to the graphics card,
> though.

The point is, by eliminating the romfile for the HDMI audio controller,
you managed to reach grub, and grub actually started your kernel. (The
above intel_rapl message comes from "drivers/powercap/intel_rapl.c".)

>From this point on, it is a guest kernel / guest GPU driver question.
Since you do have functional grub, I recommend the following:

* Edit your domain configuration so that it has an emulated serial
console. Virtio-serial is *not* good enough. Run "virsh edit", and add
the following:

    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>

... Actually, I'm just noticing that your second XML paste, at
<http://pastebin.com/Wm3E23nW>, already contains this XML fragment. Good.

* Create the $HOME/.screenrc file, with the following content:

logfile /tmp/screen.log
logfile flush 1

* Start the VM like this, from the shell:

  # launch "screen" first
  screen

  # now press [Ctrl-A Shift-H]
  # this will start logging terminal traffic to the logfile

  # launch the guest
  virsh start --console --paused xubuntu-clone

* Wait until the line "Connected to domain xubuntu-clone" appears.

* Now open the guest in virt-manager as well, and click the Pause button
to unpause the guest.

* As soon as the grub menu appears, choose to edit the kernel command
line, under the first grub menu entry (i.e., the one that you would
normally boot).

If the "quiet" parameter is there, remove it.

Independently, add the following three parameters:

  console=tty console=ttyS0,115200 ignore_loglevel

* Boot the edited command line.

* When the guest becomes stuck again, force it off (using virt-manager),
same as before. This will also make "virsh" return in your "screen"
window, and then you can type "exit" there.

* Email the /tmp/screen.log file, with the guest kernel's debug output,
to whoever provided your guest kernel. :)

Thanks
Laszlo

> These are the two domain XMLs now:
> Without additional emulated graphics: http://pastebin.com/hBTdP6Sq
> With additional emulated graphics: http://pastebin.com/Wm3E23nW