[PATCH 1/4] conf: add loader type 'none'

Fri Apr 7 18:12:22 UTC 2023

On Tue, Mar 28, 2023 at 04:46:45PM -0300, Daniel Henrique Barboza wrote:
> On 3/22/23 13:37, Andrea Bolognani wrote:
> > On Wed, Mar 22, 2023 at 10:36:20AM -0300, Daniel Henrique Barboza wrote:
> > > I'm not sure if the OS overwrites the firmware when running bare metal. Usually
> > > they provide different OS images for QEMU/libvirt and bare metal systems, probably
> > > to account for these differences. Baking the firmware in the kernel like Fedora
> > > Rawhide is doing was important a few years ago when RISC-V QEMU wasn't loading
> > > the firmware by default, but now it's getting in the way.
> > >
> > > All this said, having a look at the recently updated Fedora for RISC-V wiki, the
> > > instructions for booting with libvirt and QEMU are different. libvirt [1] is using
> > > the '-bios none' attribute with 'virt-install' + a Fedora Rawhide image, but
> > > QEMU instructions [2] doesn't use '-bios none' and it's using Fedora 37.
> >
> > The libvirt instructions in that page seem to have been updated in
> > the Fedora 30 timeframe, whereas the QEMU instructions appear to be
> > current as of Fedora 37.
> >
> > > At first I don't see any particular reason of why this Fedora 37 image would work
> > > on QEMU and not on libvirt. So what t I'll do now is do some testings with libvirt
> > > and Fedora 37. If it works then this series becomes way less important.
> >
> > The libvirt instructions tell you to use
> >
> >    --boot kernel=Fedora-Developer-Rawhide-*-fw_payload-uboot-qemu-virt-smode.elf
> >    --qemu-commandline='-bios none'
> >
> > (aka -bios none -kernel foo) while the QEMU ones suggest
> >
> >    -bios u/usr/share/uboot/qemu-riscv64_spl/u-boot-spl.bin
> >    -device loader,file=u/usr/share/uboot/qemu-riscv64_spl/u-boot.itb,addr=0x80200000
> >
> > Those are completely different approaches to booting the guest OS.
> > The latter is much saner IMO, because it keeps the firmware under
> > control of the host (as is the case for SeaBIOS/edk2) instead of
> > mixing the kernel and the firmware and requiring guest-specific files
> > to be extracted from the disk image before being able to boot.
> >
> > libvirt already provides the ability to pass arbitrary paths to -bios
> > via the <loader type='rom'> element, so making the first part work
> > should just be a matter of pointing it to the correct path. We don't
> > seem to have the -device loader part wired up though, so that would
> > need to be added. Probably as an additional attribute, in the vein of
> > nvram.template? The address would have to be specified as well.
> >
> > Note that the firmware descriptor format already supports u-boot as
> > the firmware type. So in the long run ideally you'd only need to
> > specify
> >
> >    <os firmware='uboot'>
> >
> > and, assuming the uboot-images-riscv64 package is installed on the
> > host, everything should just work.
>
> So, I got the chance to try this out. We can remove the '-loader' from the
> Fedora 37 command line. This:
>
> -device loader,file=u/usr/share/uboot/qemu-riscv64_spl/u-boot.itb,addr=0x80200000
>
> Is loading the u-boot image in the address that the 'virt' machine loads the kernel.
> So in the end it's similar to:
>
> -kernel u/usr/share/uboot/qemu-riscv64_spl/u-boot.itb

Excellent insight! I've confirmed that you are correct, and that

  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='loader,file=u/usr/share/uboot/qemu-riscv64_spl/u-boot.itb,addr=0x80200000'/>
  </qemu:commandline>

can indeed be replaced with

  <kernel>u/usr/share/uboot/qemu-riscv64_spl/u-boot.itb</kernel>

without apparently affecting the VM's ability to boot.

> This means that RISC-V Fedora 37 is bootable without any libvirt changes. Similar to
> what we already do with Ubuntu.

I've spent some time on this as well. Here are my findings.

First, let's briefly mention how RISC-V VMs had to be booted a few
years back. This is described in detail at [1] and preceding
sections, but the quick version is that you would have a single file
containing (I believe) OpenSBI, u-boot and the kernel. This file
would need to be extracted from the disk image, and would then be
passed to -kernel. Having the firmware and the kernel in a single
file, which lives on the host rather than the guest, was obviously
extremely cumbersome and unsuitable for scenarios beyond
experimentation.

Today, things are looking a lot better: the firmware lives in the
host and the kernel lives in the guest, as they should. However,
there seem to be two different approaches to loading the firmware
floating around.

The first one, described in the Fedora wiki[2], involves passing two
separate files to QEMU. They both come from the uboot-images-riscv64
package, and their use looks like

  -bios /usr/share/uboot/qemu-riscv64_spl/u-boot-spl.bin
  -kernel /usr/share/uboot/qemu-riscv64_spl/u-boot.itb

As far as I understand, this is a u-boot binary, just divided in two
parts and loaded separately. They might run in different hardware
privilege contexts? I think the first file might also embed OpenSBI
somehow.

The second approach is the one described in the Ubuntu wiki[3], and
also requires passing two files to QEMU, except this time they come
from the opensbi and u-boot-qemu packages respectively. The usage
looks like

  -bios /usr/lib/riscv64-linux-gnu/opensbi/generic/fw_jump.elf
  -kernel /usr/lib/u-boot/qemu-riscv64_smode/uboot.elf

I think in this case the first file is a minimal build of OpenSBI
that likely just initializes enough hardware before handing control
to an arbitrary payload - in this case, u-boot.

There's a twist, however: the Ubuntu wiki claims that the -bios part
is not necessary with QEMU >= 7.0, and I've verified that at least
with the QEMU 7.2 package in Fedora 37 this is accurate. I believe
this is because opensbi-riscv64-generic-fw_dynamic.bin, which is
included in the QEMU package, gets loaded automatically, and this
build of OpenSBI apparently also knows how to jump ahead to where
-kernel loads u-boot.

I've tried mixing things, that is, boot the Fedora VM using the
firmware taken from Ubuntu and vice versa, and thankfully it looks
like this is perfectly fine.

In conclusion, the current version of libvirt is perfectly capable of
booting modern RISC-V guests through a combination of <loader> and
<kernel> elements.

That said, even if things nominally work, way too much handholding is
necessary. So I'm wondering how we can push things forward to make
the process more user-friendly.

The obvious first step would be for the u-boot-qemu package in Ubuntu
to ship a JSON descriptor file containing something like

  {
    "interface-types": [
      "u-boot"
    ],
    "mapping": {
      "device": "kernel",
      "filename": "/usr/lib/u-boot/qemu-riscv64_smode/uboot.elf"
    },
    "targets": [
      {
        "architecture": "riscv64",
        "machines": [
          "virt"
        ]
      }
    ]
  }

With some relatively straightforward plumbing on the libvirt side, we
should be able to use the information provided in that file to make
the simple VM configuration

  <os firmware='u-boot'>

work out of the box.

Getting the Fedora version to work would be trickier. As far as I can
tell, there's currently no spec-compliant way to describe a firmware
that requires both -bios and -kernel to be used at the same time.
Should the spec be extended? Can we get Fedora to standardize on the,
at least to an outside observer, simpler approach that Ubuntu has
adopted?

[1] https://fedoraproject.org/wiki/Architectures/RISC-V/Installing#Boot_with_libvirt
[2] https://fedoraproject.org/wiki/Architectures/RISC-V/Installing#Boot_under_QEMU
[3] https://wiki.ubuntu.com/RISC-V#Booting_with_QEMU
-- 
Andrea Bolognani / Red Hat / Virtualization