[Libguestfs] Improving supermin appliance startup time (lkvm/qboot)

[Richard W.M. Jones]

On Sun, Mar 13, 2016 at 04:24:53PM +0200, Török Edwin wrote:
> Hi,
> 
> I remembered reading about Intel Clear Containers [1], and Hyper/Qboot [2]
> that support startup times measured in the hundreds of miliseconds range.
> 
> On an AMD FX(tm)-8350 'guestfish -a /dev/null run' takes ~4s when run the 2nd/3rd time:
> real	0m4.152s
> user	0m2.120s
> sys	0m0.564s
> 
> Are there any plans on applying these improvements to the supermin appliance?

I was looking at Clear Containers last week.

> I did some quick tests on what would be possible, see below:
> 
> If I try to use virtio 9P instead it becomes a little faster (instead of root fs on block device):
> real	0m3.299s
> user	0m2.084s
> sys	0m1.828s
> 
> To do this we need to load just the virtio and 9p modules:
> let kmods = [
> "virtio*.ko*";
> "9p*.ko*";
> ]
> Modify init.c to mount the 9p root: https://gist.github.com/anonymous/cb32986fd9b85c315ae4
> And wrap qemu to create the 9p device (the mount is just there for a quick test, in reality supermin itself should probably leave the unpacker dir around):
> https://gist.github.com/anonymous/5cdbc18974f88d6ea9e0
> 
> Next step: kvmtool [3]. Avoids some delays introduced by legacy BIOS boot, and device probing, however
> lkvm currently lacks support for additional virtio-serial channels, so - although the supermin appliance would  boot - guestfsd wouldn't be able to communicate with the host:
> https://gist.github.com/anonymous/e6b0a12e1da811f24e5b
> 
> I tried QBoot [4], which requires that you first strip down your kernel+initrd to fit into 8128k,
> and uses an unmodified Qemu otherwise.
> But then I haven't been able to get it to actually launch guestfsd, it hangs somewhere earlier:
> https://gist.github.com/anonymous/52fc2890a0884230e4f8
> Maybe I removed too many modules here?
> 
> Another possibility would be to run guestfsd with rkt+lkvm stage1, or hyper.sh; but I don't see a way
> to forward a unix socket or a virtio-serial channel through them, would guestfsd be able to work over a TCP socket
> or the main virtio console? (its probably preferable to avoid having any networking in the guestfsd appliance)
> 
> [1] https://lwn.net/Articles/644675/
> [2] https://hyper.sh/faq.html
> [3] https://git.kernel.org/cgit/linux/kernel/git/will/kvmtool.git/tree/README
> [4] https://github.com/bonzini/qboot

This is all very good analysis.

The issues that I had in brief were:

(1) We could run kvmtool, perhaps by adding a new backend, but it
seems a better idea to add the required features to qemu.  Anything
based on kvmtool wouldn't support qcow2 and the myriad other features
of qemu (see also:
https://rwmj.wordpress.com/2015/11/07/linux-kernel-library-backend-for-libguestfs/)

(2) On the kernel side, the Intel kernel contains lots of little
hacks, and many baremetal CONFIG_* options are disabled.  Hacks can be
upstreamed once we massage them into shape.  The real issue is keeping
a single baremetal + virt kernel image, since separate kernel images
would never fly with Fedora or RHEL.  That means we cannot just
disable slow drivers/subsystems by having an alternate kernel with
lots of CONFIG_* options disabled, we need to address those problems
properly.

(3) DAX / NVDIMM / etc - love them.  Not supported upstream (either
kernel or qemu) yet.

(4) Passthrough (eg 9p) filesystems.  You touched on that above.
Red Hat doesn't much like 9pfs for several reasons, yet we also don't
have a plausible alternative at the moment.  This is mainly a problem
for running fast Docker containers, rather than libguestfs though.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW