[Fedora-livecd-list] boot speed optomization (Qs for DZ mainly)

Mon Apr 2 05:33:23 UTC 2007

Hey,

On Sun, 2007-04-01 at 18:08 -0700, Jane Dogalt wrote:
> David,
> 
> I just read the historic readahead post you referenced.  I haven't
> really spent any effort thinking about massaging readahead into a boot
> speed optomization for a livecd (I'm guessing you have?).
> 
> But it happens to be very similar to the 'early boot file seperation'
> methods I've described here relating to livecds.  

Yea, there's a bunch of similarities between livecd's and real systems;
typically they share at least optimizations...

> I have a couple
> questions about your 2004 post-
> 
> "
> I've made the following observations
> 
> 1. The kernel patch, linux-2.6.3-printopen.patch, wasn't really working
>    well for me - it reported far to few files - instead I added a
>    printk() to fs/namei.c:link_path_walk() 
>    (disclaimer: I don't know much about the kernel so there may be a
>    better solution than this). 
> "
> 
> The method that I was planning on implementing, was to boot a
> livecd(-to-be) under qemu, then use file access times to construct a
> list of files that should be precached (same thing as readahead
> afaict).
> (And then by hand figure out files that are specific to the hardware
> platform, and make sure that their cousins get included in the list)
> 
> What are your thoughts on this?

I think it's the right approach - I wouldn't limit it to livecd fwiw;
one thing that's been discussed in various forums is to build/ship a set
of SystemTap scripts along with each kernel + tweak /etc/rc.sysinit
and/or the initramfs such that it recognizes an option "boot_profile" or
something. The result would be that a SystemTap script would run and
hook into e.g. sys_open() and record (access_time, filename). 

For nice UI polish, you could ship a small UI program that docks in the
notification area and upon login gives you a notification

 +------------------------------------------------+
 | Your system is being profiled. Click the "Stop |
 | Profiling" button to analyze the results.      |
 |                               [Stop Profiling] |
 +------------------------------------------------+

and then you would get a nice chart a'la

 http://bootchart.org/images/bootchart.sortreadahead.png

and also a file you can feed into readahead.

>   Any quick reason why this wasn't used
> for readahead (I'm fairly ignorant of the internals of readahead).
>
> And do any of the reasons why it wasn't used for readahead apply to
> livecd optomization case?

I think it's simply because the readahead RPM always contained an
out-of-date list of files - it's a bit hard, for example with firefox,
the files move around with every release (try "rpm -ql firefox"). Plus
readahead itself wasn't really that clever...

>From what I gather from Karel Zak (current readahead maintainer) is that
he have fixed readahead to use the directory, /etc/readahead.d, I think,
that applications can drop files into. I also recall a thread where he
wanted to use the audit subsystem rather than SystemTap scripts to e.g.
intercept sys_open(). I guess both would work.

Personally, fwiw, I think it's a dead end to have a list of files
provided by packages; it needs to be generated when significant changes
are made to the OS; e.g. when packages are installed/removed/updated as
well as on system install time (e.g. the first time you boot into the
installed OS) and, by extension, at livecd creation time.

> FYI- I see 2 obvious ways to do readahead in the livecd case.  One,
> which I've outlined here before, would be to use unionfs, and seperate
> the filesystem images into "files that should be read ahead" and "the
> rest".  Obviously unionfs, while it may eventually get into fedora,
> isn't the way to go.

Agreed.

> Its only a minor modifcation to do the same thing with devicemapper
> snapshots however.  (i.e. you create the filesystem populating it with
> the files to be read-ahead, then you dm-snapshot it, and copy the rest
> of the files in.  The resulting original fsimage(sparse ext3) along
> with the devicemapper snapshot device (damage/cow) image file,
> constitute the same effective thing as above with unionfs.
> 
> Thus all your early read-ahead files, go in a single file/track on your
> cd/dvd, reducing seek times.
> 
> Additionally, you can just read the read-ahead section into ram in one
> big chunk.  (that would waste some ram, however with md-mirror tricks,
> you can work around that)

That's an interesting idea but notably this wouldn't help on an
installed system since they way we install is dd'ing the ext3 file
system. And it would be very nice to have this on the installed system
as well (although over time it would deteriorate as packages are
updated). Especially as it might motivate people to adopt this for
mainline Fedora.

Anyway, specifically for live cd, you may also get away with the hack I
mentioned in my 2004 post; e.g. once you have an ext3 image + list of
readahead files (that you get from profiling via qemu/kvm), just create
a new ext3 image and first copy all the readahead files and then the
rest. I would be happy to take a patch to livecd-tools that does exactly
this.

(Beware of symlinks btw! E.g. surely all files in /etc/init.d/* will be
in the list but you need to create /etc/rc.d/init.d and /etc/init.d ->
rc.d/init.d before copying those.)

In fact, I think my recommendation would be to try the 2004 copy hack -
you would probably need to hookup bootchart to get tangible results +
comparisons. I found it to be very useful. 

It would be awesome if the presence of the bootchart RPM on the live cd
system automatically gave you two extra entries in syslinux ("profile
livecd" and "profile livecd [run from RAM]") along with all the
fancy-schmancy UI dialogs to get the bootchart + list of readahead
files.

> Another minor question-
> 
> "
> b. ext3 should support operations for moving blocks around; e.g.
>     optimize around the readahead fileset - when idle the system should
>     rearrange the files to facilitate faster booting
> "
> 
> Has anything happened with that in the last 3 years?  Sounds like a
> decent idea, that might be adaptable to the livecd boot optomization
> case.

Sadly, no, not that I know of. I know Jens Axboe did some interesting
work with fcache

 http://lkml.org/lkml/2006/5/15/46

but I don't think it applies to live cd as it duplicates a bunch of
stuff. I don't think there's any way to avoid an ext3 defrag API...

> Final question:  Have you had any other ideas on how livecd boot speed
> might be improved?

Yeah. I do. But you're not going to like it :-)

We simply need to do _less work_. We simply run way too many shell
scripts, programs and daemons at start up.

For example... there's absolutely no reason to start bluetooth daemons
if you don't have BT hardware (am working with the Bluez folks on
starting Bluez from HAL), there's no reason to start cups until you need
to print (printer discovery needs to be in the desktop session
_anyway_ / delayed until you actually need it), there's no reason to
have a HP printer / scanner daemon... there's no reason to start _four_
SELinux daemons.. there's no reason to write system daemons using
bloated frameworks like Python (am sure lots of people will flame me for
that)... and the list goes on.

Most, if not all, of this is fixable.. it's just a lot of hard work.

Hope this helps,
David