adventures in booting

Tue Nov 30 20:20:46 UTC 2004

On Sun, 2004-11-28 at 20:49 -0500, David Zeuthen wrote:

Awesome!  So... is no-one else going to reply to this email?  

Various comments inline below...

> Hey,
> 
> So, I've looked a bit more into the booting process and how to optimize
> it. Mostly based on the discussion triggered by Owen's boot poster
> challenge, here
> 
>  http://www.redhat.com/archives/fedora-devel-list/2004-November/msg00447.html
> 
> and also some experiments that I did - basically replacing rhgb with gdm
> as described here 
> 
>  http://www.redhat.com/archives/fedora-desktop-list/2004-November/msg00066.html
> 
> What I've done is a bit crude - I've replaced init(1) with a shell
> script based on /etc/rc.d/rc.sysinit and tried to optimize specifically
> for my system: IBM Thinkpad T41 laptop with a Pentium M processor at
> 1600MHz along with 512MB of RAM.

If I understand, you're also optimising for a specific user of that
system.  Is it worth splitting the readahead into a system-wide list of
files (enough to get to the login screen), followed by a per-user list
for logging in as a user?  To what extent will the files needed to get
to a usable desktop vary between Alice and Bob?

> 
> The results are pretty good I think, here is the general time line made
> with a wallclock
> 
> 00: exit grub; start booting the kernel
> 04: kernel prints audit()
> 11: initrd is mounted; Red Hat nash visible
>      mount / ro (normal initrd procedure)
> 13: start bootchart logging; start readahead of approx 193MB files
>      sleep until readahead is complete
> 24: readahead done; now
>      create /dev and modprobe (in background)
>      mount / rw, enable swap
>      start xfs
>      startx as user davidz in background
>      start messagebus
>      start hald
>      start acpid
>      start NetworkManager
> 32: X claims the display
> 34: GNOME desktop banner
> 40: GNOME desktop is usable (Nautilus desktop, panel fully populated)
> 
> Here is a bootchart made with the bootchart software from Ziga Mahkovec:
> 
>  http://people.redhat.com/davidz/bootchart.png
> You may notice that I also start firefox after login and it starts very
> very fast - that's because readahead loads all files used by Firefox -
> in earlier experiments I've also added files from OpenOffice.org to
> readahead and that meant I could start up OpenOffice.org Writer in about
> three seconds. More below.
> 
> I've made the following observations
> 
> 1. The kernel patch, linux-2.6.3-printopen.patch, wasn't really working
>    well for me - it reported far to few files - instead I added a
>    printk() to fs/namei.c:link_path_walk() 
>    (disclaimer: I don't know much about the kernel so there may be a
>    better solution than this). 
> 
> 2. The data captured from link_path_walk() was massaged into a list
>    of unique files to preload and sorted on sectors.
> 
> 3. While capturing the data link_path_walk() and before processing
>    I went through all the menus in the GNOME desktop (to make sure
>    their icon and desktop files would be added to the list) as well as
>    loading Firefox. The list contains 5189 unique files - 231 of these
>    from my home directory - 103 of these from gconf in my home
>    directory and 302 from gconf in /etc. 2267 were .png files and
>    814 of them were .desktop files. 488 files had ".so" in their name.
>    There was a total of 193MB of files (which says something about
>    the footprint of the GNOME desktop on Fedora :-/)
> 
> 4. Doing the readahead really helped the time from startx till a 
>    usable desktop - less than 10 seconds!
> 
> 5. Doing readahead on the 5189 files took about 45 seconds on my
>    system, mostly because the files were scattered around the disk.
>    Since I had a spare partition 17GB partition, I did this:
>     a. format spare partition as ext3
>     b. copy all readahead files to spare partition (193MB)
>     c. copy rest of files from main partition to spare partition
>        (about 9GB)
>    Now the readahead is down to 11 seconds which averages out to
>    be 18MB/s. On the other hand, I can still see (using fileblock)
>    that the files in the readahead is still scattered out and hdparm
>    says I should be able to get 33.87 MB/sec with no seeks.
> 
> 6. I made a hack to cache /dev (a dev.tar file) and the list of modules
>    to load. This could be used in production if the kernel could give
>    us basically a hash value for the kobject hierarchy representing
>    the hardware (perhaps even a 'tree /sys |md5sum' would suffice).
>    This shaved some seconds of as well.
> 
> 7. A number of things was started in parallel - I found that doing
>    readahead while running modprobe wasn't helping anything; in fact
>    it contributed negatively to performance (a bit to my surprise, I
>    guess because the kernel was busy).
> 
> 8. readahead on the right files is A Good Thing(tm). Booting my system
>    without readahead on the partition with the readahead files scattered
>    took 58 seconds (compared to 39 with readahead on the optimized
>    partition)
> 
>     http://people.redhat.com/davidz/bootchart-without-readahead-scattered.png
> 
>    and without readahead on on the optimized partition it took 43
>    seconds
> 
>     http://people.redhat.com/davidz/bootchart-without-readahead-nonscattered.png
> 
>    again compared to 39 seconds. As an added bonus, the readahead
>    makes sure that e.g Firefox loads fast; all .png and .desktop files
>    are in place for when using the menus. As mentioned, one could put
>    very big apps like e.g. OO.o in the readahead set.
> 
> So, I think these numbers are good and there's still some room for
> improvement; e.g. it takes ten seconds from grub to when the initrd is
> mounted - surely the kernel can boot faster? It's after all 25% of the
> time spent from grub until I have usable desktop.
> 
> The bad thing is that this approach is highly specific to my system (and
> thus why I'm not posting an RPM with it :-), however I think it clearly
> shows where improvements should be made; here are some random thoughts
> 
>  a. We should keep track of files being loaded and maintain the
>     readahead fileset as appropriate. printk() doesn't seem like the
>     right solution; perhaps a system daemon using inotify or the
>     kernel events layer is the road ahead? This would enable us to
>     readahead the KDE stuff if the user is e.g. using KDE a lot.
> 
>  b. ext3 should support operations for moving blocks around; e.g.
>     optimize around the readahead fileset - when idle the system should
>     rearrange the files to facilitate faster booting
> 
>  c. the start_udev and kmodule process could be cached as I did above
> 
>  d. The whole init(1) procedure seems dated; perhaps something more
>     modern built on top of D-BUS is the right choice - SystemServices
>     by Seth Nickell comes to mind [1]. Ideally services to be started
>     would have dependencies such as 1) don't start the gdm service
>     before /usr/bin/gdm is available; 2) the SSH service would only
>     be active when NetworkManager says there is a network connection;
>     /usr from LABEL=/usr would only be mounted when there is a volume
>     with that label and so forth. Also, such a system would of course
>     have support for LSB init scripts.
>     (This is probably a whole project on it's own so I'm omitting
>     detailed thinking on it for now)

Hopefully this could also allow us to make system-config-services look a
lot slicker.  I've never liked the way it has random text spewage for
each service's status - some kind of widgetry would satisfy my eye-candy
cravings.

> 
> Thanks a lot to Ziga Mahkovec for the bootchart software - it's been
> very useful.
> 
> Have fun,
> David
> 
> [1] : http://www.osnews.com/story.php?news_id=4711
>       http://www.gnome.org/~seth/blog/2003/Sep/27
> 