Instrumenting kernel for boot time profiling

[Ziga Mahkovec]

On Mon, 2005-04-18 at 10:33 -0400, Tom Heydt-Benjamin wrote:
> 	I followed with interest the "Boot Poster Challenge" thread from
> November.  If I am correct, all of the profiling results that were
> demonstrated were done in userland by instrumenting the rc files and such.

Yes, the early versions used to patch the sysinit script and use top and
iostat for collecting data.  This was later changed by replacing init(1)
and collecting all data from /proc directly.  See [1] for details.

> 	Has anyone experimented with instrumenting the kernel to try and get
> fine grained profiling data from the earliest parts of the boot
> process?  I need to do this for some experiments that we are
> performing with power management of small mobile Linux devices, and
> before I re-invent the wheel, I want to know what wisdom is available
> on collecting timing data from the kernel starting early in the boot
> process.

Depending on what kind of data you're interested in (kernel boot or
user-space boot?), there are already a number of useful interfaces in
the kernel:

1. Profiling

   Using readprofile or oprofile you can get some basic profiling 
   information. See linux/Documentation/basic_profiling.txt.

2. BSD process accounting

   Process accounting will allow you to get a detailed picture of
   the time and resources used by processes.  By enabling 
   CONFIG_BSD_PROCESS_ACCT_V3 (which also requires an updated
   user-space tool) you'll even get PID/PPID information, which is
   great for creating the process dependency graph.

3. Audit

   The audit subsystem in the 2.6 kernels provides system call
   auditing.  Since it also intercepts calls to path_lookup(), you
   can use it to build a list of all opened files for example. Take
   a look at the audit package in rawhide.

   LTT[2] could also be used for tracing system calls.  It used to
   live in the -mm patchset, but I think it's currently being
   reworked.


[1] http://www.bootchart.org/docs.html
[2] http://www.opersys.com/LTT/

-- 
Ziga