Determining what is causing load when server is idle.

[Ray Van Dolson]

I have an HP DL140G2.  Two 80GB SATA drives configured as a RAID1 software
array.  Fresh install of RHEL4.  Single processor -- hyperthreading on and
using the SMP kernel.

I have to boot up with ide0=noprobe and ide1=noprobe in order to ensure that
ata_piix is used instead of the native IDE driver to access my hard drive.

Everything runs fine, but I am bothered by the fact that although *nothing*
is running on this system, the load on the box is always around 0.10 to
0.35.

Now before you go and accuse me of overlooking some running process, let me
assure you I have killed everything that does not need to be running.  No
sendmail, no cron, etc...

I have plenty of other servers installed (not DL140 G2's though) running
software raid (on IDE drives, not SATA) where the system load when the box
is ide is 0.00 as I'd expect.

Another symptom that comes along with this weird non-0.00 load issue is that
user I/O seems to "glitch" every now and then.  Almost like the hard drives
are spinning up after being put to sleep... however, APM is disabled in my
kernel since I am running in SMP mode.

Running 'top' and sorting by total CPU usage time, the only process I really
see consuming any time is 'init' which doesn't seem all that usual.

My thought is that it has something to do with either the ata_piix driver
and/or the combination of software raid.

What tools would you guys use to track this down?  Can I see what drivers in
the kernel are consuming the most CPU time?

I have tried:
  - HyperThreading off
  - Non-SMP mode
  - Custom non-RH kernel (2.6.16).  This actually yielded the best results.
    System load was much lower when idle using the 2.6.16 kernel, but still
    not quite what I expected.
  - Different servers (identical HW config however).

I probably need to try disabling software RAID next.

Happy to provide any info necessary (/proc/interrrupts, etc).

Thanks,
Ray