computation-friendly kernels

[Bevan C. Bennett]

I suspect a full answer to this may be too involved for this list, but 
maybe someone a little more into kernel hacking than I've been lately 
could point me in the right direction.

I have two identical servers, one running FC1 and one running severn 
(RHEL3beta). The installed kernels are:

kernel-smp-2.4.22-1.2138.nptl (FC1)
kernel-smp-2.4.21-1.1931.2.399.ent (severn)

I have a very highly multithreaded application that runs generally 4x 
slower on FC1, but occassionally takes 20x as long. Interestingly, the 
"user" time in all cases is very consistant, and on the non-FC1 system 
real time is only fractionally higher than user time.  So... I suspect 
that there's some kernel parameter, memory management scheme or 
scheduling configuration difference causing the performance disparity.

My question is whether there's any hope of discovering which difference 
is involved and 'fixing' the FC1 system (or even of easily enumerating 
what the differences actually are)?

I know that FC is aimed more at a desktop/general user release (and is 
the best Linux I've used so far for that purpose), and that ideally I'd 
help support RedHat by purchasing 100 copies of RHEL for my heavy 
computing, but there are unfortunate budgetary forces involved that 
prohibit that at the moment. If the company situation improves, that's 
the way I'd like to go eventually.

I'm willing to compile custom kernels for a 4x speedup and better 
consistancy, but I'd like to keep the number of different distributions 
in active use to a minimum (every different distribution adds more work 
in creating and maintaining compatible configuration info).

Any informed suggestions from the community?