RH 2.4.18 vs RH 2.4.20 performance slowdown

[Dax Kelson]

We have a room full of *identical* boxes that we have run Red Hat Linux
classes (6.x, 7.x, 8.0 and  9) on over the past 4 years. These are
500Mhz Intel 440BX motherboard boxes.

No problems until RHL9 came out. On about 50% of the machines (identical
hardware remember, including BIOS settings) kernel system calls on RH
2.4.20 kernels run about 4x - 10x slower.

Of course with this problem the whole system runs dog slow and is
painful to use. 

The vanilla kernel.org kernels and the RH 2.4.18 kernels (from RHL8.0)
do NOT exhibit the slowdown.

The problem can be easily quantified using strace. Take a look at the
following (especially the third column):

First with vanilla ftp.kernel.org 2.4.20 compiled using
kernel-2.4.20-i686.config from RH.

[root at station9 root]# uname -r
2.4.20
[root at station9 root]# strace -c ls -al /etc > /dev/null
execve("/bin/ls", ["ls", "-al", "/etc"], [/* 30 vars */]) = 0
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 41.25    0.002895          10       289           lstat64
 12.92    0.000907          36        25           read
 12.61    0.000885          22        41        14 open
  5.34    0.000375          21        18           readlink
  5.27    0.000370          12        31           old_mmap
  5.14    0.000361          52         7           getdents64
  5.06    0.000355          15        23           munmap
  2.82    0.000198           7        30           close
  2.11    0.000148           5        28           fstat64
  1.44    0.000101           3        31           fcntl64
  1.44    0.000101          51         2           socket
  1.42    0.000100          50         2         2 connect
  1.27    0.000089           5        17           brk
  0.80    0.000056          56         1           mmap2
  0.57    0.000040           8         5           write
  0.19    0.000013           4         3         2 rt_sigaction
  0.17    0.000012           4         3         3 ioctl
  0.13    0.000009           9         1           uname
  0.06    0.000004           4         1           gettimeofday
------ ----------- ----------- --------- --------- ----------------
100.00    0.007019                   558        21 total

Now the latest RHL9 errata kernel. All RHL9 kernels and RHL8.0 kernels
>= 2.4.20 perform the same:

[root at station9 root]# uname -r; strace -c ls -al /etc > /dev/null
2.4.20-20.9
execve("/bin/ls", ["ls", "-al", "/etc"], [/* 30 vars */]) = 0
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 43.57    0.019390          67       289           lstat64
 10.88    0.004841         194        25           read
  9.82    0.004369         109        40        13 open
  5.84    0.002601         145        18           readlink
  5.78    0.002574         112        23           munmap
  4.51    0.002007          72        28           fstat64
  4.17    0.001857         265         7           getdents64
  3.12    0.001387          45        31           fcntl64
  2.53    0.001124          37        30           close
  2.42    0.001078          98        11           old_mmap
  1.87    0.000834          49        17           brk
  1.07    0.000475         238         2         2 connect
  1.06    0.000473         237         2           socket
  1.00    0.000446          20        22           mmap2
  0.91    0.000405          81         5           write
  0.52    0.000233          78         3         2 rt_sigaction
  0.44    0.000197          66         3         3 ioctl
  0.44    0.000197         197         1           uname
  0.02    0.000011          11         1           set_thread_area
  0.01    0.000003           3         1           gettimeofday
------ ----------- ----------- --------- --------- ----------------
100.00    0.044502                   559        20 total

I'm posting this message to see if anyone else has seen anything similar
or has any ideas. This same problem is 100% reproducible on multiple
machines in the classroom.

You may want to add comments or add your self to the CC list here:

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=90116

The machines in this classroom are being replaced in two weeks with P4
2.8Ghz HyperThreaded boxes, so ideally we can get this problem nailed
down soon.

Dax Kelson