RHEL AS 4 U2 Slow

[Rick Stevens]

On Tue, 2006-02-07 at 17:29 -0500, Brenda Radford wrote:
> Rick Stevens wrote:

I'm cleaning up the message a bit...there's a lot of cruft we don't
need to deal with anymore.

> >There's nothing obvious there.  What you really need to do is run
> >"top" and look at the top few processes listed there (you can usually
> >ignore the "init", "top" and "X" processes) and see what's sucking up
> >the CPU time.  Watch the "%CPU" and %MEM" columns and find the process
> >that's got the highest "%CPU" bit.  That's the one we need to look at.
> >
> >Also pay attention to the bit that looks like this:
> >
> >Cpu(s):  4.6% us,  0.0% sy,  0.0% ni, 94.4% id,  0.0% wa,  1.0% hi,
> >0.0% si
> >
> >as it shows a summary of where the CPU is spending its time:
> >
> >"us" = user state
> >"sy" = system state
> >"ni" = non-interruptible sleep
> >"id" = idle
> >"wa" = I/O wait state
> >"hi" = hardware interrupts
> >"si" = software interrupts
> >
> >Even if you don't see a process sucking up a lot of CPU, but you see
> >the CPU spending a lot of time in the "wa" state, then you have a disk
> >problem.  Look in the process list for processes in the "D" state.
> >
> >
> >  
> >
> Rick,
> 
>  From top, on two different days:
> 
> [brenda at localhost ~]$ top
> 
> top - 20:40:49 up 11 min,  2 users,  load average: 0.03, 0.05, 0.06
> Tasks:  83 total,   1 running,  82 sleeping,   0 stopped,   0 zombie
> Cpu(s):  2.3% us,  0.0% sy,  0.0% ni, 97.7% id,  0.0% wa,  0.0% hi,  0.0% si
> Mem:    905760k total,   306536k used,   599224k free,    18996k buffers
> Swap:  1799232k total,        0k used,  1799232k free,   180724k cached
> 
> [brenda at localhost ~]$ top
> 
> top - 15:31:16 up 35 min,  4 users,  load average: 0.24, 0.07, 0.02
> Tasks:  89 total,   1 running,  87 sleeping,   0 stopped,   1 zombie
> Cpu(s):  1.3% us,  0.0% sy,  0.0% ni, 98.7% id,  0.0% wa,  0.0% hi,  0.0% si
> Mem:    905760k total,   396168k used,   509592k free,    23224k buffers
> Swap:  1799232k total,        0k used,  1799232k free,   244368k cached
> 
> The only time the CPU showed any activity in the I/O wait state was when 
> top was first started,
> at the 1-4% level, and only for an instant.  It immediately went back to 
> 0.0%.  The only other
> processes that showed up at the top of the list (besides those you 
> mentioned) were gnome-terminal,
> hald, and rhn-applet-gui, but they only used tiny amounts of CPU and 
> MEM, even with 4 or 5
> terminal windows open (hence the 4 users). 
> 
> The more times I ran top, the more Memory it reported used (in the top 
> header).  It went up to more than
> 4XX,XXX used before I was finished, after running  top and ps ax 
> numerous times.

That's normal and nothing to worry about.  Start worrying if you see
the "Swap: 0k used," thing start to go non-zero.  That means you're out
of memory and the system has to swap things from memory to disk and back
to run them.  That slows the machine down a LOT.

> I didn't have any processes in the "D" state.

Ok, so we don't seem to have an I/O wait state issue.  I did notice a
zombie process there in the second top report.  I'm curious as to which
process that is and what its parent is.  Try doing a "ps ax" and find
the process that's in a "Z" state.

> Any ideas?

Well, it doesn't seem to be process or memory related.  It could be
context switching issues.  Try doing a "vmstat 5" for, say a minute,
then CTRL-C to get out of it.  Look at the "cs" column towards the
right.  If that gets to 5 digits, we have something that's causing
context switch problems and there's a bit more investigation we need
to do.

Also look at the output of "dmesg" for clues.

----------------------------------------------------------------------
- Rick Stevens, Senior Systems Engineer     rstevens at vitalstream.com -
- VitalStream, Inc.                       http://www.vitalstream.com -
-                                                                    -
-      We are born naked, wet and hungry. Then things get worse.     -
----------------------------------------------------------------------