[K12OSN] Server Tuning

[linuxk12 at mountainlake.k12.mn.us]

I have a server that is getting bogged down and am wondering how I can
tune it for better performance.

Initially when I noticed response times slowing, I used "top" to
see what was going on.  When load averages are over 3.0,
sometimes 5.0 and greater, response time is very poor.  At the
same time, top reported that the processor was 90% idle or better.

That led me wonder if there was an i/o bottleneck with the processor
idling while waiting for disk accesses to be completed. I don't know
how to check for that.

More recently I discovered "vmstat" and have been using that to monitor
what is going on.  Swapping doesn't appear to be a major problem ("si"
and "so" are usually at zero), so more memory wouldn't help me with
respect to swapping.  Would more memory help for some other reason?

What vmstat did show is that context switching is getting out of hand
during the times when the server appears overloaded.  Typically the
"cs" numbers are in the 300-500 range when things are humming along
nicely.  But something will push that number into the 1000's then the
10,000's and sometimes 100,000s and the box stalls out at that point,
waiting for the backlog to clear.  I typically use "vmstat 5" to
see five second data.

So, I'm wondering if there is something I can tune to fix this.  Are
there other monitoring tools I can use to show that my tuning is helping
things?  Failing a tuning solution, what do I need to do to ensure that
a new server has enough power to overcome this problem?

These are the specifics on the server.  This is a Dell server running
RedHat with kernel 2.4.18-3.  It is mostly a file server that supports NFS,
smb and netatalk (AppleTalk) for about 160 computers.  There are about
80 smb connections and 80 live NFS connections all day long of which
perhaps a third to half are actively transferring data at the same time.

The same server also acts as an ldap database (for the OSX clients) and
mail server (sendmail, spamassassin, imap).  Here is a count of some
of these processes that are all running at the same time as reported
by 'ps':

      5 identd
      5 spamd
      7 /var/openldap/libexec/slapd
      8 [kjournald]
      8 [nfsd]
     20 imapd    --- this looks unusual!
     22 /usr/sbin/afpd
     36 smbd

>From /proc/cpuinfo:

vendor_id	: GenuineIntel
cpu family	: 6
model		: 11
model name	: Intel(R) Pentium(R) III CPU family      1400MHz
stepping	: 1
cpu MHz		: 1396.496
cache size	: 512 KB
bogomips	: 2785.28


There are two SCSI drives formated as ext3 and the drive that sees
most of the io is mounted like this:

/dev/sdb1 /home ext3 rw,data=writeback,usrquota 0 0

So obviously I am using user quotas.  I don't remember what "writeback"
is doing for me.

Initially I was looking at performance issues to see if I should upgrade
to a gigabit ether connection, but with the current performance problems,
it doesn't appear to be able to fill even a 100Mb pipe.

Any suggestions are welcome.  If I need to invest some dollars to fix
this I will, but I want to spend them wisely.

--
Jon Harder
Technology Coordinator
Mountain Lake Public School
Mountain Lake, Minnesota