Looking for System performance tools

[Ben Russo]

Padiyath Sreekumaran wrote:
>   Hi,
>   I have installed RedHat OS version 3 on a Dell 2500 machine.
>   The system has 2 CPU's and 2GB memory. This machine is used only as
>   Archiving (to store information transferred from users via ftp and
>   nfs - no interactive login). It is connected with san to 2 disk arrays.
>   One disk array is 1KM away from the machine for which 1Gb connection
>   used. The other disk array is in the same wrack as the computer.The users
> first 
>   transfer data to the disk (via ftp,nfs) and then
>   after a predefined time interval the files are transferred to tape(LTO).
>   I see often the load(with top) of the machine very high. Upto 14 to 20.
>   Is there any tool with which I can findout the bottleneck of the problem:
>   traffic between disk and tape, memory and disk etc. Is it possible to 
>   measure the network performance of the disk array 1Km away from the
> machine?
>   When I look the top out, I see that the memory is not much used. Both the
> CPU
>   are 60-70% used. What commands can I use to find out the bottleneck?

Well if you see lot's of idle time on the CPU's in top then CPU 
speed/capacity is not a bottleneck.

Use "vmstat 5 20" to see 5 second averages of some system stats.  Always 
ignore the first line of numbers (this is a calculation based on the 
counters in the kernel since the machine has been up). For example:

# vmstat 2 5
procs                      memory      swap          io     system 
    cpu
  r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us 
sy id wa
  2  0 353728  16632 820408 128100    1    1     0     0    0     0  0 
0  0  0
  1  0 353728  16628 820408 128100    0    0     0     0  104    97 27 
23 50  0
  1  0 353728  16600 820428 128100    0    0     0    30  111   105 24 
26 49  1
  1  0 353728  16612 820428 128100    0    0     0     0  105   100 30 
20 50  0
  1  0 353728  16608 820428 128100    0    0     0     0  102    81 20 
30 50  0

This provides five 2 second averages.

The numbers you may want to look at here are in the columns under "swap"
for "si" and "so" (swap in and swap out).  If you see that those are all 
zeroes, (or at least pretty low).  And you see that you have relatively 
high numbers under the 3 "memory" category columns "free" "buffer" and 
"cache" then you know it isn't a Memory capacity bottleneck.

Next, check your Disk performance with "iostat 5 5".
For example (again, ignore the first stanza of output, I will omit it here)

# iostat 5 2
Linux 2.4.9-e.38smp (bonzo.reston.tnsi.com)     07/01/2005

...(((( IGNORE FIRST STANZA )))) ....

avg-cpu:  %user   %nice    %sys %iowait   %idle
           24.75    0.00    0.15    0.00   75.10

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               6.60       307.20       456.00       1536       2280
sda1              0.00         0.00         0.00          0          0
sda2              0.40         0.00        19.20          0         96
sda3              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda5              6.20       307.20       436.80       1536       2184
sda6              0.00         0.00         0.00          0          0
sda7              0.00         0.00         0.00          0          0
sda8              0.00         0.00         0.00          0          0
sda9              0.00         0.00         0.00          0          0
sda10             0.00         0.00         0.00          0          0
sdb               0.00         0.00         0.00          0          0
sdb1              0.00         0.00         0.00          0          0
sdc               0.00         0.00         0.00          0          0
sdc1              0.00         0.00         0.00          0          0


You say "SAN" what medium are you using to access this disk that is 1km 
away?  iSCSI?  FC?
Carefully think about the implications of doing these things, but for a 
archive server this shouldn't be too bad,  you might want to mount 
noatime on the remote storage.

Read up about bdflush settings in sysctl.conf
I have seen some people suggesting this would make a big difference
Try
vm.bdflush=100 1200 128 512 15 5000 100 0 0
in /etc/sysctl.conf
and then do a sysctl -p

The other thing to remember is that maybe a high load on this system is 
normal.