[Linux-cluster] GFS or Web Server Performance issues?

[isplist at logicore.net]

Here is ab sending a test to an LVS server in front of a 3 node web server. 
The average loads on each server was around 8.00 to 10.00. These aren't very 
good numbers and I'm wondering where to start looking.

These are pretty much default apache RPM versions out of the box.

# ab -kc 50 -t 30 http://192.168.1.150/
This is ApacheBench, Version 2.0.41-dev <$Revision: 1.141 $> apache-2.0
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.150 (be patient)
Finished 130 requests


Server Software:        Apache
Server Hostname:        192.168.1.150
Server Port:            80

Document Path:          /
Document Length:        8997 bytes

Concurrency Level:      50
Time taken for tests:   30.234185 seconds
Complete requests:      130
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      1371090 bytes
HTML transferred:       1299586 bytes
Requests per second:    4.30 [#/sec] (mean)
Time per request:       11628.532 [ms] (mean)
Time per request:       232.571 [ms] (mean, across all concurrent requests)
Transfer rate:          44.25 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   1.4      2       4
Processing:   409 6228 4357.5   5227   16317
Waiting:      369 6146 4370.8   5199   16261
Total:        409 6230 4358.1   5230   16321

Percentage of the requests served within a certain time (ms)
  50%   5230
  66%   8331
  75%   9310
  80%  10386
  90%  12820
  95%  14502
  98%  15399
  99%  15746
 100%  16321 (longest request)