NFS performance issue

[ASB]

Hello,

I have a NFS server set-up on a RHEL AS 3.0 machine with 4 Pentium IV 
Xeon's and 4 Gig of RAM that is just acting as a NFS server to around 2 
TB of disk space (2 LUNS). The NFS server has connects to a Gigabit 
switch with 4 Aggregated Gigabit NIC's. And the storage is connectd 
using Fiber connects. This NFS share is mounted by 48 node cluster 
(Rocks 3.2.0 / based on RHEL 3.0 base srpms). On the NFS server there 
are 8 instances of nfsd serving the requests and i dont see the network 
contention (no even near contention) on the NFS server and/or Disk's. 
The requests that NFS server is serving are 1000's of small 
read/write/stat calls.

What i need to know is how can i improve the NFS server performance what 
should be the options on the NFS export and NFS clients that mount the 
NFS share (i.e. should i use udp or tcp for NFS, hard vs soft mounts 
etc.. my applications should not get a file not found or error because 
they are too touchy to errors)

1. Is the number of nfsd's on a 4 processor machine enough, should i 
enable HT or not. During loads when i do a cat /proc/net/rpc/nfsd this 
is the output i get but am unable to determine that do i need more 
instances of nfsd's.

# cat /proc/net/rpc/nfsd
rc 119962 25377040 42222820
fh 3690 66565276 0 140 3637
io 3570911731 3462845160
th 8 30591462 10278.250 4629.100 3809.780 0.000 3457.170 2708.270 
2035.320 1717.180 0.000 19783.720
ra 16 3645786 226692 72466 132437 59243 60455 41709 14720 25792 13253 
1901791
net 67719822 67702906 16916 33
rpc 67719822 0 0 0 0
proc2 18 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
proc3 22 2 9122660 314574 9685297 13420650 0 6195787 19583200 2475564 
902 0 0 1772088 717 1349957 0 82830 102511 103 294 0 3312072

after 1 day of operation

rc 432346 54116787 118493699
fh 5102 167914277 0 724 48367
io 1430949976 1209324547
th 8 110245501 17130.230 6509.020 5264.770 0.000 4811.310 4098.190 
3469.520 3636.230 0.000 83702.300
ra 16 7044467 559769 182947 290285 115112 142084 99659 35098 61013 28165 
5258146
net 173042832 173025916 16916 33
rpc 173042832 0 0 0 0
proc2 18 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
proc3 22 2 25577479 3740196 30203894 38264086 0 13818455 37670132 
5648930 1252 0 0 4041598 957 3446068 0 721617 495716 276 294 0 8962512


2. Should i change the mount option in my fstab the current settings are

nfsserver:/export/lun1 /global/data/lun1 nfs hard,udp,defaults 0 0
nfsserver:/export/lun2 /global/data/lun2 nfs hard,udp,defaults 0 0

What should i concentrate on in order to improve / optimize for 
performance and reliability.

Thanks

Anand