head node has an extremely high load average.

[Miner, Jonathan W (US SSA)]

> I installed the iozone program and ran ./iozone -a.

iozone allows you to benchmark disk performance and gives you objective measurements.

> How does this information help me find the offending program?

Not sure you're looking for a "program"... I think you know what program is doing the IO on your client machines, and we know that "nfsd" is doing the IO on the server, and we know from your previous output that you have high IO wait times.  So... you should be looking at which disks are involved, and why the wait times are so high..

Are you using single drives, software raid, hardware raid? What type of bus?  


A lot goes into performance monitoring:

http://www.thegeekstuff.com/2011/03/linux-performance-monitoring-intro