poor read performance

[Glen Dosey]

I am seeing what seems to be a notable limit on read performance of an
ext3 filesystem. If anyone could offer some insight it would be helpful.

Background:
12 x 500G SATA disks in a Hardware RAID enclosure connected via 2Gb/s FC
to a 4 x 2.6 Ghz system with 4GB ram running RHEL4.5. Initially the
enclosure was configured RAID5 10+1 parity, although I've also tried
RAID 50 and currently RAID 0. I've varied chunk sizes from 64-256K. 

Problem:
No matter what I do I cannot get the ext3 read performance above
~90MB/s. Under virtually every configuration listed above the write
performance is greater than the read performance. I've run a large
number of Bonnie++ and IOzone tests, but for the sake of simplicity in
this email I'll just refer to simple dd's with /dev/zero.

Details:
Under the current RAID0 setup I see the following when dd'ing.

DD 4G from /dev/zero to /dev/sdd disk (no filesystem) & sync
28 seconds
DD 4G from /dev/sdd to /dev/null 32 seconds
DD 4G to ext3 on /dev/sdd & sync 32 seconds
DD 4G from ext3 file to /dev/null 48 seconds.

I've been watching the port usage on the FC switch and it verifies what
I am seeing, Writes max out near 2Gb/s but reads hit some artificial
limit around 90 MB/s and never ever exceed it with the filesystem,
regardless of they underlying RAID configuration. Without a filesystem
the reads are atleast 50% faster, and it can be seen on the FC switch
graphs as well.

Any help or thoughts would be appreciated.

Thanks,
~Glen