[Linux-cluster] Storage Cluster Newbie Questions - any help with answers greatly appreciated!

[Michael @ Professional Edge LLC]

Hail Linux Cluster gurus,

I have researched myself into a corner and am looking for advice.  I've 
never been a "clustered storage guy", so I apologize for the potentially 
naive set of questions.  ( I am savvy on most other aspects of networks, 
hardware, OS's etc... but not storage systems).

I've been handed ( 2 ) x86-64 boxes w/2 local disks each; and ( 2 ) 
FC-AL disk shelves w/14 disks each; and told to make a mini NAS/SAN (NFS 
required, GFS optional).  If I can get this working reliably then there 
appear to be about another ( 10 ) FC-AL shelves and a couple of Fiber 
Switches laying around that will be handed to me.

NFS filesystems will be mounted by several (less than 6) linux machines, 
and a few (less than 4) windows machines [[ microsoft nfs client ]] - 
all more or less doing web server type activities (so lots of reads from 
a shared filesystem - log files not on NFS so no issue with high IO 
writes).  I'm locked into NFS v3 for various reasons.  Optionally the 
linux machines can be clustered and GFS'd instead - but I would still 
need to come up with a solution for the windows machines - so a NAS 
solution is still required even if I do GFS to the linux boxes.

Active / Passive on the NFS is fine.

* Each of the ( 2 ) x86-64 machines have a Qlogic dual HBA 1 fiber 
direct connected to each shelf  (no fiber switches yet - but will have 
them later if I can make this all work); I've loaded RHEL 5.4 x86-64.

* Each of the ( 2 ) RHEL 5.4 boxes - used the 2 local disks w/onboard 
fake raid1 = /dev/sda - basic install so /boot and LVM for the rest - 
nothing special here (didn't do mdadm basically for simplicity of /dev/sda)

* Each of the ( 2 ) RHEL 5.4 boxes can see all the disks on both shelves 
- and since I don't have Fiber Switches yet - at the moment there is 
only 1 path to each disk; however as I assume I will figure out a method 
to make this work - I have enabled multipath - and therefore I have 
consistent names to 28 disks.

Here's my dilemma.  How do I best add Redundancy to the Disks, removing 
as many single points of failure, and preserving as much diskspace as 
possible?

My initial thought was - to take "shelf1:disk1 and shelf2:disk1" and put 
them into a software raid1 - mdadm; then put the resulting /dev/md0 into 
a LVM.  When I need more diskspace, I just then create "shelf1:disk2 and 
shelf2:disk2" as another software raid1 then just add the new "/dev/md1" 
into the LVM and expand the FS. This handles a couple things in my mind:

1. Each shelf is really a FC-AL so it's possible that a single disk 
going nuts could flood the FC-AL and all the disks in that shelf go poof 
until the controller can figure itself out and/or the bad disk is removed.

2. Efficient I am retaining 50% storage capacity after redundancy - if I 
can do the "shelf1:disk1 + shelf2:disk2" mirrors; plus all bandwidth 
used is spread across the 2 HBA fibers and nothing goes over the TCP 
network.  Conversely DRBD doesn't excite me much - as I then have to do 
both raid in the shelf (probably still with MDADM) and then I add TCP 
(ethernet) based RAID1 between the nodes - and when all is said and done 
- I only the have 25% of storage capacity still available after redundancy.

3. I easy to add more diskspace - as each new mirror (software raid1) 
can just be added to an existing LVM.

 From what I can find messing with Luci (Conga) though... is - I don't 
see any resource scripts listed for - "mdadm" (on RHEL 5.4) - so would 
my idea even work  (I have found some posts asking for a mdadm resource 
script but I've seen no response)?  I also see with RHEL 5.3 LVM has 
mirrors that can be clustered now - is this the right answer?  I've done 
a ton of reading but everything I've dug up so far; assumes that the 
fiber devices are being presented by a SAN that is doing the redundancy 
before the RHEL box sees the disk... or... there are a ton of examples 
of where fiber is not in the picture and there are a bunch of locally 
attached hosts presenting storage onto the TCP (ethernet) - but I've not 
found nearly anything on my situation...

So... here I am... :-)  I really just have 2 nodes - who can both see - 
a bunch of disks (JBOD) and I want to present them to multiple hosts via 
NFS (required) or GFS (to linux boxes only).

All ideas - are greatly appreciated!

-Michael