[Linux-cluster] Cluster with shared storage on low budget

[Digimer]

On 02/14/2011 06:39 PM, Nikola Savic wrote:
> 
>   Hello,
> 
>   I need to setup cluster of 3 servers without separate storage device
> (SAN). Servers should join their local hard drives to create shared
> storage space. Every server in cluster has public (100Mbps) and private
> (1Gbps) NIC. Private 1Gbit network will be used for exchange of data
> (files) on shared storage. Additional request is that data on shared
> storage is highly redundant (complete mirroring is required).
> 
>   I was wondering if following setup is possible and if anyone has any
> experience or comments on it:
> - Servers will export local disks of same size as iSCSI targets
> - Each server will access other's two servers disk over iSCSI initiator
> - CLVM will be used to set Volume Group on all 3 disks. In theory this
> VG will work on all servers, because they'll have access to all disks
> (either directly or over iSCSI).
> - CLVM will be used to create Logical Volume with mirroring option set
> to 3 (-m 3). Since there are 3 disks (physical devices) forming VG, each
> server will have redundant copy of same data.
> - Created Logical Volume will have GFS2 on it, so that it can be shared
> by cluster.
> - Web server will store web application files (scripts and photos) on
> created GFS.
> 
>   If it works, this setup should provide shared storage for cluster,
> built from already available local hard drives in servers forming
> cluster. By using LVM mirroring, each server will have the same copy of
> data, which should make cluster resistant to failure of any server.
> 
>   I was wondering, is LVM smart enough to optimize reading and use local
> drive for read operations?
> 
>   Looking forward to your comments.
> 
>   Best Regards,
>   Nikola

I'd recommend looking at created a three-way DRBD resource. Use this
resource as your cLVM PV/VG/LVs. On these LVs you can use GFS2 for the
actual shared file system where your data can reside.

A few notes:

- You *MUST* have fencing setup. This is not an option, and manual
fencing will not suffice. If your server have IPMI (or vendor
equivalents like DRAC, iLO, etc) then you are fine. If not, you will
need an external device like an addressable PDU (see APC or Tripplite).

- LVM optimization is not something I can comment on.

- We've not discussed complexity. Clustering is not inherently hard, but
there is a lot to know and a lot can go wrong. Give yourself ample time
to work through problems and test failure scenarios. Do not expect to
launch in a month. It will likely takes a few months at minimum to be
ready for production.

- Join the #linux-cluster IRC channel on freenode.net. There are good
people there who can help you out as you learn.

- Be patient and have fun. :)

-- 
Digimer
E-Mail: digimer at alteeve.com
AN!Whitepapers: http://alteeve.com
Node Assassin:  http://nodeassassin.org