[Linux-cluster] Homebrew NAS Cluster

[Andrew A. Neuschwander]

My setup sounds similar to yours but with a SAN for all the underlying 
storage.

I have a large FC SAN (might be cost prohibitive for you), and three 
physical (Dell PE1500s) servers. Two of them are running ESX 3.5 and one 
is running CentOS. The ESX Servers share a chunk of SAN using VMFS3. The 
rest of the san is shared by all three physical servers. I have a 
handful of virtual CentOS servers to which I've installed the shared SAN 
luns via raw device mapping (with the scsi bus' set in physical sharing 
mode).

I then put the physical and virtual CentOS machines in one GFS cluster 
to share the san (using a custom fence script). While this all works and 
is in production, the performance isn't what I'd like. Locking calls by 
the virtual centos machines really slow things down, especially when 
running samba on a vm. I think it's the nature of GFS being exacerbated 
by all the abstraction of ESX. It takes quite a bit of tuning.

The biggest caveat for ESX users is that putting a virtual machine's 
scsi bus in physical shared-bus mode, disables DRS and VMotion. You 
can't live migrate these machines. The HA feature still works well though.

-A
--
Andrew A. Neuschwander, RHCE
Linux Systems/Software Engineer
College of Forestry and Conservation
The University of Montana
http://www.ntsg.umt.edu
andrew at ntsg.umt.edu - 406.243.6310


Stephen Nelson-Smith wrote:
> Hi all,
> 
> I'm in the process of setting up a virtualisation farm which will have
> 50-60 virtual machines, running a wide range of web, application and
> database applications, all on top of vmware vi3.
> 
> My budget won't stretch to a commercial NAS solution, so it's either a
> SAN, which could get complicated and hard to manage with so many
> nodes, or a home-brew NAS solution.
> 
> Has anyone done this, on the list?  I'm wondering what the catch is?
> I'm thinking all I need to do is run NFS on top of a clustered
> filesystem, and export to ESX.
> 
> I could use some pointers, gotchas, ideas and experiences.
> 
> Thanks!
> 
> S.
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>