[Linux-cluster] Is this even possible with GFS, replacing NFS

[Robert Peterson]

Brandon Lamb wrote:
> I am on about day 4 or so of researching filesystems and cluster
> software, and ready to pull my hair out.
>
> First let me describe my situation.
>
> I have a 20 disk scsi raid-10. The server that it is connected to has
> 4 gigs of ram, and exports a single directory via NFS. We only have a
> 150gig partition, using about 100 of it. We use this for our mail data
> store, 2.8 million files using maildir format.
>
> We have 5 linux boxes connected via NFS over gigabit (NOT using jumbo
> frames) and they all write to the users maildirs via this nfs mount.
>
> Now, this IS working and has been for over 1.5-2 years, however it is
> slow as hell.
>
> Here is where my question comes in. We are planning on splitting the
> current raid into two 10 disk raid10s so that we can replicate and
> have a live spare. That part I already am 90% sure of how I will
> handle. But in rebuilding our mail system I began looking at NFS
> alternatives and came down to OpenAFS and RedHat GFS.
>
> After a ton of reading and a few usenet posts it came down to AFS
> probably was not the solution and to try GFS.
>
> Ok, so my current plan is figuring out how to mount the raid10
> parition /dev/sda1 on the "server" using iSCSI on the 5 mail heads. I
> got the iSCSI part working in a test setup with 2 machines, 1 being
> the sever and 1 being the client. This worked great, doing a test
> transfer of a 771meg maildir took 24 secs compared to the same test
> over NFS of 6 mins 32 secs.
>
> SO. Now I am at the point that I THINK I should be able to setup iscsi
> on 5 mail heads to mount this server's /dev/sda1 over gigabit ethernet
> using jumbo frames on a private switch (so bandwidth should not be a
> problem?) and then setup GFS as the filesystem for this /dev/sda1 so
> that all 5 can mount it RW.
>
> So is this actually possible or am I wasting my time? At this point I
> FINALLY got the damn cluster-1.0.3 from cvs to compile with the latest
> 2.6.19.2 kernel and some hack files from kernel.org for fs/dlm/
> Kconfig, lowcomms-tcp, locomms-sctp, Makefile and got GFS to compile
> as kernel modules and the cluster stuff compiled.
>
> But I cant seem to get it to work, cluster.conf is complaining about
> stuff and whatnot, hopefully more reading will give me answers, but is
> all this work and research in vane?
>
> My current test setup is using FC6, kernel 2.6.19.2, i have the
> cluster-1.0.3, i installed gfs2-utils, i have the iscsi target. From
> what I can tell I just need to finish getting GFS to work and then i
> should be able to mount my first gfs filesystem over iscsi on my test
> client machine.
>
> Sorry for the long post, hopefully I didnt forget anything.
Hi Brandon,

You should be able to accomplish what you've described.

You told us a lot about your environment and what you're trying to do,
but unfortunately you didn't send us your cluster.conf file or the
complaint messages you were referring to.

Also, you didn't tell us if the cman service is starting okay on all nodes,
etc.  You also didn't mention whether you're using clvmd, the clustered
logical volume manager, etc.

Regards,

Bob Peterson
Red Hat Cluster Suite