[Linux-cluster] Ext3/ext4 in a clustered environement

[Colin Simpson]

On Fri, 2011-11-04 at 14:05 -0400, Nicolas Ross wrote:
> Hi !
> 
> So, I'm thinking of putting each of thos directories into a single ext4 
> filesystem of about 100 gigs to speed up all of those process. Where those 
> huge directories are used, they are used by one service and one service 
> alone. So, I would do a file system in cluster.conf, something like :
> 
> <fs device="/dev/VGx/documentsA" force_unmount="1" fstype="ext4" 
> mountpoint="/GFSVolume1/Service1/documentsA" name="documentsA" 
> options="noatime,quota=off"/>
> 
> So, first, is this doable ?

Yup, I have had some tasks that have needed to switch over to ext4 from
GFS2 (I'm on RHEL6 in case any of this makes a difference).

I'm using,

<fs device="/dev/cluvg00/lv00home" force_fsck="1" force_unmount="1"
mountpoint="/data/home" name="homefs" nfslock="1" options="acl"
quick_status="0" self_fence="0"/>

I fully let cluster.conf manage ext4 and have no mention of it in fstab.

> 
> Second, is this risky ? In the sens of that with force_unmont true, I assume 
> that no other node would mount that filesystem before it is unmounted on the 
> stopping service. I know that for some reason umount could hang, but it's 
> not likely since this data is mostly read-only. In that case the service 
> would be failed and need to be manually restarted. What would be the 
> consequence if the filesystem happens to be mounted on 2 nodes ?

The failing to umount mostly happens to me because I NFS export this
file system. Now in theory the cluster should take care of this by
freeing the NFS lockd's, but doesn't always happen for me. But you are
probably in a better position as it doesn't sound like you are doing NFS
on this. I've never seen it fail when NFS isn't involved on a fs. 

If the filesystems fails to umount, the service gets marked as failed,
so won't start on another node (and so won't mount on another node). 
However badness will happen if you manually disable the service and
reenable it on another node. The other node will assume the filesystem
isn't mounted anywhere else and mount it itself. The "solution", of
course, is to check any failed service to see where it was last running
on and make sure it's dependant fs's are umounted from that node, before
disabling it and bringing it back up. 

There was a resource agent patch floating around somewhere (that didn't
make it in so far) that would (as I remember) lock the clvmd to prevent
double mounting of non-clustered fs's. But I guess most people are using
GFS2 so not really a priority.

As you say below a failing to umount can be tackled by a self_fence, but
I haven't needed to go there yet. 

Also depending on how quickly you need the service back, quick_status
and force_fsck will have to be set accordingly. I wanted the paranoia of
checking for a good file system, others may want faster start times. 

> 
> One could add self_fence="1" to the fs line, so that even if it fails, it 
> will self-fence the node to force the umount. But I'm not there yet.
> 
> Third, I've been told that it's not recommended to mount a file system like 
> this "on top" of another clustered fs. Why is that ? I suppose I'll have to 
> mount under /mnt/something and symlink to that.
> 

Don't know on this. Maybe due to extra dependency issues that might
effect operations (or maybe just not thoroughly tested in GFS2, as not a
priority).  

> Thanks for any insights. 

Hopefully

Thanks

Colin

> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster

This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed.  If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the original.