[Linux-cluster] managing GFS corruption on large FS

[Riaan van Niekerk]

hi all

We have a large GFS consisting of 4 TB of maildir data. We have a 
corruption on this GFS which causes nodes to be withdrawn intermittently.

The cause of the fs corruption is due to user error and lack of 
documentation (initially not having the clustered flag enabled on the VG 
when growing the LV/GFS). We now know better, and will avoid this 
particular cause of corruption. However, management wants to know from 
us how we can prevent corruption, or minimize the downtime incurred if 
this should happen again.

For the current problem, since a gfs_fsck will take too long (we cannot 
afford the 1 - 3 days of downtime it will take to complete the fsck), we 
are planning to migrate the data to a new GFS, and at the same time set 
up the new environment optimally to cause the minimum of downtime, if a 
corruption were to happen again.

One option is to split the one big GFS into a number of smaller GFS's. 
Unfortunately, our environment does not lean itself to being split up in 
(for example) a number of 200GB GFS's. Also, this negates a lot of the 
advantages of GFS (e.g. having your storage consolidated onto one big 
GFS, and scaling it out by growing the GFS and adding nodes).

I would really like to know how others on this list manage the 
threat/risk of FS corruption, and the corruption itself, if it does 
happen. Also, w.r.t. data protection, if you do snapshots, SAN-based 
mirroring, backup to disk/tape, I  would really appreciate it if you 
could give me detail information like
a) mechanism (e.g snaps, backup, etc)
b) type of data (e.g. many small files)
c) size of GFS
d) the time it takes to perform the action

thank you
Riaan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: riaan.vcf
Type: text/x-vcard
Size: 310 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20061129/f7102053/attachment.vcf>