[Linux-cluster] GFS2 corruption/withdrawal/crash

[Wendell Dingus]

This recently happened to us and I'm wondering if there's anything else we can do to prevent it and/or more fully recover from it.

3 physical nodes, 64-bit, 2.6.18-128.2.1.el5xen, 3 GFS2 filesystems mounted by each. 

A zero-byte file in a subdir about 3 levels deep that when touched in any way causes total meltdown. Details below...

We took the filesystem offline (all nodes) and ran gfs2_fsck against it. The FS is 6.2TB in size, living on 2gb/sec fibrechannel array. It took 9 hours to complete which was not as bad as I had feared it would be. Afterwards the filesystem was remounted and that zero-byte file was attempted to be removed and the same thing happened again. So it appears gfs2_fsck did not fix it. Since there are 3 GFS filesystems the bad part was that access to all 3 went away when one of them had an issue because GFS itself appears to have crashed. That's the part I don't understand and am pretty sure was not what should have happened.

After a full reboot we renamed the directory holding the "bad" zero-byte file to a directory in the root of that GFS filesystem and are simply avoiding it at this point. 

Thanks...

Description from a co-worker on what he found from researching this:

While hitting something on the filesystem, it runs in to an invalid metadata block, realizes the error & problem, and attempts to take the FS offline because it's bad. (To not risk additional corruption) 

Jul 22 04:11:48 srvname kernel: GFS2: fsid=cluname:raid1.1: fatal: invalid metadata block 
Jul 22 04:11:48 srvname kernel: GFS2: fsid=cluname:raid1.1: bh = 1633350398 (magic number) 
Jul 22 04:11:48 srvname kernel: GFS2: fsid=cluname:raid1.1: function = gfs2_meta_indirect_buffer, file = /builddir/build/BUILD/gfs2-kmod-1.92/_kmod_build_xen/meta_io.c, line = 33 
4 
Jul 22 04:11:48 srvname kernel: GFS2: fsid=cluname:raid1.1: about to withdraw this file system 
Jul 22 04:11:48 srvname kernel: GFS2: fsid=cluname:raid1.1: telling LM to withdraw 
Jul 22 04:11:57 srvname kernel: GFS2: fsid=cluname:raid1.1: withdrawn 
Jul 22 04:11:57 srvname kernel: 

Unfortunately... For some reason, when it completes the withdrawal process, gfs crashes... I'm sure it's not supposed to do that... It should continue allowing access to all of the other GFS filesystems, but since the gfs module is dieing, it kills access to any gfs filesystems. 

Jul 22 04:11:57 srvname kernel: Call Trace: 
Jul 22 04:11:57 srvname kernel: [<ffffffff8854891a>] :gfs2:gfs2_lm_withdraw+0xc1/0xd0 
Jul 22 04:11:57 srvname kernel: [<ffffffff80262907>] __wait_on_bit+0x60/0x6e 
Jul 22 04:11:57 srvname kernel: [<ffffffff80215788>] sync_buffer+0x0/0x3f 
Jul 22 04:11:57 srvname kernel: [<ffffffff80262981>] out_of_line_wait_on_bit+0x6c/0x78 
Jul 22 04:11:57 srvname kernel: [<ffffffff8029a01a>] wake_bit_function+0x0/0x23 
Jul 22 04:11:57 srvname kernel: [<ffffffff8021a7f1>] submit_bh+0x10a/0x111 
Jul 22 04:11:57 srvname kernel: [<ffffffff8855a627>] :gfs2:gfs2_meta_check_ii+0x2c/0x38 
Jul 22 04:11:57 srvname kernel: [<ffffffff8854c168>] :gfs2:gfs2_meta_indirect_buffer+0x104/0x160 
Jul 22 04:11:57 srvname kernel: [<ffffffff8853c786>] :gfs2:recursive_scan+0x96/0x175 
Jul 22 04:11:57 srvname kernel: [<ffffffff8853c82c>] :gfs2:recursive_scan+0x13c/0x175 
Jul 22 04:11:57 srvname kernel: [<ffffffff8853d65a>] :gfs2:do_strip+0x0/0x358 
Jul 22 04:11:57 srvname kernel: [<ffffffff802639f9>] _spin_lock_irqsave+0x9/0x14 
Jul 22 04:11:57 srvname kernel: [<ffffffff8853c8fe>] :gfs2:trunc_dealloc+0x99/0xe7 
Jul 22 04:11:57 srvname kernel: [<ffffffff8853d65a>] :gfs2:do_strip+0x0/0x358 
Jul 22 04:11:57 srvname kernel: [<ffffffff88545149>] :gfs2:gfs2_glock_dq+0x1e/0x132 
Jul 22 04:11:57 srvname kernel: [<ffffffff8020b7bf>] kfree+0x15/0xc5 
Jul 22 04:11:57 srvname kernel: [<ffffffff8853df97>] :gfs2:gfs2_truncatei+0x5e5/0x70d 
Jul 22 04:11:57 srvname kernel: [<ffffffff88545149>] :gfs2:gfs2_glock_dq+0x1e/0x132 
Jul 22 04:11:57 srvname kernel: [<ffffffff88544b28>] :gfs2:gfs2_glock_put+0x1a/0xe2 
Jul 22 04:11:57 srvname kernel: [<ffffffff88550b83>] :gfs2:gfs2_setattr+0xe6/0x335 
Jul 22 04:11:57 srvname kernel: [<ffffffff88550acd>] :gfs2:gfs2_setattr+0x30/0x335 
Jul 22 04:11:57 srvname kernel: [<ffffffff8026349f>] __down_write_nested+0x35/0x9a 
Jul 22 04:11:57 srvname kernel: [<ffffffff8022caf2>] notify_change+0x145/0x2e0 
Jul 22 04:11:57 srvname kernel: [<ffffffff802ce6ae>] do_truncate+0x5e/0x79 
Jul 22 04:11:57 srvname kernel: [<ffffffff8020db96>] permission+0x81/0xc8 
Jul 22 04:11:57 srvname kernel: [<ffffffff80212b01>] may_open+0x1d3/0x22e 
Jul 22 04:11:57 srvname kernel: [<ffffffff8021b1c2>] open_namei+0x2c4/0x6d5 
Jul 22 04:11:57 srvname kernel: [<ffffffff802275c9>] do_filp_open+0x1c/0x38 
Jul 22 04:11:57 srvname kernel: [<ffffffff80219d14>] do_sys_open+0x44/0xbe 
Jul 22 04:11:57 srvname kernel: [<ffffffff8025f2f9>] tracesys+0xab/0xb6 
Jul 22 04:11:57 srvname kernel: 

Tail end of the gfs2_fsck run:
Ondisk and fsck bitmaps differ at block 1633350906 (0x615af4fa)
Ondisk status is 1 (Data) but FSCK thinks it should be 0 (Free)
Metadata type is 0 (free)
Succeeded.
Ondisk and fsck bitmaps differ at block 1633350907 (0x615af4fb)
Ondisk status is 1 (Data) but FSCK thinks it should be 0 (Free)
Metadata type is 0 (free)
Succeeded.
RG #1633288608 (0x615a01a0) free count inconsistent: is 48931 should be 49441
Resource group counts updated
Pass5 complete
Writing changes to disk
gfs2_fsck complete

PS. Access to this file which we know has caused us to crash for sure: mv file1 file2 and rm -f file1 and echo "asdf" >file1