[Linux-cluster] CLVM/GFS hangs while rebuilding an AoE RAID5 LUN, anyone experienced that before? any clue?

[Bob Peterson]

----- "Abraham Alawi" <a.alawi at auckland.ac.nz> wrote:
| Thanks Steve, Yes, sadly I can confirm the file system has been
| corrupted but I still don't understand why I/O will stop flowing at
| the LVM level (& doesn't fence it either) and why fsck keeps crashing
| without a useful error message, is there any signal I can send to
| gfs_fsck to by pass certain stages? Also to speed up the fsck process,
| I was thinking of utilizing the RAM and increase the read_ahead
| parameter (hdparm -a) of the PV device (an AoE device) by 1GB since
| that will hugely optimize the sequential read and fscking is mostly a
| sequential read process and very bit of writings, what do you think?
| 
| Herein the tail of the last fsck log file:
| (metawalk.c:516)	Extended attributes exist for inode #34020861.
| (metawalk.c:413)	Checking EA leaf block #34020862.
| (pass1.c:485)	Setting block #34020862 to eattr block
| (pass1.c:907)	Checking metadata block 34020862
| (pass1.c:923)	Metadata block 34020862 not an inode or free metadata
| (pass1.c:907)	Checking metadata block 34020863
| (link.c:22)	Setting link count to 1 for 34020863
| (metawalk.c:516)	Extended attributes exist for inode #34020863.
| (metawalk.c:413)	Checking EA leaf block #34020864.
| (pass1.c:485)	Setting block #34020864 to eattr block
| (pass1.c:907)	Checking metadata block 34020864
| (pass1.c:923)	Metadata block 34020864 not an inode or free metadata
| (pass1.c:907)	Checking metadata block 34020865
| (link.c:22)	Setting link count to 1 for 34020865
| (pass1.c:213)	Setting 34020917 to data block
| (pass1.c:213)	Setting 34020918 to data block
| (pass1.c:213)	Setting 34020919 to data block
| (pass1.c:213)	Setting 34020920 to data block
| (metawalk.c:516)	Extended attributes exist for inode #34020865.
| (metawalk.c:413)	Checking EA leaf block #34020866.
| (pass1.c:485)	Setting block #34020866 to eattr block
| (pass1.c:907)	Checking metadata block 34020866
| (pass1.c:923)	Metadata block 34020866 not an inode or free metadata
| (pass1.c:907)	Checking metadata block 34020867
| 
| 
| Thanks,
| 
|   -- Abraham

Hi Abraham,

The only way to tell that is to examine the metadata.  Can you
use "gfs2_edit savemeta" to save the gfs metadata to a file, and
send it to me?  (gfs2_edit works on gfs1 and gfs2 metadata).
I'm willing to take a look at it to see what's wrong.

Be sure you use a very recent version of gfs2_edit though, because
some older versions don't always save everything they should.

If gfs2_edit crashes during the save, there's no good way to solve
this other than to debug gfs_fsck with the gdb debugger.  The gfs_fsck
program would have to be compiled with the -g option for debugging.

Regards,

Bob Peterson
Red Hat File Systems