[Linux-cluster] Incomplete gfs2_fsck aborting without error

[Tom Lanyon]

Hi list,

Can anyone offer a suggestion on how to further debug an issue with a GFS2 filesystem quietly aborting its fsck:

cman-2.0.115-1.el5.x86_64
gfs-utils-0.1.18-1.el5.x86_64
openais-0.80.6-8.el5.x86_64
kernel-xen-2.6.18-164.el5.x86_64
gfs2-utils-0.1.62-1.el5.x86_64

* new gfs2 filesystem (not in use but had data loaded to it for testing);
* filesystem mounted on 4 nodes started reporting ENOSPC 'No space left on device' at 68% usage;
* filesystem was unmounted on all nodes and gfs2_fsck run on it from one node;
* gfs2_fsck aborted half-way through pass2 with no errors and was then unmountable due to the fsck_dlm lock proto change;
* re-running gfs2_fsck on the device gives the same results, so was re-attempted with verbose flags (output below)... I see a backtrace and then it dumps me back to the shell; still not really enough to help me debug further:

# gfs2_fsck -vvvv /dev/xvdb
(pass2.c:713)   Checking directory inode at block 3041180 (0x2e679c)
(pass2.c:344)   Found . dentry
(link.c:69)     Incremented counted links to 2 for 3041180 (0x2e679c)
(pass2.c:416)   Found .. dentry
(link.c:69)     Incremented counted links to 144 for 3039824 (0x2e6250)
(pass2.c:491)   Found non-dir inode dentry
(link.c:69)     Incremented counted links to 1 for 7443729 (0x719511)
(metawalk.c:246)        Last entry processed.
(pass2.c:713)mplChecking directory inode at block 4587735 (0x4600d7)
(pass2.c:344)   Found . dentry
(link.c:69)     Incremented counted links to 2 for 4587735 (0x4600d7)
(pass2.c:416)   Found .. dentry
(link.c:69)     Incremented counted links to 9 for 22 (0x16)
(metawalk.c:246)        Last entry processed.
(pass2.c:713)   Checking directory inode at block 4587739 (0x4600db)
(metawalk.c:411)        expected count 4 - di_depth 8, leaf depth 6
(metawalk.c:166)        Checking leaf 4588682 (0x46048a)
(pass2.c:491)   Found non-dir inode dentry
(link.c:69)     Incremented counted links to 1 for 4592189 (0x46123d)
(pass2.c:249)   <backtrace> - check_dentry()
(metawalk.c:236)        <backtrace> - check_entries()
(metawalk.c:426)        <backtrace> - check_leaf()
(metawalk.c:892)        <backtrace> - check_dir()
(pass2.c:730)   <backtrace> - pass2()
#

# gfs2_tool sb /dev/xvdb all
  mh_magic = 0x01161970
  mh_type = 1
  mh_format = 100
  sb_fs_format = 1801
  sb_multihost_format = 1900
  sb_bsize = 4096
  sb_bsize_shift = 12
  no_formal_ino = 2
  no_addr = 23
  no_formal_ino = 1
  no_addr = 22
  sb_lockproto = fsck_dlm
#

Regards,
Tom