[Linux-cluster] gfs_fsck fails on large filesystem

[Stephen Willey]

We fscked the filesystem because we'd started seeing the following
errors following a power failure.

GFS: fsid=nearlineA:gfs1.0: fatal: invalid metadata block
GFS: fsid=nearlineA:gfs1.0:   bh = 2644310219 (type: exp=4, found=5)
GFS: fsid=nearlineA:gfs1.0:   function = gfs_get_meta_buffer
GFS: fsid=nearlineA:gfs1.0:   file =
/usr/src/redhat/BUILD/gfs-kernel-2.6.9-49/smp/src/gfs/dio.c, line = 1223
GFS: fsid=nearlineA:gfs1.0:   time = 1154425344
GFS: fsid=nearlineA:gfs1.0: about to withdraw from the cluster
GFS: fsid=nearlineA:gfs1.0: waiting for outstanding I/O
GFS: fsid=nearlineA:gfs1.0: telling LM to withdraw
lock_dlm: withdraw abandoned memory
GFS: fsid=nearlineA:gfs1.0: withdrawn

And another instance:

GFS: fsid=nearlineA:gfs1.1: fatal: filesystem consistency error
GFS: fsid=nearlineA:gfs1.1:   inode = 2384574146/2384574146
GFS: fsid=nearlineA:gfs1.1:   function = dir_e_del
GFS: fsid=nearlineA:gfs1.1:   file =
/usr/src/redhat/BUILD/gfs-kernel-2.6.9-49/smp/src/gfs/dir.c, line = 1495
GFS: fsid=nearlineA:gfs1.1:   time = 1154393717
GFS: fsid=nearlineA:gfs1.1: about to withdraw from the cluster
GFS: fsid=nearlineA:gfs1.1: waiting for outstanding I/O
GFS: fsid=nearlineA:gfs1.1: telling LM to withdraw
lock_dlm: withdraw abandoned memory
GFS: fsid=nearlineA:gfs1.1: withdrawn



Running gfs_fsck -vvv -y /dev/gfs1_vg/gfs1_lv

Returns the following after chewing all the physical and swap RAM. The
machines have 4Gb or RAM and 2Gb of swap.  We can increase the swap
size, but is this just gonna keep running out of RAM?

We're running on x86_64 so it can use as much memory as it likes.  The
filesystem is roughly 45Tb.


Initializing fsck
Initializing lists...
Initializing special inodes...
Setting block ranges...
Creating a block list of size 11105160192...
Unable to allocate bitmap of size 1388145025
Segmentation fault
[root at ns1a ~]# gfs_fsck -vvv -y /dev/gfs1_vg/gfs1_lv
Initializing fsck
Initializing lists...
(bio.c:140)     Writing to 65536 - 16 4096
Initializing special inodes...
(file.c:45)     readi:  Offset (640) is >= the file size (640).
(super.c:208)   8 journals found.
(file.c:45)     readi:  Offset (7116576) is >= the file size (7116576).
(super.c:265)   74131 resource groups found.
Setting block ranges...
Creating a block list of size 11105160192...
(bitmap.c:68)   Allocated bitmap of size 5552580097 with 2 chunks per byte
Unable to allocate bitmap of size 1388145025
(block_list.c:72)       <backtrace> - block_list_create()
Segmentation fault


-- 
Stephen Willey

Senior Systems Engineer, Framestore-CFC
+44 (0)207 344 8000
http://www.framestore-cfc.com