[Linux-cluster] GFS Problem: invalid metadata block

[Matt Eagleson]

Hello,

I have been evaluating a GFS cluster as an NFS solution and have unfortunately run in to a serious problem which I cannot explain.  Both of the GFS filesystems I am exporting became corrupt and unusable.

The system is Redhat AS4 with 2.6.9-42.0.2.ELsmp.  I cannot find anything unusual on the host or the SAN at the time of this error.  Nobody was logged in to the nodes.

Can anyone help me understand what is happening here?

Here are the logs:

Node 2:
1:14:01 file04 kernel: GFS: fsid=nfs:data.0: jid=1: Trying to acquire journal lock...
1:14:01 file04 kernel: GFS: fsid=nfs:data.0: jid=1: Looking at journal...
1:14:01 file04 kernel: GFS: fsid=nfs:data.0: jid=1: Acquiring the transaction lock...
1:14:01 file04 kernel: GFS: fsid=nfs:data.0: jid=1: Replaying journal...
1:14:01 file04 kernel: GFS: fsid=nfs:data.0: jid=1: Replayed 6 of 7 blocks
1:14:01 file04 kernel: GFS: fsid=nfs:data.0: jid=1: replays = 6, skips = 0, sames = 1
1:14:01 file04 kernel: GFS: fsid=nfs:data.0: jid=1: Journal replayed in 1s
1:14:01 file04 kernel: GFS: fsid=nfs:data.0: jid=1: Done
1:14:14 file04 kernel: GFS: fsid=nfs:data.0: fatal: invalid metadata block
1:14:14 file04 kernel: GFS: fsid=nfs:data.0:   bh = 65578 (magic)
1:14:14 file04 kernel: GFS: fsid=nfs:data.0:   function = gfs_rgrp_read
1:14:14 file04 kernel: GFS: fsid=nfs:data.0:   file = /usr/src/redhat/BUILD/gfs-kernel-2.6.9-58/smp/src/gfs/rgrp.c, line = 830
1:14:14 file04 kernel: GFS: fsid=nfs:data.0:   time = 1159924454
1:14:14 file04 kernel: GFS: fsid=nfs:data.0: about to withdraw from the cluster
1:14:14 file04 kernel: GFS: fsid=nfs:data.0: waiting for outstanding I/O
1:14:14 file04 kernel: GFS: fsid=nfs:data.0: telling LM to withdraw
1:14:14 file04 kernel: lock_dlm: withdraw abandoned memory
1:14:14 file04 kernel: GFS: fsid=nfs:data.0: withdrawn
1:14:41 file04 kernel: GFS: fsid=nfs:home.0: fatal: invalid metadata block
1:14:41 file04 kernel: GFS: fsid=nfs:home.0:   bh = 24 (magic)
1:14:41 file04 kernel: GFS: fsid=nfs:home.0:   function = gfs_get_meta_buffer
1:14:41 file04 kernel: GFS: fsid=nfs:home.0:   file = /usr/src/redhat/BUILD/gfs-kernel-2.6.9-58/smp/src/gfs/dio.c, line = 1223
1:14:41 file04 kernel: GFS: fsid=nfs:home.0:   time = 1159924481
1:14:41 file04 kernel: GFS: fsid=nfs:home.0: about to withdraw from the cluster
1:14:41 file04 kernel: GFS: fsid=nfs:home.0: waiting for outstanding I/O
1:14:41 file04 kernel: GFS: fsid=nfs:home.0: telling LM to withdraw
1:14:43 file04 kernel: lock_dlm: withdraw abandoned memory
1:14:43 file04 kernel: GFS: fsid=nfs:home.0: withdrawn

Node 1:
1:14:46 file03 kernel: GFS: fsid=nfs:data.1: fatal: invalid metadata block
1:14:46 file03 kernel: GFS: fsid=nfs:data.1:   bh = 24 (magic)
1:14:46 file03 kernel: GFS: fsid=nfs:data.1:   function = gfs_get_meta_buffer
1:14:46 file03 kernel: GFS: fsid=nfs:data.1:   file = /usr/src/redhat/BUILD/gfs-kernel-2.6.9-58/smp/src/gfs/dio.c, line = 1223
1:14:46 file03 kernel: GFS: fsid=nfs:data.1:   time = 1159924486
1:14:46 file03 kernel: GFS: fsid=nfs:data.1: about to withdraw from the cluster
1:14:46 file03 kernel: GFS: fsid=nfs:data.1: waiting for outstanding I/O
1:14:46 file03 kernel: GFS: fsid=nfs:data.1: telling LM to withdraw
1:14:48 file03 kernel: lock_dlm: withdraw abandoned memory
1:14:48 file03 kernel: GFS: fsid=nfs:data.1: withdrawn
1:15:30 file03 kernel: GFS: fsid=nfs:home.1: jid=0: Trying to acquire journal lock...
1:15:30 file03 kernel: GFS: fsid=nfs:home.1: jid=0: Looking at journal...
1:15:30 file03 kernel: GFS: fsid=nfs:home.1: jid=0: Acquiring the transaction lock...
1:15:30 file03 kernel: GFS: fsid=nfs:home.1: jid=0: Replaying journal...
1:15:30 file03 kernel: GFS: fsid=nfs:home.1: jid=0: Replayed 4 of 4 blocks
1:15:30 file03 kernel: GFS: fsid=nfs:home.1: jid=0: replays = 4, skips = 0, sames = 0
1:15:30 file03 kernel: GFS: fsid=nfs:home.1: jid=0: Journal replayed in 1s
1:15:30 file03 kernel: GFS: fsid=nfs:home.1: jid=0: Done
1:15:58 file03 kernel: GFS: fsid=nfs:home.1: fatal: invalid metadata block
1:15:58 file03 kernel: GFS: fsid=nfs:home.1:   bh = 21 (magic)
1:15:58 file03 kernel: GFS: fsid=nfs:home.1:   function = gfs_rgrp_read
1:15:58 file03 kernel: GFS: fsid=nfs:home.1:   file = /usr/src/redhat/BUILD/gfs-kernel-2.6.9-58/smp/src/gfs/rgrp.c, line = 830
1:15:58 file03 kernel: GFS: fsid=nfs:home.1:   time = 1159924558
1:15:58 file03 kernel: GFS: fsid=nfs:home.1: about to withdraw from the cluster
1:15:58 file03 kernel: GFS: fsid=nfs:home.1: waiting for outstanding I/O
1:15:58 file03 kernel: GFS: fsid=nfs:home.1: telling LM to withdraw
1:15:58 file03 kernel: lock_dlm: withdraw abandoned memory
1:15:58 file03 kernel: GFS: fsid=nfs:home.1: withdraw