[Linux-cluster] fsck.gfs2 The root dinode block is destroyed.

[Bob Peterson]

----- Original Message -----
> Good Morning,
> 
> We have a large cluster with 50 gfs2 SAN mounts.  The mounts range in
> size from 1TB to 15TB each.  We have some with 6-8TB of data but most
> average around 3TB used right now.  We were doing network testing a
> while back to check our redundancy incase of a switch failure, and the
> tests failed..  multiple times.  We ended having the SAN mounts yanked
> out from under the cluster.  Long story short, we seem to have
> corruption.  I can still bring the volumes up with the cluster but
> when i take everything down and do a fsck I get the following:
> 
> 
> (ran with fsck -n /dev/$device)
> 
> Found a copy of the root directory in a journal at block: 0x501ca.
> Damaged root dinode not fixed.
> The root dinode should be at block 0x2f3b98b7 but it seems to be destroyed.
> Found a copy of the root directory in a journal at block: 0x501d2.
> Damaged root dinode not fixed.
> The root dinode should be at block 0x28a3ac7f but it seems to be destroyed.
> Found a copy of the root directory in a journal at block: 0x501da.
> Damaged root dinode not fixed.
> Unable to locate the root directory.
> Can't find any dinodes that might be the root; using master - 1.
> Found a possible root at: 0x16
> The root dinode block is destroyed.
> At this point I recommend reinitializing it.
> Hopefully everything will later be put into lost+found.
> The root dinode was not reinitialized; aborting.
> 
> 
> This particular device had 4698 "seems to be destroyed..  found a
> copy" messages before the final, "Can't find any dinodes" message.  I
> fear that we have a number of mounts in this state.
> 
>  Is there any way to recover?  Thanks in advance.

Hi Megan,

I can't tell what's "really" going on unless I examine the GFS2 file system
metadata up close. If you save the metadata (gfs2_edit savemeta <device> <file>)
and also the first 1MB of the block device, and somehow get it to me, I might
be able to figure out what's going on, how it got that way, and what to do to
recover it. Ordinarily, the root directory appears early in the metadata and
it should not be deleted. What was the history of the file system? Was it
converted from GFS1 with gfs2_convert or something?

Regards,

Bob Peterson
Red Hat File Systems