[Linux-cluster] GFS file system corruption?

[Matthew B. Brookover]

Aj, thank you for the quick response.

The patches changed the situation.  Postmark now runs without any
errors, gfs_fsck still has some problems. I do not understand why
postmark did not report any errors because the patches all appeared to
be for fsck.

Output from gfs_fsck:

[root at illusion root]# gfs_fsck -y /dev/pool/u_as
Initializing fsck
Clearing journals (this may take a while)
.Cleared journals
Starting pass1
Pass1 complete
Starting pass1b
Pass1b complete
Starting pass1c
Pass1c complete
Starting pass2
Pass2 complete
Starting pass3
Pass3 complete
Starting pass4
Pass4 complete
Starting pass5
Converting 134 unused metadata blocks to free data blocks...
Converting 162 unused metadata blocks to free data blocks...
Converting 131 unused metadata blocks to free data blocks...
Converting 152 unused metadata blocks to free data blocks...
Converting 133 unused metadata blocks to free data blocks...
Converting 132 unused metadata blocks to free data blocks...
Converting 152 unused metadata blocks to free data blocks...
Converting 133 unused metadata blocks to free data blocks...
Converting 168 unused metadata blocks to free data blocks...
Converting 161 unused metadata blocks to free data blocks...
Converting 174 unused metadata blocks to free data blocks...
Converting 147 unused metadata blocks to free data blocks...
Converting 148 unused metadata blocks to free data blocks...
Converting 134 unused metadata blocks to free data blocks...
Converting 134 unused metadata blocks to free data blocks...
Converting 137 unused metadata blocks to free data blocks...
Converting 143 unused metadata blocks to free data blocks...
Converting 138 unused metadata blocks to free data blocks...
Converting 132 unused metadata blocks to free data blocks...
Converting 148 unused metadata blocks to free data blocks...
Converting 156 unused metadata blocks to free data blocks...
Converting 179 unused metadata blocks to free data blocks...
Converting 132 unused metadata blocks to free data blocks...
Converting 133 unused metadata blocks to free data blocks...
Converting 164 unused metadata blocks to free data blocks...
Converting 139 unused metadata blocks to free data blocks...
Converting 133 unused metadata blocks to free data blocks...
Converting 132 unused metadata blocks to free data blocks...
Converting 131 unused metadata blocks to free data blocks...
Converting 133 unused metadata blocks to free data blocks...
Converting 131 unused metadata blocks to free data blocks...
Converting 152 unused metadata blocks to free data blocks...
Converting 133 unused metadata blocks to free data blocks...
Converting 141 unused metadata blocks to free data blocks...
Converting 151 unused metadata blocks to free data blocks...
Converting 150 unused metadata blocks to free data blocks...
Converting 174 unused metadata blocks to free data blocks...
Converting 132 unused metadata blocks to free data blocks...
Converting 170 unused metadata blocks to free data blocks...
Converting 140 unused metadata blocks to free data blocks...
Converting 150 unused metadata blocks to free data blocks...
Converting 131 unused metadata blocks to free data blocks...
Converting 133 unused metadata blocks to free data blocks...
Converting 135 unused metadata blocks to free data blocks...
Converting 145 unused metadata blocks to free data blocks...
Converting 143 unused metadata blocks to free data blocks...
Converting 187 unused metadata blocks to free data blocks...
Converting 166 unused metadata blocks to free data blocks...
Converting 152 unused metadata blocks to free data blocks...
Converting 133 unused metadata blocks to free data blocks...
Converting 152 unused metadata blocks to free data blocks...
Converting 168 unused metadata blocks to free data blocks...
Converting 180 unused metadata blocks to free data blocks...
Converting 139 unused metadata blocks to free data blocks...
Converting 134 unused metadata blocks to free data blocks...
Converting 145 unused metadata blocks to free data blocks...
Converting 178 unused metadata blocks to free data blocks...
Converting 131 unused metadata blocks to free data blocks...
Converting 135 unused metadata blocks to free data blocks...
Converting 140 unused metadata blocks to free data blocks...
Converting 131 unused metadata blocks to free data blocks...
Converting 161 unused metadata blocks to free data blocks...
Converting 160 unused metadata blocks to free data blocks...
Converting 151 unused metadata blocks to free data blocks...
Converting 132 unused metadata blocks to free data blocks...
Converting 171 unused metadata blocks to free data blocks...
Converting 156 unused metadata blocks to free data blocks...
Converting 183 unused metadata blocks to free data blocks...
Converting 132 unused metadata blocks to free data blocks...
Converting 142 unused metadata blocks to free data blocks...
Converting 142 unused metadata blocks to free data blocks...
Converting 163 unused metadata blocks to free data blocks...
Converting 166 unused metadata blocks to free data blocks...
Converting 134 unused metadata blocks to free data blocks...
Converting 180 unused metadata blocks to free data blocks...
Converting 156 unused metadata blocks to free data blocks...
Converting 150 unused metadata blocks to free data blocks...
Converting 141 unused metadata blocks to free data blocks...
Converting 143 unused metadata blocks to free data blocks...
Converting 177 unused metadata blocks to free data blocks...
Converting 163 unused metadata blocks to free data blocks...
Converting 136 unused metadata blocks to free data blocks...
Converting 131 unused metadata blocks to free data blocks...
Converting 143 unused metadata blocks to free data blocks...
Converting 174 unused metadata blocks to free data blocks...
Converting 133 unused metadata blocks to free data blocks...
Converting 137 unused metadata blocks to free data blocks...
Converting 133 unused metadata blocks to free data blocks...
Converting 134 unused metadata blocks to free data blocks...
Converting 134 unused metadata blocks to free data blocks...
Converting 132 unused metadata blocks to free data blocks...
Converting 133 unused metadata blocks to free data blocks...
Converting 172 unused metadata blocks to free data blocks...
Converting 169 unused metadata blocks to free data blocks...
Converting 158 unused metadata blocks to free data blocks...
Converting 135 unused metadata blocks to free data blocks...
Converting 131 unused metadata blocks to free data blocks...
Converting 133 unused metadata blocks to free data blocks...
Converting 136 unused metadata blocks to free data blocks...
Converting 131 unused metadata blocks to free data blocks...
Converting 134 unused metadata blocks to free data blocks...
Converting 132 unused metadata blocks to free data blocks...
Converting 140 unused metadata blocks to free data blocks...
Converting 187 unused metadata blocks to free data blocks...
Converting 162 unused metadata blocks to free data blocks...
Converting 174 unused metadata blocks to free data blocks...
Converting 131 unused metadata blocks to free data blocks...
Converting 130 unused metadata blocks to free data blocks...
Converting 159 unused metadata blocks to free data blocks...
Converting 171 unused metadata blocks to free data blocks...
Pass5 complete
[root at illusion root]#


What I did:
* compiled GFS with the patch
* shut down GFS, lock_gulmd, ccsd, and pool manager on all 6 nodes
* loaded the new RPM
* rebooted all 6 nodes and started up the pool manager, and ran gfs_fsck
on all of the file sytems.  It was clean.
* Started gulmd
* mounted the GFS file systems and ran post mark several times. 
Postmark did not report any errors.
* unmounted the GFS file system, and ran gfs_fsck and got the above
messages.

Any ideas?

thanks you for your assistance

Matt

On Mon, 2005-08-29 at 10:48, AJ Lewis wrote:

> On Sun, Aug 28, 2005 at 02:31:13PM -0600, Matthew B. Brookover wrote:
> > I have 6 computers running Redhat Enterrpise 3 release 5, running kernel
> > 2.4.21-32.0.1.ELsmp.
> > 
> > >From the source code I compiled GFS 6.0.2.20-2.  The SAN is an ISCSI
> > based storage system from LeftHand Networks.   Using ext3, the postmark
> > disk test works fine, on a GFS file system, we get a number of errors. 
> > The output from both postmark runs is below.
> 
> Looks like you're running into bugs
> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=160835
> and
> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=160525
> which have been fixed in the latest code.  Unfortunately there is not a GFS
> 6.0 srpm released yet that contains this fix, but I've attached a patch that
> clears them up, as well as a couple other minor issues.
> 
> I'm not sure about the postmark errors you're seeing - hopefully someone else
> on the list can shed some light on them.  But try the fsck with these patches
> and see if there are any actual problems with the fs shown.
> 
> Regards,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20050829/a82bb51d/attachment.htm>