ext3 filesystem is not recognized

[Dennison Williams]

Stephen Samuel wrote:
> Given what  you've described, then only drive that it would make sense to
> pull out would be the one that was dropped and then re-inserted.

I did this with the following set of commands:
mdadm -S /dev/md1
mdadm -A /dev/md1 /dev/sdf /dev/sdg /dev/sdh
mdadm --run /dev/md1
lvchange -a y /dev/volume_group/logical_volume
losetup -e aes /dev/loop1 /dev/volume_group/logical_volume
mount -t ext3 -o ro /dev/loop1 /mnt/logical_volume

and got the same errors; "mount: wrong fs type, bad option, bad
superblock on /dev/loop1"

>>> Check the SMART logs for each of the drives to see if they've had any
>>> problems.
>> there are messages like this:
>> /dev/sdc, failed to read SMART Attribute Data
>> ...but this wasn't one of the disks that was removed from the raid device
> 
> If there are complaints about SDC, then I'd be inclined to do a long test of
> it
> in smart. it's possible that the real problem started here.
> 
> A badblock read test (or just a dd if=/dev/sdc of=/dev/null) would also test
> the I/O path between the drive and the CPU. If there are complaints about
> that drive, then .. at this point, you should consider it suspicious.

Ran "dd if=/dev/sdc of=/dev/null" while monitoring /var/log/messages,
with no messages.  Must have been a fluke.  I will try doing a extended
run of smartctl.

>> Try pullling the (candidate) compromized drive out of the array and see if
>> the (degraded) filesystem works OK and has good data.  If it does, then
> I'd
>> guess that the pulled drive had bad data written to it somehow --- re-add
> it
>> (as if it was hot-swapped in), and hope it doesn't happen again.
>> Try that with each of the  drives, in turn until you find the badly
> written
>> drive.  If one of the drives has badly written data, the system really
> can't
>> tell, for sure, which one is wrong.

I ended up doing this with each drive as above and still the FS wasn't
recognized.

One thing that confuses me though is that the data seems to be partially
valid.  When the array device is assembled and running the logical
volume is recognized, and furthermore losetup accepts the correct
password.  The only thing that doesn't seem to be in working order is
the ext3 filesystem.

In the linux encryption howto
(http://encryptionhowto.sourceforge.net/Encryption-HOWTO-6.html, section
6.1), there is a entry describing possible problems if the kernel was
compiled without CONFIG_BLK_LOOP_DEV_USE_REL_BLOCK.  I can't find this
option anywhere in the config for my kernel (2.6.18-1.2798.fc6xen).  At
this point I am thinking that the problem is at the cryptoloop or ext3
level, but I am not sure what else I can do to check.  Any more ideas?

Sincerely,
Dennison Williams