toasted ext3 filesystem under lvm2

[Dan Stromberg]

This fixed the problem:

On FC3's rescue disk...

1) Do startup network interfaces
2) Don't try to automatically mount the filesystems - not even readonly
3) lvm vgchange --ignorelockingfailure -P -a y
4) fdisk -l, and guess which partition is which based on size: the small one was /boot, and the large one was /
5) mkdir /mnt/boot
6) mount /dev/hda1 /mnt/boot
7) Look up the device node for the root filesystem in /mnt/boot/grub/grub.conf
8) A first tentative step, to see if things are working: fsck -n /dev/VolGroup00/LogVol00
9) Dive in: fsck -f -y /dev/VolGroup00/LogVol00
10) Wait a while...  Be patient.  Don't interrupt it
11) Reboot


On Wed, 15 Dec 2004 15:41:04 -0800, Dan Stromberg wrote:

> 
> I have a Fedora Core 3 system at home, that was running fine, but now
> won't boot.
> 
> Someone shut the power off on it without doing an orderly shutdown, and
> also I sometimes apply patches with "yum -y update" without doing a reboot
> immediately afterward - I suppose either of these could be related to my
> system not booting.
> 
> I have a lot of information about the early stages of the problem at:
> 
> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=142737
> 
> In short, FC3 won't boot, and the FC3 rescue CD's automatic filesystem
> mounting crashes.  Mounting the filesystem manually errors with "invalid
> argument".  I suspect this is either ext3 corruption, or lvm2 corruption,
> or both.
> 
> I've made a copy of the partition on another system, and have been testing
> various things on that copy, like various fsck commands, e2salvage,
> e2extract, but fsck complains about bad superblocks, e2salvage apparently
> ran out of memory, and e2extract just listed millions of 0 length files.
> 
> I wrote a small python script that hunts for ext3 magic numbers, and it
> found some in both a known-good ext3, as well as my corrupt partition
> image.  The first offsets were the same, but others were different.  All
> ended in hexadecimal 38.  Does anyone know how to convert such numbers,
> relative to the beginning of the partition in bytes, to an appropriate
> fsck -b argument?  What units does fsck -b take?
> 
> The disk itself appears to be fine, as I can mount its /boot, and I got no
> errors when I dd'd off the partition image.
> 
> When I left for work this morning, I had for loop with 1 million fsck's
> with different -b's (and -vn) running against a copy of the partition, to
> see if it would eventually hit upon a usable superblock (assuming mkfs -n
> isn't doing what it should, and also, I just don't want to type every
> last number...). But it doesn't seem likely to bear fruit, really.
> 
> I also ran memtest86 on the system that had the trouble for a little over
> an hour, but found no errors.
> 
> The machine was a $299 deal from bilsystem.com, which arrived unassembled.
> However, it's been stable until now, other than a time I had to
> replace its RAM.
> 
> Does anyone have any suggestions for me?  I'd really like to get this data
> back!
> 
> PS: I wrote something very much like e2extract for the atari 800 when I
> was in high school...  If anyone has any thoughts about the general
> structure of such a program for ext3...  I might dive into writing one.  A
> small tree diagram of the on-disk data structures involved with 1-n and
> n-1 and n-n relationships might be enough to get a good start on it.  But
> I'd rather not reinvent the wheel if it's already out there.
> 
> Thanks!