toasted ext3 filesystem under lvm2

[Dan Stromberg]

I have a Fedora Core 3 system at home, that was running fine, but now
won't boot.

Someone shut the power off on it without doing an orderly shutdown, and
also I sometimes apply patches with "yum -y update" without doing a reboot
immediately afterward - I suppose either of these could be related to my
system not booting.

I have a lot of information about the early stages of the problem at:

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=142737

In short, FC3 won't boot, and the FC3 rescue CD's automatic filesystem
mounting crashes.  Mounting the filesystem manually errors with "invalid
argument".  I suspect this is either ext3 corruption, or lvm2 corruption,
or both.

I've made a copy of the partition on another system, and have been testing
various things on that copy, like various fsck commands, e2salvage,
e2extract, but fsck complains about bad superblocks, e2salvage apparently
ran out of memory, and e2extract just listed millions of 0 length files.

I wrote a small python script that hunts for ext3 magic numbers, and it
found some in both a known-good ext3, as well as my corrupt partition
image.  The first offsets were the same, but others were different.  All
ended in hexadecimal 38.  Does anyone know how to convert such numbers,
relative to the beginning of the partition in bytes, to an appropriate
fsck -b argument?  What units does fsck -b take?

The disk itself appears to be fine, as I can mount its /boot, and I got no
errors when I dd'd off the partition image.

When I left for work this morning, I had for loop with 1 million fsck's
with different -b's (and -vn) running against a copy of the partition, to
see if it would eventually hit upon a usable superblock (assuming mkfs -n
isn't doing what it should, and also, I just don't want to type every
last number...). But it doesn't seem likely to bear fruit, really.

I also ran memtest86 on the system that had the trouble for a little over
an hour, but found no errors.

The machine was a $299 deal from bilsystem.com, which arrived unassembled.
However, it's been stable until now, other than a time I had to
replace its RAM.

Does anyone have any suggestions for me?  I'd really like to get this data
back!

PS: I wrote something very much like e2extract for the atari 800 when I
was in high school...  If anyone has any thoughts about the general
structure of such a program for ext3...  I might dive into writing one.  A
small tree diagram of the on-disk data structures involved with 1-n and
n-1 and n-n relationships might be enough to get a good start on it.  But
I'd rather not reinvent the wheel if it's already out there.

Thanks!