Smashing EXT3 for fun and profit (or: how to loose all your data)

[Hans Yperman]

Hello everyone,

I've just lost my whole EXT3 linux partition by what was probably a
bug.  For your reading pleasure, and in the hope there is enough
information to fix this problem in the future, here the story of a
violent ending:

This tragic history starts actually on windows: MS Word had wiped out
an important file on a floppy, and I got the task of retrieving what
was possible.  Using Linux, I made an image with dd,and put it on the
now extinct EXT3 partition. I used an undelete programma ,  and then
mounted the image with a loopback device:
mount -o loop /tmp/image.img /floppy
  As it turns out,the undeleter managed to screw up the FAT, and the
loopback device complains about reading past the end of the device. 
After fixing the floppy on another computer, I come back to the linux
computer.  The console is full of error messages.

What happened?  A first bug: Linux remounted the loopback-device
read-only because  of the bad FAT on the image.  BUT this did not work
out right: not only the loopback device, but the whole EXT3-partition
were now read-only. Every little write action results in an error,
hence all the messages.  I did not really think much of it at that
point, and just did a
mount -o remount,rw /

At this point, I am already screwed, but I don't realize it yet:  The
computer works completely normal from here on.  The problem happens
the next time I boot: fsck complains about problems (weird, fsck is
not supposed to run for EXT3).  Specifically, fsck complains about
double-allocated blocks, does a pass 1B and 1C (I'd never seen these
before either), dumps pages and pages and pages of block numbers,
get's very very veeeeryyy slow, and crashes.  I restart fsck.  This
time it starts asking me tons of yes/no questions because it wants to
know what to do with the double-allocated block.  I yes them all
(There is no real right answer anyhow) and reboot.

And that was it:  init starts, and complains about not having an
/etc/inittab (and asks me which runlevel to start.  Never seen that
before either). Then it crashes.   Booting with knoppix reveals lots
and lost of damaged files.  Everything that was cached seems to be
damaged, and some random files are also dead (my gues is ext3 screwed
up while updating atimes or something like that).  Game over.

I guess these 2 facts need fixing:
1) loopback devices should not pass errors over to their underlying filesystems.
2) ext3 suicidally allows remounting read-write when parts of its data
are invalid.

Now I don't complain much.  I have a 1 day old backup of my home
directory (thanks, unison). I lost all my tweaks to /etc, but, well,
the hard drive image was copied/resized from computer to computer to
computer, and initially started its life under linux 2.0.35 on a
pentium 133Mhz.  A rewrite was probably a good idea anyway.  I lost
all my MP3's, but a very nice girl promised me to help me re-rip them
all from my CD's. (Thanks to ext3 I get to spend some time with a very
sexy girl.  Lots of it by talking and laughing while we wait for lame
to end.  I actually start to think my hard drive should get erased
more often ;-)  ).

Other people might not like loosing a whole partition, so I mail this
sad story to you all.  A bit of advice: if you ever see ext3
complaining about being read-only, press the reset button.  It might
save your partition.

I did not test my claim of the loopback being the bug, as I am busy
reinstalling right now (on EXT2 this time).

Have a nice day, everyone,

Hans.