Smashing EXT3 for fun and profit (or: how to loose all your data)

[Theodore Ts'o]

On Fri, May 13, 2005 at 12:35:16AM +0200, Hans Yperman wrote:
> This tragic history starts actually on windows: MS Word had wiped out
> an important file on a floppy, and I got the task of retrieving what
> was possible.  Using Linux, I made an image with dd,and put it on the
> now extinct EXT3 partition. I used an undelete programma ,  and then
> mounted the image with a loopback device:
> mount -o loop /tmp/image.img /floppy
>   As it turns out,the undeleter managed to screw up the FAT, and the
> loopback device complains about reading past the end of the device. 
> After fixing the floppy on another computer, I come back to the linux
> computer.  The console is full of error messages.

What version of the kernel are you using?  What undelete program were
you using?  Most undelete programs don't require that you mount the
filesystem; in fact, they often require that you *don't* mount them.

> What happened?  A first bug: Linux remounted the loopback-device
> read-only because  of the bad FAT on the image.  BUT this did not work
> out right: not only the loopback device, but the whole EXT3-partition
> were now read-only. Every little write action results in an error,
> hence all the messages.  I did not really think much of it at that
> point, and just did a
> mount -o remount,rw /

Without the logs, it sounds like the ext3 filesystem got corrupted,
and so it was mounted remounted read-only.  How this happened is not
clear, and you didn't give us enough information to determine that;
but it's consistent with e2fsck displaying errors.

> At this point, I am already screwed, but I don't realize it yet:  The
> computer works completely normal from here on.  The problem happens
> the next time I boot: fsck complains about problems (weird, fsck is
> not supposed to run for EXT3).  

When the kernel discovered a filesystem corruption, it marks the
filesystem as containing errors, and remounts it read-only.  When fsck
will run, it will note the fact that filesystem has problems, and try
to fix it.

> Specifically, fsck complains about
> double-allocated blocks, does a pass 1B and 1C (I'd never seen these
> before either), dumps pages and pages and pages of block numbers,
> get's very very veeeeryyy slow, and crashes.  I restart fsck.  This
> time it starts asking me tons of yes/no questions because it wants to
> know what to do with the double-allocated block.  I yes them all
> (There is no real right answer anyhow) and reboot.

What version of e2fsck are you running?  It must be an ancient one if
got really slow like that.  You wouldn't be running Debian
Obsolete^H^H^H^H^H^H^H Stable, are you?

> And that was it:  init starts, and complains about not having an
> /etc/inittab (and asks me which runlevel to start.  Never seen that
> before either). Then it crashes.   Booting with knoppix reveals lots
> and lost of damaged files.  Everything that was cached seems to be
> damaged, and some random files are also dead (my gues is ext3 screwed
> up while updating atimes or something like that).  Game over.

The filesystem was probably screwed up much earlier than that.
Probably something with the undelete program was run, or perhaps
because you remounted the filesystem read-write after errors were
uncovered, but it's going to be hard to reconstruct without a lot more
details.  (What specific messages were printed by the kernel
describing the errors, exactly what version of the kernel, e2fsprogs,
and undelete program you were using, etc.)

I will say that while remounting a filesystem read/write after errors
is dangerous, the fact that e2fsck displayed pages and pages of block
numbers tends to indicate that that there was something more that went
wrong.  Merely remounting a filesystem read/write might result in a
some multiply claimed blocks, which pass 1b/1c/1d are designed to
resolve, but how many you have depends on how many files are written
and how badly corrupted were the block allocation bitmaps.  

Assuming that you didn't run the system for very long before you
rebooted, or didn't write a lot of files during this interim, it seems
somewhat unlikely that it would have resulted in "pages and pages and
pages" of block numbers.  That would tend to argue that portions of
the inode table got written to the wrong location, which is generally
caused by a hardware error.  It might have been caused by the undelete
program, but that seems hard to believe.  But then again, I don't know
which undelete program you used, and it does seem very surprising that
the undelete program would work with a mounted filesystem, so that
part sounds like another user error (but not one that would be
expected to cause major filesystem corruption).  So the bottom line is
I can't really tell you what could have happened with the limited
facts that you've given me.

> I guess these 2 facts need fixing:
> 1) loopback devices should not pass errors over to their underlying filesystems.

Loopback devices don't pass errors back over to their underlying
filesystems.  

> 2) ext3 suicidally allows remounting read-write when parts of its data
> are invalid.

Linux will allow you to do many things that might be, well,
ill-advised.  When the kernel printed all of the warnings, it warned
you that the filesystem had errors.  Remounting it read/write was a
really bad idea --- but then again, so is running the command
"dd if=/dev/zero of=/dev/hda1" as root.


> Other people might not like loosing a whole partition, so I mail this
> sad story to you all.  A bit of advice: if you ever see ext3
> complaining about being read-only, press the reset button.  It might
> save your partition.

Or run e2fsck manually yourself; there are a number of things that you
can do.  Blindly remounting the filesystem read/write is certainly not
one of them.  Saving all of the error messages from the kernel
describing the filesystem corruption is a really good idea.  As is
saving the messages from e2fsck, so people can figure out what
happened after the fact.

The one good thing is that you kept good backups, so you didn't lose
that much; I definitely commend that.  :-)

						- Ted