Overwritten beginning of ext3 filesystem. Recovery?

Florian Weber florian.weber at bn-paf.de
Mon Dec 27 16:38:09 UTC 2010

Hello list

I accidentally trashed the first ~10-20GB of a 1TB ext3 filesystem with
a heedless RAID1 rebuild (excruciating detail below). I'm now looking for
options to get as much as possible of the remaining data back.

I've been searching the web for over a day now but all my results are  
either not
what I need (MBR, partition table and superblock are OK) or too lowlevel
(revocering many thousands of nameless and structureless mails/jpgs/docs just
doesn't cut it here, IMVHO).
My main problem is not that I accidentally deleted files, but that  
basically my
/ directory just went "poof" and left the rest sitting around.

Since the damaged filesystem was clean before my accident, I'm figuring I just
might get most of the data back: even much of the directory structure should
still be there if I only knew how to get at it.

I'd be most grateful for any tips, tools, or even documentation to aid in
writing my own tool.

Thans in advance for your time,
Florian Weber

PS: that _was_ my backup :-( Thanks for not mentioning it.



Starting point:
I've been running the following setup on my machine:
* Two same-size harddisks, currently 1TB, one big partition each -->  
sda[1], sdb[1]
* Linux software RAID1 consisting of these partitions --> md0
* A single ext3 filesystem, default parameters, reserved blocks lowered to 1%
* All system and data inside this single partition, ca. 350-400GB
* (Much too) infrequent backups ... yes, yes, I know, I know ...

After many years, I wanted to move from Gentoo to KUbuntu. No big deal:
* Shutdown PC, pull disk sdb from the RAID
* Install Ubuntu on sda as if working on a blank disk (setup as above, with
one of the RAID1 disks physically missing during the install)
* Boot the new system from sda, still in degraded mode
* Treating sdb like a standalone ext3 disk: mount, copy configs and /home,
* Get the system into working order (config files reconciled, all applications
* Determine that the "old" stuff is not needed anymore
* Put sdb back into the RAID1 and rebuild

What went wrong:
Before the initial shutdown, I did not change the partition type on sdb from
0xFD to 0x83 to prevent RAID autodetection. Booting with sdb  
reattached (to get
at my personal data) would therefore (correctly) have resulted in a  
RAID rebuild --> very bad.

So I figured: I'll attach the disk, boot with "raid=noautodetect" in the
kernel commandline, and I'll be fine. But: unlike my previous setup,  
Ubuntu has
a silent bootloader and I missed my chance to enter the commandline.

And the RAID instantly started rebuilding itself onto my backup disk :-O

I quickly realised what was happening and cleanly shut down my
system (incurring some additional damage from the running rebuild, but  
the worst
was already done). Total running time was about 3 minutes, in parallel to the
system booting up and shutting down.

What I have now:
* A working, new Ubuntu installation on a degraded RAID1 array, without
personal data. I'm currently typing on this system.

* A harddisk (sdb) that previously contained a working system with a total of
350-400GB data, but was subject to a RAID1 rebuild for <3-4 minutes
at <=100MB/sec. The disc is not connected at the moment.

* The MBR on sdb is the new one. That's OK.
* The partition table on sdb is the new one. It looks identical to the  
old one.
* The ext3 superblock on sdb1 is the new one. It's basically the same  
as the old
one. I compared it against one of the (old) backup superblocks at the  
end of the
* I have a dd image of partition sdb1
* I can mount the image of sdb1 and do an ls. I see data from the new  
system. Much content is missing, obviously, since it was not synced  
over yet
* I can "fsck -n" the image of sdb1. Many errors of course ("inode contains
invalid block", "too many illegal blocks", "i_size wrong", "i_Blocks wrong"),
since much stuff was not synced over yet
* At some point, "fsck -n" stops with "illegal indirect block"
* I have not yet tried to "fsck -y". That would be my next step.

* I have 1TB of free space available and can organise more

I do realise this is not for the faint of heart, but I'm done with my  
fainting for this instance ;-)

Still with hope,
Florian Weber

Buergernetz Pfaffenhofen Webmail - http://www.bn-paf.de

More information about the Ext3-users mailing list