files missing

[Theodore Ts'o]

On Mon, Nov 22, 2004 at 06:05:56PM -0800, Jeff Dinisco wrote:
> I have a 1 TB ext3 filesystem mounted via iscsi on a redhat 9 system w/
> kernel version - 2.4.20-30.9.  I'm not sure when it happened, but today
> there appears to be about 7,000 files (600GB) missing.  The output from
> df implies that the files are still there.  It shows 861 GB utilized.
> But du shows only 300 GB of data.  I'm sure that there are no processes
> holding onto deleted files because I have unmounted/mounted the
> filesystem several times, synced, etc.  Here's an excerpt from e2fsck
> -nf /dev/sdf ...

The output from df is calculated by using the free blocks field in the
superblock, and is only as accurate as the superblock statistics will
be.

It does look like garbage has been written into your filesystem, and
the situation is probably pretty grim; there will almost certainly be
some data loss.  It may not be as bad as you think, however, since
there may be disconnected inodes (and possibly entire directory
hierarchies) which e2fsck can recover and link into the lost+found
directory. 

> Things look pretty grim right now.  As soon as I run e2fsck -p, these
> inodes will be deleted and I will lose data, correct?  

Actually, e2fsck -y will be needed to fix the data.  E2fsck -p will
make "safe" fixes, but anything that might require human judgement
will cause e2fsck -p to abort (since it is intended to be used in
unattended boot scripts).  You will need to answer each of e2fsck's
questions manually, or use e2fsck -y to cause e2fsck to automatically
assume an answer of "yes" for each question.  I'd strongly suggest
making a bit-for-bit image backup of the filesystem before proceeding,
since it might be possible for an expert to try other means of
recoverying data after getting back what data you can with e2fsck -y.

> I suspect the
> cause of this issue was multiple hosts mounting this filesystem r/w.

Ah, yup, that would do it.  How long was the filesystem mounted by
multiple hosts?

> Does anyone know of any method that could be used to recover this data?
> Any help would be greatly appreciated.

You can try e2fsck -y and hope for the best.  Aside from that, you can
grep the disk looking for a specific text pattern if there is a few
extremely valuable files that have to be recovered at all costs.
Failing that, it will be restore from backup times.....  (you did keep
regular backups, right?  :-)

						- Ted