2GB memory limit running fsck on a +6TB device

[santi at usansolo.net]

On Tue, 10 Jun 2008 22:18:00 -0400, Theodore Tso <tytso at mit.edu> wrote:

> True, but it compresses well.  :-) And the aside from the first part
> of the dumpe2fs, the part that I was most interested could have been
> summarized by simply doing a "grep directories dumpe2fs.out".

:D

"grep directories" is available at:

http://santi.usansolo.net/tmp/dumpe2fs_directories.txt.gz (317K)

Full "dumpe2fs" output compressed is 34M and available at:

http://santi.usansolo.net/tmp/dumpe2fs.txt.gz



> But simply looking at your dumpe2fs, and take an average from the
> first 6 block groups which you included in the pastebin, I can
> extrapolate and guess that you have about 63 million directories, out
> of approximately 114 million total inodes (so about 51 million regular
> files, nearly all of which have hard link counts > 1).

# grep directories dumpe2fs.txt | awk '{sum += $7} END {print sum}'
78283294



> BackupPC blows out of the water all of our memory reduction
> hueristics.  I estimate you need something like 2.6GB to 3GB of memory
> just for these data structures alone.  (Not to mention 94 MB for each
> inode bitmap, and 188 MB for each block bitmap.)  The good news is
> that 4GB of memory should do you --- just.  (I'd probably put in a bit
> more physical memory just to be on the safe side, or enable swap
> before running e2fsck).  The bad news is you really, REALLY need a
> 64-bit kernel on your system.

Unfortunately, I have killed the process, in 21 hours only 2.5% of the fsck
is completed ;-(

'scratch_files' directory has grown to 311M

===================================================================
# time fsck -y /dev/sda4
fsck 1.40.8 (13-Mar-2008)
e2fsck 1.40.8 (13-Mar-2008)
Adding dirhash hint to filesystem.

/dev/sda4 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes

/dev/sda4: e2fsck canceled.                                                
                                                       
/dev/sda4: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sda4: ********** WARNING: Filesystem still has errors **********

real    1303m19.306s
user    1079m58.898s
sys     217m10.130s
===================================================================
 
> Because /var/cache/e2fsck is on the same disk spindle as the
> filesystem you are checking, you're probably getting killed on seeks.
> Moving /var/cache/e2fsck to another disk partition will help (or
> better yet, battery backed memory device), but the best thing you can
> do is get a 64-bit kernel and not need to use the auxiliary storage in
> the first place.

I'm trying a fast test with "mount tmpfs /var/cache/e2fsck -t tmpfs -o
size=2048M", but appears that will take a long time to complete too.. so
the next test will be with a 64-bit LiveCD :)



> As far as what to advice to give you, why are you running e2fsck?  Was
> this an advisory thing caused by the mount count and/or length of time
> between filesystem checks?  Or do you have real reason to believe the
> filesystem may be corrupt?

No, it's not related with mount count and/or length of time between
filesystem checks. When booting we get this error/warning:

EXT3-fs warning: mounting fs with errors, running e2fsck is recommended
EXT3 FS on sda4, internal journal
EXT3-fs: mounted filesystem with writeback data mode.

And "tune2fs" returns that ext3 is in "clean with errors" state.. so, we
think that completing a full fsck process is a good idea; what means in
this case "clean with errors" state, running a fsck is not needed?

Thanks again for all the help and advices!!

--
Santi Saez