2GB memory limit running fsck on a +6TB device

[Theodore Tso]

On Mon, Jun 09, 2008 at 03:50:32PM -0600, Andreas Dilger wrote:
> This message is a bit tricky to nail down because it doesn't exist anywhere
> in the code directly.  It is encoded into "e2fsck abbreviations", and
> the expansion that is normally in the corresponding comment is different.
> It is PR_1_ALLOCATE_DBCOUNT returned from the call chain:
> 	ext2fs_init_dblist->
> 	  make_dblist->
> 	    ext2fs_get_num_dirs()
> 
> which is counting the number of directories in the filesystem, and allocating
> two 12-byte array element for each one.  This implies you have 77M directories
> in your filesystem, or an average of only 10 files per directory?

There are a number of backup solutions that use hardlinks to conserve
space between increment snapshots.  So yeah, with these worklodas
you'll see something like 80-85M inodes, of which 77M-odd will be
directories.  When you combine the vast number of directories used by
these filesystems, and the fact that e2fsck tries to opimize memory
use by observing that on most normal filesystems, most files have
n_link count of 1, which is NOT true on these filesystems used for
backups, e2fsck's tricks to optimize for speed by caching information
to avoid re-reading them from disk end up costing a large amount of
memory.

> I don't know offhand how important the dblist structure is, so I'm not
> sure if there is a way to reduce the memory usage for it.  I believe
> that in low-memory situations it is possible to use tdb in newer versions
> of e2fsck for the dblist, but I don't know much of the details.

Yep, please see [scratch_files] section in e2fsck.conf.  It is
described in the e2fsck.conf(5) man page.

					- Ted