ext3 efficiency, larger vs smaller file system, lots of inodes...

[Theodore Tso]

On Tue, May 19, 2009 at 09:01:47AM -0700, Joe Armstrong wrote:
> 
> A bit of a rambling subject there but I am trying to figure out if
> it is more efficient at runtime to have few very large file systems
> (8 TB) vs a larger number of smaller file systems.  The file systems
> will hold many small files.

No, it's not really more efficient to have large filesystems ---
efficiency at least in terms of performance, that is.  In fact,
depending on your workload, it sometimes can be more efficiency to
have smaller filesystems, since it the journal is a single choke-point
if you have a fsync-heavy workload.  Other advantages of smaller
filesystems is that it's faster to fsck a particular filesystem.

The disadvantage of breaking up a large filesystem are the obvious
ones; you have less flexibility about space allocation, and you can't
hard link across different filesystems, which can be a big deal for
some folks.

> Is memory consumption higher for the inode tables if
> there are more small ones vs one really large one ?

No, because we don't keep a entire filesystem inode table in memory;
pieces of it are brought in as needed, and when they aren't needed
they are released from memory.  About the only thing which is
permanently pinned into memory are the block group descriptors, which
take up 32 bytes per block group descriptor, where a block group
descriptor represents 32 megabytes of storage on disk.  So 1 GB of
filesystem will require 1k of space, and a 1TB filesystem will require
1 megabyte of memory in terms of block group descriptors.  There are
some other overheads, but most of them are fixed overheads, and
normally not a problem.  The struct superblock data structure a
kilobyte or so, for example.  The buffer heads for the block group
descriptors are 56 bytes per 4k of block group descriptors, so 1
megabytes of block grouptors also requires 14k of buffer heads.

Unless you're creating some kind of embedded NAS system, I doubt
memory consuption will be a major problem for you.

       		       	    	  	      - Ted