ext3 efficiency, larger vs smaller file system, lots of inodes...

[Ric Wheeler]

On 05/19/2009 12:28 PM, Joe Armstrong wrote:
>
> -----Original Message-----
> From: Eric Sandeen [mailto:sandeen at redhat.com]
> Sent: Tuesday, May 19, 2009 9:21 AM
> To: Joe Armstrong
> Cc: ext3-users at redhat.com
> Subject: Re: ext3 efficiency, larger vs smaller file system, lots of inodes...
>
> Joe Armstrong wrote:
>> (... to Nabble Ext3:Users - reposted by me after I joined the
>> ext3-users mailing list - sorry for the dup...)
>>
>> A bit of a rambling subject there but I am trying to figure out if it
>> is more efficient at runtime to have few very large file systems (8
>> TB) vs a larger number of smaller file systems.  The file systems
>> will hold many small files.
>>
>> My preference is to have a larger number of smaller file systems for
>> faster recovery and less impact if a problem does occur, but I was
>> wondering if anybody had information from a runtime performance
>> perspective  - is there a difference between few large and many small
>> file systems ?  Is memory consumption higher for the inode tables if
>> there are more small ones vs one really large one ?
>
> It's the vfs that caches dentries&  inodes; whether they come from
> multiple filesystems or one should not change matters significantly.
>
> The other downside to multiple smaller filesystems is space management,
> when you wind up with half of them full and half of them empty, it may
> be hard to rearrange.
>
> But the extra granularity for better availability and fsck/recovery time
> may be well worth it.  It probably depends on what your application is
> doing and how it can manage the space.  You might want to test filling
> an 8T filesystem and see for yourself how long fsck will take... it'll
> be a while.  Perhaps a very long while.  :)
>
>> Also, does anybody have a reasonable formula for calculating memory
>> requirements of a given file system ?
>
> Probably the largest memory footprint will be the cached dentries&
> inodes, though this is a "soft" requirement since it's mostly just cached.
>
> Each journal probably has a bit of memory requirement overhead, but I
> doubt it'll be a significant factor in your decision unless every byte
> is at a premium...
>
> -Eric
>

How you do this also depends on the type of storage you use. If you have 
multiple file systems on one physical disk (say 2 1TB partitions on a 2TB S-ATA 
disk), you need to be careful not to bash on both file systems at once since you 
will thrash the disk heads.

In general, it is less of an issue with arrays, but still can have a performance 
impact.

Ric