ext3 efficiency, larger vs smaller file system, lots of inodes...

[Joe Armstrong]

> -----Original Message-----
> From: Ric Wheeler [mailto:rwheeler at redhat.com]
> Sent: Tuesday, May 19, 2009 9:54 AM
> To: Joe Armstrong
> Cc: ext3-users at redhat.com
> Subject: Re: ext3 efficiency, larger vs smaller file system, lots of
> inodes...
> 
> On 05/19/2009 12:28 PM, Joe Armstrong wrote:
> >
> > -----Original Message-----
> > From: Eric Sandeen [mailto:sandeen at redhat.com]
> > Sent: Tuesday, May 19, 2009 9:21 AM
> > To: Joe Armstrong
> > Cc: ext3-users at redhat.com
> > Subject: Re: ext3 efficiency, larger vs smaller file system, lots of
> inodes...
> >
> > Joe Armstrong wrote:
> >> (... to Nabble Ext3:Users - reposted by me after I joined the
> >> ext3-users mailing list - sorry for the dup...)
> >>
> >> A bit of a rambling subject there but I am trying to figure out if
> it
> >> is more efficient at runtime to have few very large file systems (8
> >> TB) vs a larger number of smaller file systems.  The file systems
> >> will hold many small files.
> >>
> >> My preference is to have a larger number of smaller file systems for
> >> faster recovery and less impact if a problem does occur, but I was
> >> wondering if anybody had information from a runtime performance
> >> perspective  - is there a difference between few large and many
> small
> >> file systems ?  Is memory consumption higher for the inode tables if
> >> there are more small ones vs one really large one ?
> >
> > It's the vfs that caches dentries&  inodes; whether they come from
> > multiple filesystems or one should not change matters significantly.
> >
> > The other downside to multiple smaller filesystems is space
> management,
> > when you wind up with half of them full and half of them empty, it
> may
> > be hard to rearrange.
> >
> > But the extra granularity for better availability and fsck/recovery
> time
> > may be well worth it.  It probably depends on what your application
> is
> > doing and how it can manage the space.  You might want to test
> filling
> > an 8T filesystem and see for yourself how long fsck will take...
> it'll
> > be a while.  Perhaps a very long while.  :)
> >
> >> Also, does anybody have a reasonable formula for calculating memory
> >> requirements of a given file system ?
> >
> > Probably the largest memory footprint will be the cached dentries&
> > inodes, though this is a "soft" requirement since it's mostly just
> cached.
> >
> > Each journal probably has a bit of memory requirement overhead, but I
> > doubt it'll be a significant factor in your decision unless every
> byte
> > is at a premium...
> >
> > -Eric
> >
> 
> How you do this also depends on the type of storage you use. If you
> have
> multiple file systems on one physical disk (say 2 1TB partitions on a
> 2TB S-ATA
> disk), you need to be careful not to bash on both file systems at once
> since you
> will thrash the disk heads.
> 
> In general, it is less of an issue with arrays, but still can have a
> performance
> impact.
> 
> Ric

Just for completeness, we will be using Striped LUN's (RAID-6 underneath), so I hope that the striping will distribute the IO's while the RAID-6 device will provide the HA/recovery capabilities.

Joe