RFC: Tuning ext3

[Mingming Cao]

On Thu, 2007-05-17 at 16:16 -0400, Tod Hagan wrote:
> All,
> 
> I'm requesting comments from the expert readers of ext3-users on these
> notes for tuning ext3 for performance.
> 
> Most helpful would be feedback pertinent to RHEL 5; as XFS isn't
> supported under Red Hat Enterprise Linux these items are an attempt to
> match XFS performance with ext3.
> 
> These items were culled from a number of sources. Will they be effective
> for achieving the performance goals? Is the risk assessment correct?
> Are any too risky for consideration? Has anything been left out?
> 
> Thanks.
> 
> 
> 
> ext3 Performance Tuning
> 
> This document contains notes about optimizing ext3 filesystems for
> storage nodes on Linux clusters used for scientific research.
> 
> Each of the goals and assumptions has been assigned an identifier in
> square brackets which is referenced in the applicable tuning options.
> 
> Goals:
> 
> [BigSeqIO] Maximize sequential I/O performance with large files (> 1GB),
> including simultaneous access to several large files.
> 
> [MaxStor] Maximize the amount of usable storage.
> 
> [BigFS] Allow large filesystems (> 8 TB).
> 
> [GenNFSPerf] Optimize for NFS and general performance.
> 
> Assumptions:
> 
> [NoHA] High availability isn't needed.
> 
> [IntegFS] The integrity of the filesystem is important -- don't lose
> existing files.
> 
> [Recreate] Newly-created files are unimportant as they can be
> re-created.
> 
> [UPS] Power to the hardware is guaranteed by a UPS.
> 
> [NoBigDir] Directories don't get large.
> 
> [NoSysFiles] The filesystems hold data and not system files.
> 
> Tuning:
> 
> A. Create using -O sparse_super (default) to save space on large
> filesystems. [MaxStor]
> 
> B. Create using -T largefile4 (one inode per 4 megabytes) to avoid
> wasting space on unused inodes. [MaxStor]
> 
> C. Create using -m 0 to reserve no blocks for the super-user.
> [NoSysFiles][MaxStor]
> 
> D. Create using -E stride=N where N matches the underlying RAID.
> [GenNFSPerf]
> 
> E. Use a kernel >= 2.6.19 (patches for extents and 48-bit support,
> requires Ubuntu 7.04 feisty or Fedora Core 7 or custom kernel) to allow
> filesystems > 8TB on Intel/AMD chips. [BigFS]
> 
> F. Use an external journal on a separate high-RPM drive. [GenNFSPerf]
> 
> G. Use a large journal. mkfs -J size=8192 [GenNFSPerf]
> 
> H. Mount using -o orlov to use the Orlov block allocator (default,
> requires 2.6 kernel). Minimizes seeks by clustering files together. No
> risk. [GenNFSPerf]
> 
> I. Mount using -o noatime,nodiratime. No risk. [GenNFSPerf]
> 
> J. Mount using -o reservation to speed writes to multiple files in the
> same directory. No risk. [BigSeqIO][GenNFSPerf]
> 
> K. Mount using -o data=writeback. Comparable to XFS and JFS, relaxes all
> restrictions on writing cached data. Risky.
> [Recreate][UPS][BigSeqIO][GenNFSPerf]
> 
> L. Mount using -o commit=Nsec where N > 5 (requires 2.6 kernel, default
> is 5 seconds). Reduces sync interval. Risky. [Recreate][UPS][GenNFSPerf]
> 
> M. Mount using -o barrier=0 and enable write-back caching on the
> controllers and drives. The most(?) risky. [Recreate][UPS][GenNFSPerf]
> 
> 
Two more things:
1. Using "mke2fs -I [inode size]" option at format time to allow ext3
support larger inode sizes.  The inode size can be any power-of-two
(larger than default 128 bytes size, up to the filesystem block size).
With larger inode,  extended attributes can be stored in inode body
directly, thus greatly improve performance on workloads which heavily
uses extended attributes (xattrs). 

2. Turn on dir_index feature on ext3 fs with " tue2fs -O dir_index".
This will greatly speed  up lookups in directories with lots of files.
This feature also can be enabled via /etc/mkfs.conf comes with
e2fsprogs-1.39


Mingming