ext3 with maildir++ = huge disk latency and high load
tytso at mit.edu
Sun Sep 25 21:21:27 UTC 2011
On Sun, Sep 25, 2011 at 12:16:12AM -0600, Andreas Dilger wrote:
> It would be possible to do something like this in the ext4 readdir
> code to do dirent readahead, sort, and then prefetch the inodes
> in order (partially or completely, depending on the directory size),
> but as yet we aren't working on anything at the ext4 level.
What we have in ext4 right now is if we need to do disk i/o to read
from the inode table, we will read in adjacent blocks from the inode
table, on the theory that the effort needed to read in 32k versus 4k
is pretty much the same. So if the inodes were allocated all at the
same time, they will be sequentially ordered, and so the inode table
readahead should help quite a lot.
I'll note that with really large maildirs, especially on a mail server
with many other maildirs, over time the inodes for each individual
file will get scattered all over the place, and so pretty much any
scheme that uses a inode table separate from the blocks where the
directory entries are stored is going to get hammered by this use
Ultimately, the best way to solve this problem is a more intelligent
application that caches the contents of the key headers in a database,
so you don't need to scan the contents of the entire Maildir when
doing common IMAP operations.
More information about the Ext3-users