Things to do this week instead of arguing about mixers

[Ric Wheeler]

Eric Sandeen wrote:
> Now that we have ext4 as the new default filesystem, it'd be nice if we
> can get more applications to take advantage of some of the features.
> 
> One big feature that has already been brought up on the list[1] is file
> preallocation, which allows an application to pre-allocate blocks it
> knows that it will eventually write into, thereby making sure it won't
> run out of space, and also generally getting a more efficient/contiguous
> file layout.
> 
> Only a few applications are taking advantage of this so far, in part
> because it's new.[2]  The transmission bittorrent client is using it,
> but only if you tweak a configfile in (IMHO) non-obvious ways.
> 
> So it'd be great to help more applications take advantage of this, and
> evangelize the interface a bit, and maybe do it in a semi-organized
> fashion.  First a bit of background on what this interface is:
> 
> Filesystems which can flag ranges of blocks as allocated but not
> initialized can preallocate those blocks to a file very quickly, without
> needing to write 0s or do any actual file data IO to perform the
> allocation.  When read, they return 0 because they are flagged as
> uninitialized.

This is a great idea!

> 
> ext4 (as well as btrfs, ocfs2, and xfs) has an ->fallocate inode
> operation which is the hook to this fast preallocation interface.  The
> low-level interface is via a syscall, but this is unlikely what we'd
> like the applications to use directly.  There are 2 other paths to the
> interface:
> 
> posix_fallocate(3):
> int posix_fallocate(int fd, off_t offset, off_t len);
> 
> This has existed for some time, but recent glibc will call the efficient
> syscall if the underlying filesystem supports it.  It will fall back to
> essentially writing 0s to the file if not (or if on older glibc), and
> this may not be desired behavior; preallocating 10G could take a very
> long time this way.
> 
> fallocate(2):
> long fallocate(int fd, int mode, loff_t offset, loff_t len);
> 
> This is directly wired to the syscall, so only succeeds on filesystems
> that support it.  It also takes a FALLOC_FL_KEEP_SIZE mode argument,
> which allows one to allocate blocks without updating the file size if
> desired (blocks can then be allocated past EOF).  This call is only
> wired up in very recent glibc, but it is available in F11.
> 
> So, tasks I see to be done, to get this started:
> 
> * Come up with some template autoconf magic to make it easy for
>   apps to detect fallocate() at build time, and some example
>   code on how to use it
>   - Should it fall back to posix_fallocate if fallocate is absent?

I would not fall back to posix_fallocate() if fallocate() is not there since it 
could be horribly slow to preallocate & zero out very large files. That would 
probably discourage users (and application writers) from using this neat mechanism.

> 
> * Decide on some consistent buildt-time, run-time, and
>   configuration behavior when enabling this
>   - should build time use posix_fallocate if only it is available?
>   - config enabled == use fallocate whenever the fs supports it?
>   - config enabled == fall back to posix_fallocate or not?
>   - I'd be happy enough with exclusively using fallocate()
> 
> * Come up with a list of apps which could benefit:
>   - all torrent clients?
>   - rsync? (some patches have floated before)
>   - rpm? (file installation and/or db files?)
>   - databases?
>   - file downloaders?
>   - virt image tools?
>   - ____ ?

GNU tar?
cp?

> 
> * Work with Fedora package maintainers and/or upstream to get this
>   hooked up where appropriate
> 
> * (Make a wiki page or a tracker bug to follow all this?)
> 
> Whaddya say?  Anyone want to help with this?
> 
> Thanks,
> -Eric
> 
> [1]https://www.redhat.com/archives/fedora-devel-list/2009-April/msg00110.html
> [2]https://www.redhat.com/archives/fedora-devel-list/2009-April/msg02078.html
>