[Libguestfs] [PATCH 1/2] Preallocate output file

[Pádraig Brady]

On 10/23/2013 07:37 AM, Richard W.M. Jones wrote:
> On Wed, Oct 23, 2013 at 03:38:30AM +0100, Pádraig Brady wrote:
> [...]
> 
> By the way, Eric Sandeen solved the problem.  It's a genuine
> misfeature in ext4 called auto_da_alloc which causes a flush on close
> if the file has been truncated (ftruncate or O_TRUNC) and the file
> size is zero bytes.  I added these patches which work around the
> issue:
> 
> http://git.annexia.org/?p=pxzcat.git;a=commitdiff;h=68640d56b2ea96401a1355ab56603b0837058d21
> http://git.annexia.org/?p=pxzcat.git;a=commitdiff;h=05f0de58de6cbcbdc40f5a661d406b3fbe5a9060
> 
>>>     /* Tell the kernel we won't read the output file. */
>>>    posix_fadvise (fd, 0, 0, POSIX_FADV_RANDOM|POSIX_FADV_DONTNEED);
>>
>> I know this isn't related to your patch,
>> but I don't think you can set a flag like this.
>> These are rather operations that need to be done
>> while writing I think.
>>
>> For ref I made the above call after writing various chunks to disk in:
>> https://github.com/pixelb/dvd-vr/blob/master/dvd-vr.c#L322
>> and also dd:
>> http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=commit;h=5f31155
> 
> This does explain why the posix_fadvise call has no effect.
> 
> How about POSIX_FADV_RANDOM?  I would have thought that should be done
> before the write operations ...

Yes that is a flag, but I would have thought that only significant for read().
It just disables the read ahead cache which you probably wouldn't
want to do as I presume you're giving large chunks of the file
to each process thus having the usually very beneficial read ahead
only read an insignificant amount of redundant data at the seek boundaries.

On a similar point with writes; i.e. getting the kernel to do I/O
in parallel to your processing, you might want to look at sync_file_range()
http://lkml.indiana.edu/hypermail/linux/kernel/1005.2/01845.html

cheers,
Pádraig.