Things to do this week instead of arguing about mixers

[Eric Sandeen]

Basil Mohamed Gohar wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 04/28/2009 03:36 AM, Eric Sandeen wrote:
>> Seth Vidal wrote:
>>> On Mon, 27 Apr 2009, Eric Sandeen wrote:
>>>
>>>> * Come up with some template autoconf magic to make it easy for
>>>>  apps to detect fallocate() at build time, and some example
>>>>  code on how to use it
>>>>  - Should it fall back to posix_fallocate if fallocate is absent?
>>>>
>>>> * Decide on some consistent buildt-time, run-time, and
>>>>  configuration behavior when enabling this
>>>>  - should build time use posix_fallocate if only it is available?
>>>>  - config enabled == use fallocate whenever the fs supports it?
>>>>  - config enabled == fall back to posix_fallocate or not?
>>>>  - I'd be happy enough with exclusively using fallocate()
>>>>
>>>> * Come up with a list of apps which could benefit:
>>>>  - all torrent clients?
>>>>  - rsync? (some patches have floated before)
>>>>  - rpm? (file installation and/or db files?)
>>>>  - databases?
>>>>  - file downloaders?
>>>>  - virt image tools?
>>>>  - ____ ?
>>>>
>>>> * Work with Fedora package maintainers and/or upstream to get this
>>>>  hooked up where appropriate
>>> Is any of the above implemented at a layer where yum could take advantage 
>>> of it? b/c I can imagine being able to say "we're going to need X amount 
>>> of space to download all these pkgs" and then have rpm say "and we'll need 
>>> X amount of space in order to actually RUN the transaction"
>> This reservation is really done on a per-file granularity; it's not for
>> doing things like "reserve me 8G of space for some number of files I may
>> write later" - it's more like "reserve me 5.2M of space on this file
>> descriptor; ok that worked now I'll write to it; rinse; repeat"
>>
>> So I'm not sure this directly helps the case where, say, the test
>> transaction squeaks in with enough space, something else uses up some
>> space after that, and then the real install of the rpm set fails ...
>>
>> -Eric
>>
> I think for downloading RPMs, yum would know on a per-file basis how
> much space was needed.  I think this is what Seth meant.
> 
> Likewise, for installing, it's possible (I think?) from the RPM file
> itself to know the sizes of the files that will be created during the
> transaction, and thus preallocate each one.  This logic would go into
> RPM itself, I guess.
> 
> So, that would mean that files will be more contiguous at install time,
> which would be great...if possible.

Yep, this is how I'd envisioned RPM using it.  I might only use it for
larger files; generally files of a few K should be allocated just fine
w/o needing the interface - especially when delayed allocation is in play.

> None of this would guarantee that installed files would be physically
> close together on the disk, though, right?  I imagine locality of
> reference could theoretically be taken advantage of when an RPM is
> installing all files for a certain application, which could speed up
> load time as well...

Unfortunately there is no context between the allocation calls, just the
normal allocator heuristics that will hopefully do a good job.

There has been talk of a defragmenter interface which could do this sort
of rearranging.

-Eric