malloc and 'Active' memory

[Deron Meranda]

On 8/1/05, Matthijs van der Klip <matthijs at spill.nl> wrote:
> On Fri, 29 Jul 2005, Rick Stevens wrote:
> > Matthijs van der Klip wrote:
> > Well, malloc() will fail if you request a chunk of memory and there
> > isn't a SINGLE chunk available of that size.  So if memory gets fragged,
> > there isn't a single 7GB chunk available and malloc() will fail.
> > fillmem allocates in smaller chunks, then releases it all so the
> > memory defragger can clean things up.
> 
> I see what you mean and this was entirely my first thought when I ran into
> this problem. However I was told (true or not true) that the malloc
> implementation on Fedora Core 4 could not suffer from memory fragmentation
> in the way I described (e.g. the same way you describe).

The malloc(3) function is entirely userspace code, part of glibc.  It's
not a kernel feature.

The kernel usually allocates memory to processes via the sbrk(2) system
call in whole pages.  From a user-process perspective this can not be
fragmented since the pages are virtually mapped into a linear address
space regardless of their actual physical address.  The malloc(3)
function though will use mmap(2) instead of sbrk(2) when allocating
very large chunks of memory; although it's still only getting whole
pages at a time.

It is true though that the malloc(3) function in glibc is very advanced and
is mostly imune to many forms of traditional memory fragmentation.
For an overview of the algorithms it uses read this document by
Doug Lea: http://gee.cs.oswego.edu/dl/html/malloc.html

It's not exactly the same as the Linux implmentation, but it's pretty
close.  BTW the glibc version comes from ptmalloc2 by Wolfram Gloger,
which is found here:  http://www.malloc.de/en/index.html

Oh, from the malloc.c source code under the function free() we
have this comment:

 * Unless disabled (using mallopt), freeing very large spaces will
 * when possible, automatically trigger operations that give
 * back unused memory to the system, thus reducing program footprint.

You can fine-tune the agressiveness when memory is released back
to the system by using the mallopt(3) call with the option of
M_TRIM_THRESHOLD.  Or you can manually force a release using
the malloc_trim(3) function.  (only the non-fragmented memory is
released).

Deron Meranda