malloc and 'Active' memory

[Rick Stevens]

Matthijs van der Klip wrote:
> Hi,
> 
> I hope someone can enlighten me on the behaviour of malloc with respect to 
> 'Active' memory.
> 
> Circumstances:
> 
> I have installed FC4 x86_64 on a Dual Opteron machine with 8GB of physical 
> memory. I am planning to use this box solely for hosting a large MySQL 
> database, so I configured MySQL 4.1 (official x86_64, glibc2.3 rpm) to use 
> 7GB for it's InnoDB buffer pool (innodb_buffer_pool_size). So far this 
> works flawlessly.
> 
> 
> The problem:
> 
> After doing several compiles on the machine (it is my only x86_64 machine 
> at the moment), some memory is used by the kernel for several uses. Let's 
> have a look at /proc/meminfo:
> 
> MemTotal:      8124080 kB
> MemFree:        385772 kB
> Buffers:        201632 kB
> Cached:         661568 kB
> SwapCached:          0 kB
> Active:        5958788 kB
> Inactive:      1463764 kB
> HighTotal:           0 kB
> HighFree:            0 kB
> LowTotal:      8124080 kB
> LowFree:        385772 kB
> SwapTotal:           0 kB
> SwapFree:            0 kB
> Dirty:               0 kB
> Writeback:           0 kB
> Mapped:         270844 kB
> Slab:           284044 kB
> CommitLimit:   4062040 kB
> Committed_AS:  2618264 kB
> PageTables:      16772 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed:       288 kB
> VmallocChunk: 34359738043 kB
> HugePages_Total:     0
> HugePages_Free:      0
> Hugepagesize:     2048 kB
> 
> 
> Only roughly 375 MB 'free' memory left. Not a problem by itself, the 
> kernel is expected to make good use of the memory it can get. However when 
> I stop MySQL and then try to restart it, I get the following error:
> 
> 050726 13:56:21  InnoDB: Error: cannot allocate 7340048384 bytes of
> InnoDB: memory with malloc! Total allocated memory
> InnoDB: by InnoDB 78086952 bytes. Operating system errno: 12
> InnoDB: Check if you should increase the swap file or
> InnoDB: ulimits of your operating system.
> InnoDB: On FreeBSD check you have compiled the OS with
> InnoDB: a big enough maximum process size.
> InnoDB: We keep retrying the allocation for 60 seconds...
> InnoDB: Fatal error: cannot allocate the memory for the buffer pool
> 
> Even after hours later I cannot reclaim the memory. Only after decreasing
> the InnoDB buffer pool to roughly 1.6GB (found this by trial and error),
> MySQL can be successfully started again. Now it sound rather ridiculous
> that I can use only 1.6GB of 8GB physical memory. Can anybody explain
> where the rest of the memory has gone? According to /proc/meminfo output I 
> would guess it be in the 'Active' region:
> 
> 'Active: Memory that has been used more recently and usually not reclaimed 
> unless absolutely necessary.' 
> (http://www.redhat.com/advice/tips/meminfo.html)
> 
> 
> How do I reclaim this memory? I need it for MySQL. The only way I have 
> found so far is rebooting the machine, but I cannot imagine that would be 
> the ultimate solution.
> 
> Some snippets of MySQL code:
> 
>         ulint   n,           /* in: number of bytes to allocate */
> 
>         ret = malloc(n + sizeof(ut_mem_block_t));
> 
>         if (ret == NULL && retry_count < 60) {
>                 if (retry_count == 0) {
>                         ut_print_timestamp(stderr);
> 
>                         fprintf(stderr,
>                 "  InnoDB: Error: cannot allocate %lu bytes of\n"
> 
> 
> 
> Thanks for your time.

The question is what did MySQL do with the memory?  If it was left
as a malloc'd region, it should get returned to the free pool by the
garbage collection system but it's not instantaneous.  I'd wait a bit,
then check the free space and try again.

If the memory was, instead, given over to some IPC mechansim (shared
memory, unnamed pipes, messages, semaphores, etc.), then it won't be
cleaned up and you'll have to kill those off manually with an ipcrm
command (see "man ipcs", "man ipcrm" for details).

Of course, MySQL should trap all signals and, on receiving a SIGTERM,
SIGHUP, SIGQUIT, etc., go to a routine that releases all of its memory
before it terminates.  That's just good practice and something that is
overlooked by many programmers far too often.
----------------------------------------------------------------------
- Rick Stevens, Senior Systems Engineer     rstevens at vitalstream.com -
- VitalStream, Inc.                       http://www.vitalstream.com -
-                                                                    -
- Millihelen, adj: The amount of beauty required to launch one ship. -
----------------------------------------------------------------------