"double free or corruption" error using PHP on Fedora Core 3

[Alex Greg]

Hi,


Recently I migrated the front-end of our bulletin board (running
phpBB, patched against the recently highlight vulnerability) to a pair
of servers running Fedora Core 3. I compiled Apache 1.3.33 and PHP
4.3.10 from source.


This morning, the servers slowed down to the point that they were
almost completely unresponsive. After a while I managed to SSH into
the machines and saw that the httpd processes had grown to 35MB each
and were using up all the swap and RAM on the machines (they have 1GB
RAM, 1GB swap). I then set MaxClients to 30 and restarted Apache, but
the slowdown happened again:


top - 18:18:51 up 32 days,  4:51,  1 user,  load average: 15.22, 41.62, 39.75
Tasks:  90 total,   1 running,  89 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.2% us,  0.0% sy,  0.0% ni, 99.7% id,  0.2% wa,  0.0% hi,  0.0% si
Mem:   1033484k total,   898000k used,   135484k free,     1268k buffers
Swap:  1052248k total,   896880k used,   155368k free,    10896k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
19305 nobody    16   0 41892  35m 5732 S  0.0  3.5   0:38.49 /www/bin/httpd
19219 nobody    18   0 41940  29m 5732 S  0.0  3.0   0:20.87 /www/bin/httpd
19290 nobody    18   0 41940  29m 5732 S  0.0  3.0   0:42.64 /www/bin/httpd
19217 nobody    18   0 48780  29m 5736 S  0.0  2.9   0:51.25 /www/bin/httpd
19252 nobody    16   0 41916  29m 5732 S  0.0  2.9   0:45.81 /www/bin/httpd
19298 nobody    18   0 41940  29m 5732 S  0.0  2.9   0:16.61 /www/bin/httpd
19296 nobody    18   0 44052  25m 5732 S  0.0  2.6   0:42.79 /www/bin/httpd
19294 nobody    18   0 41796  25m 5732 S  0.0  2.5   0:11.85 /www/bin/httpd
19215 nobody    17   0 41892  25m 5732 S  0.0  2.5   0:22.84 /www/bin/httpd
19218 nobody    17   0 41940  25m 5732 S  0.0  2.5   0:17.84 /www/bin/httpd
19221 nobody    16   0 41900  24m 5732 S  0.0  2.5   0:45.56 /www/bin/httpd
19299 nobody    17   0 41796  24m 5732 S  0.0  2.5   0:11.63 /www/bin/httpd
19302 nobody    17   0 41796  24m 5732 S  0.0  2.5   0:13.46 /www/bin/httpd
19220 nobody    16   0 42300  24m 5732 S  0.0  2.4   0:24.97 /www/bin/httpd
19292 nobody    16   0 41796  21m 5732 S  0.0  2.1   0:37.17 /www/bin/httpd
19303 nobody    16   0 43736  19m 5732 S  0.0  1.9   0:45.86 /www/bin/httpd
19300 nobody    18   0 42184  16m 5732 S  0.0  1.6   0:31.80 /www/bin/httpd
19222 nobody    18   0 42264  14m 5732 S  0.0  1.4   0:16.17 /www/bin/httpd
19291 nobody    16   0 41904  13m 5732 S  0.0  1.4   0:12.57 /www/bin/httpd
19293 nobody    16   0 41796  12m 5732 S  0.0  1.3   0:33.04 /www/bin/httpd
19295 nobody    18   0 44040  11m 5732 S  0.0  1.1   0:35.78 /www/bin/httpd
19224 nobody    17   0 42292 8600 5732 S  0.0  0.8   0:15.22 /www/bin/httpd
19304 nobody    16   0 43804 7908 5732 S  0.0  0.8   0:37.06 /www/bin/httpd
19301 nobody    16   0 43736 7316 5732 S  0.0  0.7   0:40.19 /www/bin/httpd
19537 nobody    18   0 21112 7020 5732 S  0.0  0.7   0:05.79 /www/bin/httpd
19250 nobody    16   0 42272 5712 5732 S  0.0  0.6   0:15.20 /www/bin/httpd
19223 nobody    16   0 41940 5248 5732 S  0.0  0.5   0:45.97 /www/bin/httpd
19216 nobody    16   0 41892 4828 5732 S  0.0  0.5   0:42.77 /www/bin/httpd
19287 nobody    18   0 41892 4312 5732 S  0.0  0.4   0:38.22 /www/bin/httpd
19288 nobody    18   0 41892 4096 5732 S  0.0  0.4   0:08.98 /www/bin/httpd



I then checked the error log and found this:

*** glibc detected *** double free or corruption: 0x097ec7e8 ***
*** glibc detected *** double free or corruption: 0x097ec7e8 ***
*** glibc detected *** double free or corruption: 0x097e6d58 ***
*** glibc detected *** double free or corruption: 0x096ec7b8 ***
*** glibc detected *** double free or corruption: 0x09723188 ***
*** glibc detected *** double free or corruption: 0x097ed3b8 ***
*** glibc detected *** double free or corruption: 0x097e8870 ***
*** glibc detected *** double free or corruption: 0x097ffe88 ***
*** glibc detected *** double free or corruption: 0x096ee320 ***
*** glibc detected *** double free or corruption: 0x08f147e8 ***
*** glibc detected *** double free or corruption: 0x08f147f8 ***
*** glibc detected *** double free or corruption: 0x08f147f8 ***

and the same on the other server:

*** glibc detected *** double free or corruption: 0x09957ac0 ***
*** glibc detected *** double free or corruption: 0x09a50710 ***
*** glibc detected *** double free or corruption: 0x09a622e0 ***
*** glibc detected *** double free or corruption: 0x09978768 ***
*** glibc detected *** double free or corruption: 0x099df900 ***
*** glibc detected *** double free or corruption: 0x0993ca08 ***
*** glibc detected *** double free or corruption: 0x0992b6d8 ***
*** glibc detected *** double free or corruption: 0x0992b6d8 ***
*** glibc detected *** double free or corruption: 0x0992b5f8 ***

This bulletin board application has been running fine for a month or
so on a pair of Fedora Core 1 servers running PHP 4.3.9 (which I've
recently upgraded to 4.3.10), and they didn't have the above problem.


Any advice would be much appreciated.


Regards,


-- Alex