MEMORY ISSUE 2

Thu Mar 8 15:51:37 UTC 2007

Hi,

1) Your kernel version is too old (2.4.21-27.ELsmp) and in fact RedHat's 
  older errata mentions serious memory leak and data corruption problems 
applicable to the version of kernel you are running here:
http://rhn.redhat.com/errata/RHSA-2005-472.html

Get your system to the latest kernel and you might consider running with 
hugemem kernels as well, especially if you ever go above the 16 Gig 
limit. Also it is wise to increase the size of your swap to reflect more 
than 16 Gigs.

2)The /proc/meminfo data you provided below indicate that most of the 
memory is used for caching to increase performance, which is normal and 
default behavior. If your system demands more RAM (aka you launch more 
applications), it will reduce the cache size and provide the RAM to the 
new apps. If you launch more Oracle instances/threads for instance, you 
should see the size of cache decreasing and the size of active memory 
increasing. Depending on the what else is running and at what state it 
is (blocked, running, stopped), inevitably some things will be swapped 
out. Going to 100 Meg or 200 Meg swap (previous data you gave) is not 
really a big issue for a 16 Gig RAM system. If you saw Gigs of swap, 
then yes that would be of concern.

3)The latest data you provide below do not match the system load you 
presented in your original email. The system appear to be less in swap 
now and relatively more quiet than when you reported the problem.

You really should NOT be running Oracle on this old kernel! I would RHN 
the system to the latest and the greatest that your Oracle vendor 
supports, increase the swap partition(s) and probe the system again with 
an up-to-date kernel if I were you. With the data corruption issue, you 
are asking for trouble.

GM

-- 
-- 
George Magklaras

Senior Computer Systems Engineer/UNIX Systems Administrator
EMBnet Technical Management Board
The Biotechnology Centre of Oslo,
University of Oslo
http://www.biotek.uio.no/

EMBnet Norway:    http://www.biotek.uio.no/EMBNET/

Redouane N. wrote:
> Dear All, 
> Check the Output for cat /proc/meminfo & vmstat 1 :
> 
> total: used: free: shared: buffers: cached:
> Mem: 16537456640 16482951168 54505472 0 37412864 13384646656
> 
> Swap: 2093129728 30281728 2062848000
> 
> MemTotal: 16149860 kB
> 
> MemFree: 53228 kB
> 
> MemShared: 0 kB
> 
> Buffers: 36536 kB
> 
> Cached: 13065616 kB
> 
> SwapCached: 5328 kB
> 
> Active: 4166644 kB
> 
> ActiveAnon: 3625432 kB
> 
> ActiveCache: 541212 kB
> 
> Inact_dirty: 8502352 kB
> 
> Inact_laundry: 2309564 kB
> 
> Inact_clean: 263032 kB
> 
> Inact_target: 3048316 kB
> 
> HighTotal: 15531996 kB
> 
> HighFree: 30088 kB
> 
> LowTotal: 617864 kB
> 
> LowFree: 23140 kB
> 
> SwapTotal: 2044072 kB
> 
> SwapFree: 2014500 kB
> 
> Committed_AS: 9139880 kB
> 
> HugePages_Total: 0
> 
> HugePages_Free: 0
> 
> Hugepagesize: 2048 kB
> 
> Linux S0999001 2.4.21-27.ELsmp #1 SMP Wed Dec 1 21:59:02 EST 2004 i686 i686 i386 GNU/Linux
> 
> procs memory swap io system cpu
> 
> r b swpd free buff cache si so bi bo in cs us sy wa id
> 
> 1 1 29572 53212 36536 13065624 0 1 115 138 28 33 6 4 10 80
> 
> 1 1 29572 53116 36544 13065716 0 0 8 364 253 1072 14 0 24 61
> 
> 1 1 29572 52568 36608 13065728 0 0 52 680 296 1984 15 1 27 57
> 
> 18 2 29572 52456 36636 13065780 0 0 12 360 320 961 12 43 14 31
> 
> 1 1 29572 52756 36680 13065788 0 0 0 792 340 2992 16 25 26 34
> 
> 1 1 29572 52636 36692 13065924 0 0 4 580 313 1525 7 1 25 68
> 
> 0 1 29572 52560 36692 13066144 0 0 0 404 293 2336 4 1 24 71
> 
> 0 1 29572 52024 36696 13066324 0 0 4 408 289 2147 3 1 24 72
> 
> 0 1 29572 54684 36696 13066588 0 0 0 400 348 2984 4 3 25 68
> 
> 1 2 29572 54440 36696 13066800 0 0 0 464 345 2035 2 0 25 72
> 
> 0 2 29572 53576 36704 13066852 0 0 0 784 363 2115 2 1 47 50
> 
> 0 1 29572 53456 36708 13066972 0 0 4 456 294 1983 3 0 31 65
> 
> 0 1 29572 53068 36708 13067220 0 0 0 412 290 1761 2 2 26 70
> 
> 0 1 29572 52704 36708 13067452 0 0 0 484 333 2595 3 1 27 69
> 
> 1 1 29572 40472 36712 13067496 0 0 4 544 324 2009 5 2 28 65
> 
> 1 2 29572 51888 36724 13067612 0 0 0 904 397 4443 8 2 27 63
> 
> 1 2 29572 50440 36732 13067668 0 0 0 492 376 3787 7 6 28 60
> 
> 2 1 29572 50288 36732 13067732 0 0 0 436 290 2259 7 9 22 62
> 
> 0 1 29572 51360 36732 13067912 0 0 0 404 307 1900 4 4 24 68
> 
> 0 1 29572 51440 36732 13068176 0 0 0 420 330 2301 3 0 27 70
> 
> 0 2 29572 51272 36800 13068312 0 0 8 1012 389 2349 3 0 37 60
> 
> procs memory swap io system cpu
> 
> r b swpd free buff cache si so bi bo in cs us sy wa id
> 
> 1 1 29572 50960 36816 13068564 0 0 0 436 316 1650 2 0 26 72
> 
> 3 1 29572 50568 36816 13068728 0 0 0 420 371 4716 11 3 21 66
> 
> 2 1 29572 50076 36816 13068792 324 0 324 500 555 3715 4 1 29 66
> 
> 0 1 29572 49888 36820 13068924 40 0 68 308 533 1228 1 0 28 71
> 
> 0 1 29572 49556 36832 13069056 12 0 40 804 357 1362 2 0 25 73
> 
> 0 2 29572 49360 36832 13069080 0 0 0 432 318 1403 3 1 32 65
> 
> 0 1 29572 49748 36836 13069356 0 0 152 524 438 2118 3 0 33 64
> 
> 0 1 29572 49744 36840 13069680 0 0 24 436 398 1496 2 0 25 73
> 
> 1 0 29572 49988 36844 13069752 0 0 4 364 229 1515 4 0 9 87
> 
> 0 0 29572 49968 36848 13069752 0 0 0 376 226 3585 4 1 6 89
> 
> 0 0 29572 49912 36848 13069752 0 0 0 0 164 864 1 0 0 99
> 
> 5 1 29572 49300 36852 13069760 0 0 4 528 229 3247 9 1 1 89
> 
> 0 1 29572 34152 36852 13069756 0 0 0 10544 327 2350 13 3 18 66
> 
> 0 3 29812 18440 36572 13084376 0 240 4 33100 431 1986 7 5 47 41
> 
> 1 3 29812 18932 36572 13084376 0 0 0 128 460 1069 3 0 73 24
> 
> 1 4 29812 18712 36600 13084420 0 0 48 3136 445 4248 9 1 70 20
> 
> 1 1 29812 18208 36600 13084292 0 0 0 6008 324 4497 19 1 34 46
> 
> 2 1 29812 37176 36612 13084244 0 0 12 368 399 3913 9 1 22 68
> 
> 0 1 29812 36900 36616 13084428 0 0 4 432 281 1083 1 0 26 72
> 
> 0 1 29812 36236 36624 13084604 0 0 0 596 319 2382 4 2 23 71
> 
> 0 1 29812 36072 36636 13084824 0 0 8 596 303 1356 1 0 25 73
> 
> procs memory swap io system cpu
> 
> r b swpd free buff cache si so bi bo in cs us sy wa id
> 
> 2 1 29812 32944 36640 13085012 0 0 4 416 339 1501 2 1 25 72
> 
> 0 2 29812 35264 36644 13085548 0 0 4 1316 322 2321 6 3 21 69
> 
> 0 1 29812 35140 36660 13085628 0 0 0 532 334 3702 5 1 23 71
> 
> 1 3 29812 33596 36672 13085824 164 0 168 536 356 2754 5 4 26 65
> 
> 0 1 29284 36028 36672 13085768 40 0 40 316 434 3493 4 1 27 67
> 
> 2 1 29284 19900 36688 13085980 0 0 148 392 368 2328 9 3 23 65
> 
> 2 1 30268 33944 35428 13075744 0 984 0 1948 328 3763 17 13 19 50
> 
> 3 1 30268 33400 35436 13075816 0 0 8 372 322 2186 9 11 21 58
> 
> 0 1 30216 85016 35464 13075868 0 0 28 1264 406 3767 7 6 27 60
> 
> 0 1 30216 84904 35464 13075992 0 0 0 308 345 1619 2 0 24 74
> 
> 0 1 30216 81716 35468 13076208 0 0 24 540 459 2395 2 2 26 69
> 
> 1 1 30216 81520 35476 13076316 0 0 8 284 342 981 1 1 25 74
> 
> 0 1 30216 81588 35496 13076604 0 0 0 996 352 3908 9 2 32 57
> 
> 0 0 30216 80196 35504 13076664 0 0 0 716 343 2068 3 1 30 67
> 
> 0 0 30216 80636 35504 13076660 0 0 0 16 287 3022 4 0 1 95
> 
> 1 1 30216 80388 35512 13076660 0 0 0 360 488 2474 5 1 7 87
> 
> 0 1 30216 79908 35516 13077440 0 0 4 1168 420 3769 6 3 19 73
> 
> 0 3 30216 61900 35516 13077540 4 0 4 492 320 2544 6 2 26 66
> 
> 0 1 30216 79716 35528 13077604 0 0 0 680 361 2874 6 2 30 62
> 
> 13 2 30216 79132 35536 13077820 0 0 112 544 327 1852 3 23 20 54
> 
> 0 2 30216 77372 35584 13077912 0 0 44 816 396 4262 5 59 23 13
> 
> procs memory swap io system cpu
> 
> r b swpd free buff cache si so bi bo in cs us sy wa id
> 
> 0 1 30216 77716 35620 13077972 0 0 4 664 352 2108 2 2 32 64
> 
> 0 0 30216 63368 35628 13078016 0 0 0 528 327 2079 7 2 17 73
> 
> 0 0 30216 63404 35632 13078012 0 0 4 64 281 1894 3 0 2 95
> 
> 0 0 30216 63452 35640 13078032 0 0 32 16 300 1841 3 0 1 96
> 
> 1 0 30216 44608 35668 13078028 0 0 12 584 302 4047 10 3 5 82
> 
> 0 0 30216 62480 35668 13078040 0 0 0 48 290 3920 3 1 2 94
> 
> 0 0 30216 61668 35676 13078040 0 0 0 372 290 2415 6 1 7 86
> 
> 0 0 30216 61968 35776 13078020 0 0 0 496 400 1798 2 0 9 88
> 
> 0 2 30216 57168 35776 13078040 0 0 20 136 295 2283 6 1 7 85
> 
> 1 0 30216 60656 35780 13078056 0 0 16 256 314 2394 3 1 15 81
> 
> 1 0 30216 60632 35780 13078056 0 0 0 0 242 1930 2 0 0 98
> 
> 0 0 30216 60548 35788 13078056 0 0 0 404 279 1985 2 0 6 91
> 
> 0 0 30216 60572 35788 13078064 0 0 0 112 329 1766 2 1 4 93
> 
> 2 0 30216 60720 35788 13078068 0 0 0 152 357 2246 4 2 5 89
> 
> 1 0 30216 60260 35792 13078088 0 0 28 48 315 2023 4 2 2 92
> 
> 0 1 30216 56996 35792 13078088 0 0 0 76 324 2610 4 1 1 94
> 
> 1 0 30216 59244 35808 13078096 0 0 0 468 278 1862 4 6 5 85
> 
> 1 0 30216 59012 35808 13078096 0 0 0 0 260 2533 10 10 0 81
> 
> 1 0 30216 59580 35808 13078096 0 0 0 40 220 1471 3 4 1 92
> 
> 2 0 30216 37800 35828 13078092 0 0 4 468 275 2863 8 4 2 86
> 
> 2 0 30216 46232 35832 13078108 0 0 4 56 239 5979 10 3 2 85
> 
> procs memory swap io system cpu
> 
> r b swpd free buff cache si so bi bo in cs us sy wa id
> 
> 1 0 30216 48324 35844 13078104 0 0 0 700 357 5279 8 3 8 80
> 
> 0 0 30216 48256 35844 13078104 0 0 0 0 223 2348 3 0 0 97
> 
> 0 0 30216 48156 35844 13078104 0 0 0 0 210 2236 3 0 0 97
> 
> 0 1 30216 47916 35844 13078104 0 0 0 140 238 1922 4 1 8 87
> 
> 2 0 30216 49284 35844 13078108 0 0 0 228 248 2077 4 1 10 85
> 
> 1 0 30216 49768 35860 13078112 0 0 0 256 242 1882 3 1 3 93
> 
> 0 1 30216 49556 35860 13078112 0 0 0 4016 291 2696 9 1 8 83
> 
> 2 1 30216 49360 35860 13078112 0 0 0 5324 308 1990 4 0 14 81
> 
> 0 2 30216 48672 35880 13078528 0 0 64 1100 362 1945 4 1 29 66
> 
> 0 1 30216 48164 35880 13078532 0 0 0 464 356 3561 5 2 29 64
> 
> 1 1 30216 47460 35896 13079072 0 0 4 1828 333 2242 4 1 44 51
> 
> 0 2 30216 47452 35952 13079184 0 0 8 648 374 2155 3 0 33 63
> 
> 0 2 30216 46732 36112 13079280 0 0 0 976 431 1399 2 1 47 50
> 
> 2 1 30216 46608 36192 13079324 0 0 0 652 321 1581 3 0 36 61
> 
> 2 2 30216 36408 36208 13079436 0 0 0 2276 293 2590 8 2 31 60
> 
> 0 1 30216 31312 36216 13079488 0 0 0 8060 337 1943 7 2 44 48
> 
> 1 1 30216 30456 36216 13080112 0 0 0 1176 276 1833 2 0 27 71
> 
> 2 1 30216 25664 36224 13068396 0 0 8 1352 321 2481 11 3 26 60
> 
> 1 1 30216 25532 36224 13068432 0 0 0 292 349 1650 15 1 23 61
> 
> 1 2 30216 25152 36224 13068468 0 0 0 332 331 1165 14 0 25 61
> 
> 2 1 30216 24708 36248 13068548 0 0 24 824 379 2900 16 1 27 56
> 
> procs memory swap io system cpu
> 
> r b swpd free buff cache si so bi bo in cs us sy wa id
> 
> 1 1 30216 24712 36248 13068572 0 0 0 300 280 1723 15 1 26 58
> 
> 4 0 30592 18068 35888 13057860 0 376 80 596 229 2890 20 3 14 63
> 
> 3 0 31080 18000 35888 13040464 0 488 0 680 262 3876 35 4 6 55
> 
> 2 0 31460 37148 35888 13039164 0 380 0 508 397 4340 39 1 5 55
> 
> 3 0 31460 36100 35924 13039304 0 0 168 668 526 2443 29 1 7 62
> 
> 2 0 31460 34444 35960 13039436 0 0 144 32 288 3501 35 3 3 59
> 
> 3 0 31460 32988 35960 13039436 0 0 0 16 193 1055 31 8 1 60
> 
> 2 0 31460 50180 35960 13039440 0 0 0 48 237 4637 32 10 1 57
> 
> 2 0 31460 51284 35960 13039432 0 0 0 32 202 1460 16 1 0 83
> 
> 2 0 31460 34216 35972 13039432 0 0 0 520 248 3107 28 3 5 64
> 
> 2 0 31460 21784 35972 13032424 0 0 0 64 231 2443 31 3 2 63
> 
> 3 0 31460 59696 35980 13032432 0 0 0 576 319 4940 38 2 3 57
> 
> 1 0 31460 71548 35980 13032424 0 0 0 0 233 1365 15 1 0 84
> 
> 1 0 31460 68312 35980 13032424 0 0 0 80 226 1830 17 1 3 79
> 
> 1 0 31460 65288 35988 13032432 0 0 0 424 275 2024 16 3 4 76
> 
> 2 0 31460 65272 35988 13032432 0 0 0 8 212 638 14 1 0 84
> 
> 1 4 31460 62588 36084 13032400 0 0 84 8892 365 2025 26 2 29 43
> 
> 1 1 31460 64536 36124 13032384 0 0 24 6328 393 1624 17 1 33 48
> 
> 3 1 31460 64464 36132 13032484 0 0 8 492 322 1542 14 0 25 61
> 
> 1 1 31460 79504 36148 13032592 0 0 8 580 380 2713 10 2 20 68
> 
> 2 2 31460 66840 36176 13032636 0 0 12 4048 474 3713 15 4 29 52
> 
> 
> 
> 
> 
> .....