oom-killer executes

[Dustin Larmeir]

What does your current free memory look like on the system? It sounds 
like your simply resource starved - Dustin

On 07/12/2010 12:14 AM, Paolo Galtieri wrote:
> Over the last few days I have noticed that the out of memory killer task
> keeps getting executed on my RHEL 5.4 system
>
> Jul 11 18:31:00 truckin kernel: Mem-info:
> Jul 11 18:31:00 truckin kernel: Node 0 DMA per-cpu:
> Jul 11 18:31:00 truckin kernel: cpu 0 hot: high 0, batch 1 used:0
> Jul 11 18:31:00 truckin kernel: cpu 0 cold: high 0, batch 1 used:0
> Jul 11 18:31:00 truckin kernel: cpu 1 hot: high 0, batch 1 used:0
> Jul 11 18:31:00 truckin kernel: cpu 1 cold: high 0, batch 1 used:0
> Jul 11 18:31:00 truckin kernel: Node 0 DMA32 per-cpu:
> Jul 11 18:31:00 truckin kernel: cpu 0 hot: high 186, batch 31 used:180
> Jul 11 18:31:00 truckin kernel: cpu 0 cold: high 62, batch 15 used:38
> Jul 11 18:31:00 truckin kernel: cpu 1 hot: high 186, batch 31 used:168
> Jul 11 18:31:00 truckin kernel: cpu 1 cold: high 62, batch 15 used:15
> Jul 11 18:31:00 truckin kernel: Node 0 Normal per-cpu: empty
> Jul 11 18:31:00 truckin kernel: Node 0 HighMem per-cpu: empty
> Jul 11 18:31:00 truckin kernel: Free pages:        7652kB (0kB HighMem)
> Jul 11 18:31:00 truckin kernel: Active:133974 inactive:353502 dirty:2
> writeback:0 unstable:0 free:1913 slab:9969 mapped-file:538
> mapped-anon:487532 pagetables:7514
> Jul 11 18:31:00 truckin kernel: Node 0 DMA free:1976kB min:28kB low:32kB
> high:40kB active:0kB inactive:0kB present:10520kB pages_scanned:0
> all_unreclaimable? yes
> Jul 11 18:31:00 truckin kernel: lowmem_reserve[]: 0 2002 2002 2002
> Jul 11 18:31:00 truckin kernel: Node 0 DMA32 free:5676kB min:5708kB
> low:7132kB high:8560kB active:535896kB inactive:1414008kB present:2050848kB
> pages_scanned:5262216 all_unreclaimable? yes
> Jul 11 18:31:00 truckin kernel: lowmem_reserve[]: 0 0 0 0
> Jul 11 18:31:00 truckin kernel: Node 0 Normal free:0kB min:0kB low:0kB
> high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0
> all_unreclaimable? no
> Jul 11 18:31:00 truckin kernel: lowmem_reserve[]: 0 0 0 0
> Jul 11 18:31:00 truckin kernel: Node 0 HighMem free:0kB min:128kB low:128kB
> high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0
> all_unreclaimable? no
> Jul 11 18:31:00 truckin kernel: lowmem_reserve[]: 0 0 0 0
> Jul 11 18:31:00 truckin kernel: Node 0 DMA: 4*4kB 7*8kB 3*16kB 4*32kB 5*64kB
> 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1976kB
> Jul 11 18:31:00 truckin kernel: Node 0 DMA32: 1151*4kB 0*8kB 1*16kB 1*32kB
> 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 5676kB
> Jul 11 18:31:00 truckin kernel: Node 0 Normal: empty
> Jul 11 18:31:00 truckin kernel: Node 0 HighMem: empty
> Jul 11 18:31:00 truckin kernel: 1284 pagecache pages
> Jul 11 18:31:00 truckin kernel: Swap cache: add 1166071, delete 1165884,
> find 72770/87468, race 0+69
> Jul 11 18:31:00 truckin kernel: Free swap  = 0kB
> Jul 11 18:31:00 truckin kernel: Total swap = 2031608kB
> Jul 11 18:31:00 truckin kernel: Free swap:            0kB
> Jul 11 18:31:00 truckin kernel: 523914 pages of RAM
> Jul 11 18:31:00 truckin kernel: 9408 reserved pages
> Jul 11 18:31:00 truckin kernel: 15073 pages shared
> Jul 11 18:31:00 truckin kernel: 187 pages swap cached
> Jul 11 18:31:00 truckin kernel: Out of memory: Killed process 29111, UID
> 500, (gnome-panel).
> Jul 11 18:31:38 truckin gconfd (pgaltieri-29082): Resolved address
> "xml:readwrite:/home/pgaltieri/.gconf" to a writable configuration source at
> position 0
> Jul 11 18:39:14 truckin kernel: Xorg invoked oom-killer: gfp_mask=0xd0,
> order=0, oomkilladj=0
> Jul 11 18:39:14 truckin kernel:
> Jul 11 18:39:14 truckin kernel: Call Trace:
> Jul 11 18:39:14 truckin kernel:  [<ffffffff800c7161>]
> out_of_memory+0x8e/0x2f3
> Jul 11 18:39:14 truckin kernel:  [<ffffffff8000f4b1>]
> __alloc_pages+0x27f/0x308
> Jul 11 18:39:14 truckin kernel:  [<ffffffff8002e105>] __wake_up+0x38/0x4f
> Jul 11 18:39:14 truckin kernel:  [<ffffffff8003c4b1>]
> __get_free_pages+0xe/0x71
> Jul 11 18:39:14 truckin kernel:  [<ffffffff8001eb0b>] __pollwait+0x58/0xe2
> Jul 11 18:39:14 truckin kernel:  [<ffffffff80030236>] unix_poll+0x18/0x99
> Jul 11 18:39:14 truckin kernel:  [<ffffffff80011880>] do_select+0x287/0x462
> Jul 11 18:39:14 truckin kernel:  [<ffffffff8001eab3>] __pollwait+0x0/0xe2
> Jul 11 18:39:14 truckin kernel:  [<ffffffff8008cfa1>]
> default_wake_function+0x0/0xe
> Jul 11 18:39:28 truckin last message repeated 9 times
> Jul 11 18:39:28 truckin kernel:  [<ffffffff800e91de>]
> core_sys_select+0x1bc/0x265
> Jul 11 18:39:28 truckin kernel:  [<ffffffff8000ce26>]
> do_sync_read+0xc7/0x104
> Jul 11 18:39:28 truckin kernel:  [<ffffffff800e06f4>]
> do_readv_writev+0x26e/0x291
> Jul 11 18:39:28 truckin kernel:  [<ffffffff80043aad>]
> sys_rt_sigreturn+0x323/0x356
> Jul 11 18:39:28 truckin kernel:  [<ffffffff80016854>] sys_select+0xc1/0x17c
> Jul 11 18:39:28 truckin kernel:  [<ffffffff8005d28d>] tracesys+0xd5/0xe0
> Jul 11 18:39:28 truckin kernel:
>
> There are lots of similar backtraces in /var/log/messages.  My system:
>
> [root at truckin ~]# cat /proc/cpuinfo
> processor    : 0
> vendor_id    : GenuineIntel
> cpu family    : 15
> model        : 4
> model name    :               Intel(R) Pentium(R) D CPU 3.20GHz
> stepping    : 4
> cpu MHz        : 3192.161
> cache size    : 1024 KB
> fpu        : yes
> fpu_exception    : yes
> cpuid level    : 3
> wp        : yes
> flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
> pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall lm
> constant_tsc pni monitor ds_cpl est cid cx16 xtpr
> bogomips    : 6384.32
> clflush size    : 64
> cache_alignment    : 128
> address sizes    : 36 bits physical, 48 bits virtual
> power management:
>
> processor    : 1
> vendor_id    : GenuineIntel
> cpu family    : 15
> model        : 4
> model name    :               Intel(R) Pentium(R) D CPU 3.20GHz
> stepping    : 4
> cpu MHz        : 3192.161
> cache size    : 1024 KB
> physical id    : 0
> siblings    : 2
> core id        : 1
> cpu cores    : 2
> apicid        : 1
> fpu        : yes
> fpu_exception    : yes
> cpuid level    : 5
> wp        : yes
> flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
> pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall lm
> constant_tsc pni monitor ds_cpl est cid cx16 xtpr
> bogomips    : 6384.23
> clflush size    : 64
> cache_alignment    : 128
> address sizes    : 36 bits physical, 48 bits virtual
> power management:
>
> [root at truckin ~]# cat /proc/meminfo
> MemTotal:      2058024 kB
> MemFree:        546028 kB
> Buffers:         26840 kB
> Cached:        1322780 kB
> SwapCached:      20704 kB
> Active:         332964 kB
> Inactive:      1087384 kB
> HighTotal:           0 kB
> HighFree:            0 kB
> LowTotal:      2058024 kB
> LowFree:        546028 kB
> SwapTotal:     2031608 kB
> SwapFree:      1947144 kB
> Dirty:            3292 kB
> Writeback:           0 kB
> AnonPages:       64512 kB
> Mapped:          17488 kB
> Slab:            59896 kB
> PageTables:       9112 kB
> NFS_Unstable:        0 kB
> Bounce:              0 kB
> CommitLimit:   3060620 kB
> Committed_AS:   420348 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed:    264208 kB
> VmallocChunk: 34359473583 kB
> HugePages_Total:     0
> HugePages_Free:      0
> HugePages_Rsvd:      0
> Hugepagesize:     2048 kB
>
> During the time that I've noticed this I have been trying to download one of
> the iso images for RHEL6.0 Beta.  I've tried using firefox to download the
> image as well as wget.  I will start the download and come back sometime
> later and discover that I've been logged out.  I have also observed that
> running firefox 3.6.4 (firefox-3.6.4-8) will sometimes cause my X session to
> freeze up requiring me to login from another system and kill the X server in
> order to use it again.  This is a problem that I have only recently
> observed.
>
> Another thing I have noticed is that quite often while doing the download
> the destination file will stop getting updated.  The time stamp of the file
> will not change and the file size will not update even though when I run
> wireshark it shows the connection to ftp.redhat.com is still live and
> packets are being received and the sequence number in the packets are
> changing which implies data is arriving but not going to the file.
>
> I have been trying to download the RHEL 6 Beta iso images since last
> Thursday onto my RHEL 5.4 system and have not succeeded yet.
>
> Has anyone else seen this issue while trying to download a large file?
>
> Any help in trying to determine the cause and if possible a solution to the
> problem is greatly appreciated.
>
> Thanks,
> Paolo
>