oom-killer executes

Dustin Larmeir dustin at larmeir.com
Mon Jul 12 08:05:30 UTC 2010 

Nvm on my last comment( I am half asleep) . Are you running a firewall on
the system or an IDS/IPS?

On Mon, Jul 12, 2010 at 1:09 AM, Dustin Larmeir <dustin at larmeir.com> wrote:

> What does your current free memory look like on the system? It sounds like
> your simply resource starved - Dustin
>
>
> On 07/12/2010 12:14 AM, Paolo Galtieri wrote:
>
>> Over the last few days I have noticed that the out of memory killer task
>> keeps getting executed on my RHEL 5.4 system
>>
>> Jul 11 18:31:00 truckin kernel: Mem-info:
>> Jul 11 18:31:00 truckin kernel: Node 0 DMA per-cpu:
>> Jul 11 18:31:00 truckin kernel: cpu 0 hot: high 0, batch 1 used:0
>> Jul 11 18:31:00 truckin kernel: cpu 0 cold: high 0, batch 1 used:0
>> Jul 11 18:31:00 truckin kernel: cpu 1 hot: high 0, batch 1 used:0
>> Jul 11 18:31:00 truckin kernel: cpu 1 cold: high 0, batch 1 used:0
>> Jul 11 18:31:00 truckin kernel: Node 0 DMA32 per-cpu:
>> Jul 11 18:31:00 truckin kernel: cpu 0 hot: high 186, batch 31 used:180
>> Jul 11 18:31:00 truckin kernel: cpu 0 cold: high 62, batch 15 used:38
>> Jul 11 18:31:00 truckin kernel: cpu 1 hot: high 186, batch 31 used:168
>> Jul 11 18:31:00 truckin kernel: cpu 1 cold: high 62, batch 15 used:15
>> Jul 11 18:31:00 truckin kernel: Node 0 Normal per-cpu: empty
>> Jul 11 18:31:00 truckin kernel: Node 0 HighMem per-cpu: empty
>> Jul 11 18:31:00 truckin kernel: Free pages:        7652kB (0kB HighMem)
>> Jul 11 18:31:00 truckin kernel: Active:133974 inactive:353502 dirty:2
>> writeback:0 unstable:0 free:1913 slab:9969 mapped-file:538
>> mapped-anon:487532 pagetables:7514
>> Jul 11 18:31:00 truckin kernel: Node 0 DMA free:1976kB min:28kB low:32kB
>> high:40kB active:0kB inactive:0kB present:10520kB pages_scanned:0
>> all_unreclaimable? yes
>> Jul 11 18:31:00 truckin kernel: lowmem_reserve[]: 0 2002 2002 2002
>> Jul 11 18:31:00 truckin kernel: Node 0 DMA32 free:5676kB min:5708kB
>> low:7132kB high:8560kB active:535896kB inactive:1414008kB
>> present:2050848kB
>> pages_scanned:5262216 all_unreclaimable? yes
>> Jul 11 18:31:00 truckin kernel: lowmem_reserve[]: 0 0 0 0
>> Jul 11 18:31:00 truckin kernel: Node 0 Normal free:0kB min:0kB low:0kB
>> high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0
>> all_unreclaimable? no
>> Jul 11 18:31:00 truckin kernel: lowmem_reserve[]: 0 0 0 0
>> Jul 11 18:31:00 truckin kernel: Node 0 HighMem free:0kB min:128kB
>> low:128kB
>> high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0
>> all_unreclaimable? no
>> Jul 11 18:31:00 truckin kernel: lowmem_reserve[]: 0 0 0 0
>> Jul 11 18:31:00 truckin kernel: Node 0 DMA: 4*4kB 7*8kB 3*16kB 4*32kB
>> 5*64kB
>> 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1976kB
>> Jul 11 18:31:00 truckin kernel: Node 0 DMA32: 1151*4kB 0*8kB 1*16kB 1*32kB
>> 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 5676kB
>> Jul 11 18:31:00 truckin kernel: Node 0 Normal: empty
>> Jul 11 18:31:00 truckin kernel: Node 0 HighMem: empty
>> Jul 11 18:31:00 truckin kernel: 1284 pagecache pages
>> Jul 11 18:31:00 truckin kernel: Swap cache: add 1166071, delete 1165884,
>> find 72770/87468, race 0+69
>> Jul 11 18:31:00 truckin kernel: Free swap  = 0kB
>> Jul 11 18:31:00 truckin kernel: Total swap = 2031608kB
>> Jul 11 18:31:00 truckin kernel: Free swap:            0kB
>> Jul 11 18:31:00 truckin kernel: 523914 pages of RAM
>> Jul 11 18:31:00 truckin kernel: 9408 reserved pages
>> Jul 11 18:31:00 truckin kernel: 15073 pages shared
>> Jul 11 18:31:00 truckin kernel: 187 pages swap cached
>> Jul 11 18:31:00 truckin kernel: Out of memory: Killed process 29111, UID
>> 500, (gnome-panel).
>> Jul 11 18:31:38 truckin gconfd (pgaltieri-29082): Resolved address
>> "xml:readwrite:/home/pgaltieri/.gconf" to a writable configuration source
>> at
>> position 0
>> Jul 11 18:39:14 truckin kernel: Xorg invoked oom-killer: gfp_mask=0xd0,
>> order=0, oomkilladj=0
>> Jul 11 18:39:14 truckin kernel:
>> Jul 11 18:39:14 truckin kernel: Call Trace:
>> Jul 11 18:39:14 truckin kernel:  [<ffffffff800c7161>]
>> out_of_memory+0x8e/0x2f3
>> Jul 11 18:39:14 truckin kernel:  [<ffffffff8000f4b1>]
>> __alloc_pages+0x27f/0x308
>> Jul 11 18:39:14 truckin kernel:  [<ffffffff8002e105>] __wake_up+0x38/0x4f
>> Jul 11 18:39:14 truckin kernel:  [<ffffffff8003c4b1>]
>> __get_free_pages+0xe/0x71
>> Jul 11 18:39:14 truckin kernel:  [<ffffffff8001eb0b>] __pollwait+0x58/0xe2
>> Jul 11 18:39:14 truckin kernel:  [<ffffffff80030236>] unix_poll+0x18/0x99
>> Jul 11 18:39:14 truckin kernel:  [<ffffffff80011880>]
>> do_select+0x287/0x462
>> Jul 11 18:39:14 truckin kernel:  [<ffffffff8001eab3>] __pollwait+0x0/0xe2
>> Jul 11 18:39:14 truckin kernel:  [<ffffffff8008cfa1>]
>> default_wake_function+0x0/0xe
>> Jul 11 18:39:28 truckin last message repeated 9 times
>> Jul 11 18:39:28 truckin kernel:  [<ffffffff800e91de>]
>> core_sys_select+0x1bc/0x265
>> Jul 11 18:39:28 truckin kernel:  [<ffffffff8000ce26>]
>> do_sync_read+0xc7/0x104
>> Jul 11 18:39:28 truckin kernel:  [<ffffffff800e06f4>]
>> do_readv_writev+0x26e/0x291
>> Jul 11 18:39:28 truckin kernel:  [<ffffffff80043aad>]
>> sys_rt_sigreturn+0x323/0x356
>> Jul 11 18:39:28 truckin kernel:  [<ffffffff80016854>]
>> sys_select+0xc1/0x17c
>> Jul 11 18:39:28 truckin kernel:  [<ffffffff8005d28d>] tracesys+0xd5/0xe0
>> Jul 11 18:39:28 truckin kernel:
>>
>> There are lots of similar backtraces in /var/log/messages.  My system:
>>
>> [root at truckin ~]# cat /proc/cpuinfo
>> processor    : 0
>> vendor_id    : GenuineIntel
>> cpu family    : 15
>> model        : 4
>> model name    :               Intel(R) Pentium(R) D CPU 3.20GHz
>> stepping    : 4
>> cpu MHz        : 3192.161
>> cache size    : 1024 KB
>> fpu        : yes
>> fpu_exception    : yes
>> cpuid level    : 3
>> wp        : yes
>> flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
>> cmov
>> pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall lm
>> constant_tsc pni monitor ds_cpl est cid cx16 xtpr
>> bogomips    : 6384.32
>> clflush size    : 64
>> cache_alignment    : 128
>> address sizes    : 36 bits physical, 48 bits virtual
>> power management:
>>
>> processor    : 1
>> vendor_id    : GenuineIntel
>> cpu family    : 15
>> model        : 4
>> model name    :               Intel(R) Pentium(R) D CPU 3.20GHz
>> stepping    : 4
>> cpu MHz        : 3192.161
>> cache size    : 1024 KB
>> physical id    : 0
>> siblings    : 2
>> core id        : 1
>> cpu cores    : 2
>> apicid        : 1
>> fpu        : yes
>> fpu_exception    : yes
>> cpuid level    : 5
>> wp        : yes
>> flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
>> cmov
>> pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall lm
>> constant_tsc pni monitor ds_cpl est cid cx16 xtpr
>> bogomips    : 6384.23
>> clflush size    : 64
>> cache_alignment    : 128
>> address sizes    : 36 bits physical, 48 bits virtual
>> power management:
>>
>> [root at truckin ~]# cat /proc/meminfo
>> MemTotal:      2058024 kB
>> MemFree:        546028 kB
>> Buffers:         26840 kB
>> Cached:        1322780 kB
>> SwapCached:      20704 kB
>> Active:         332964 kB
>> Inactive:      1087384 kB
>> HighTotal:           0 kB
>> HighFree:            0 kB
>> LowTotal:      2058024 kB
>> LowFree:        546028 kB
>> SwapTotal:     2031608 kB
>> SwapFree:      1947144 kB
>> Dirty:            3292 kB
>> Writeback:           0 kB
>> AnonPages:       64512 kB
>> Mapped:          17488 kB
>> Slab:            59896 kB
>> PageTables:       9112 kB
>> NFS_Unstable:        0 kB
>> Bounce:              0 kB
>> CommitLimit:   3060620 kB
>> Committed_AS:   420348 kB
>> VmallocTotal: 34359738367 kB
>> VmallocUsed:    264208 kB
>> VmallocChunk: 34359473583 kB
>> HugePages_Total:     0
>> HugePages_Free:      0
>> HugePages_Rsvd:      0
>> Hugepagesize:     2048 kB
>>
>> During the time that I've noticed this I have been trying to download one
>> of
>> the iso images for RHEL6.0 Beta.  I've tried using firefox to download the
>> image as well as wget.  I will start the download and come back sometime
>> later and discover that I've been logged out.  I have also observed that
>> running firefox 3.6.4 (firefox-3.6.4-8) will sometimes cause my X session
>> to
>> freeze up requiring me to login from another system and kill the X server
>> in
>> order to use it again.  This is a problem that I have only recently
>> observed.
>>
>> Another thing I have noticed is that quite often while doing the download
>> the destination file will stop getting updated.  The time stamp of the
>> file
>> will not change and the file size will not update even though when I run
>> wireshark it shows the connection to ftp.redhat.com is still live and
>> packets are being received and the sequence number in the packets are
>> changing which implies data is arriving but not going to the file.
>>
>> I have been trying to download the RHEL 6 Beta iso images since last
>> Thursday onto my RHEL 5.4 system and have not succeeded yet.
>>
>> Has anyone else seen this issue while trying to download a large file?
>>
>> Any help in trying to determine the cause and if possible a solution to
>> the
>> problem is greatly appreciated.
>>
>> Thanks,
>> Paolo
>>
>>
>
>

More information about the redhat-list mailing list