oom-killer executes

Mon Jul 12 16:02:05 UTC 2010

On 07/12/2010 01:05 AM, Dustin Larmeir wrote:
> Nvm on my last comment( I am half asleep) . Are you running a firewall on
> the system or an IDS/IPS?
>
> On Mon, Jul 12, 2010 at 1:09 AM, Dustin Larmeir<dustin at larmeir.com>  wrote:
>
>    
>> What does your current free memory look like on the system? It sounds like
>> your simply resource starved - Dustin
>>
>>
>> On 07/12/2010 12:14 AM, Paolo Galtieri wrote:
>>
>>      
>>> Over the last few days I have noticed that the out of memory killer task
>>> keeps getting executed on my RHEL 5.4 system
>>>
>>> Jul 11 18:31:00 truckin kernel: Mem-info:
>>> Jul 11 18:31:00 truckin kernel: Node 0 DMA per-cpu:
>>> Jul 11 18:31:00 truckin kernel: cpu 0 hot: high 0, batch 1 used:0
>>> Jul 11 18:31:00 truckin kernel: cpu 0 cold: high 0, batch 1 used:0
>>> Jul 11 18:31:00 truckin kernel: cpu 1 hot: high 0, batch 1 used:0
>>> Jul 11 18:31:00 truckin kernel: cpu 1 cold: high 0, batch 1 used:0
>>> Jul 11 18:31:00 truckin kernel: Node 0 DMA32 per-cpu:
>>> Jul 11 18:31:00 truckin kernel: cpu 0 hot: high 186, batch 31 used:180
>>> Jul 11 18:31:00 truckin kernel: cpu 0 cold: high 62, batch 15 used:38
>>> Jul 11 18:31:00 truckin kernel: cpu 1 hot: high 186, batch 31 used:168
>>> Jul 11 18:31:00 truckin kernel: cpu 1 cold: high 62, batch 15 used:15
>>> Jul 11 18:31:00 truckin kernel: Node 0 Normal per-cpu: empty
>>> Jul 11 18:31:00 truckin kernel: Node 0 HighMem per-cpu: empty
>>> Jul 11 18:31:00 truckin kernel: Free pages:        7652kB (0kB HighMem)
>>> Jul 11 18:31:00 truckin kernel: Active:133974 inactive:353502 dirty:2
>>> writeback:0 unstable:0 free:1913 slab:9969 mapped-file:538
>>> mapped-anon:487532 pagetables:7514
>>> Jul 11 18:31:00 truckin kernel: Node 0 DMA free:1976kB min:28kB low:32kB
>>> high:40kB active:0kB inactive:0kB present:10520kB pages_scanned:0
>>> all_unreclaimable? yes
>>> Jul 11 18:31:00 truckin kernel: lowmem_reserve[]: 0 2002 2002 2002
>>> Jul 11 18:31:00 truckin kernel: Node 0 DMA32 free:5676kB min:5708kB
>>> low:7132kB high:8560kB active:535896kB inactive:1414008kB
>>> present:2050848kB
>>> pages_scanned:5262216 all_unreclaimable? yes
>>> Jul 11 18:31:00 truckin kernel: lowmem_reserve[]: 0 0 0 0
>>> Jul 11 18:31:00 truckin kernel: Node 0 Normal free:0kB min:0kB low:0kB
>>> high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0
>>> all_unreclaimable? no
>>> Jul 11 18:31:00 truckin kernel: lowmem_reserve[]: 0 0 0 0
>>> Jul 11 18:31:00 truckin kernel: Node 0 HighMem free:0kB min:128kB
>>> low:128kB
>>> high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0
>>> all_unreclaimable? no
>>> Jul 11 18:31:00 truckin kernel: lowmem_reserve[]: 0 0 0 0
>>> Jul 11 18:31:00 truckin kernel: Node 0 DMA: 4*4kB 7*8kB 3*16kB 4*32kB
>>> 5*64kB
>>> 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1976kB
>>> Jul 11 18:31:00 truckin kernel: Node 0 DMA32: 1151*4kB 0*8kB 1*16kB 1*32kB
>>> 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 5676kB
>>> Jul 11 18:31:00 truckin kernel: Node 0 Normal: empty
>>> Jul 11 18:31:00 truckin kernel: Node 0 HighMem: empty
>>> Jul 11 18:31:00 truckin kernel: 1284 pagecache pages
>>> Jul 11 18:31:00 truckin kernel: Swap cache: add 1166071, delete 1165884,
>>> find 72770/87468, race 0+69
>>> Jul 11 18:31:00 truckin kernel: Free swap  = 0kB
>>> Jul 11 18:31:00 truckin kernel: Total swap = 2031608kB
>>> Jul 11 18:31:00 truckin kernel: Free swap:            0kB
>>> Jul 11 18:31:00 truckin kernel: 523914 pages of RAM
>>> Jul 11 18:31:00 truckin kernel: 9408 reserved pages
>>> Jul 11 18:31:00 truckin kernel: 15073 pages shared
>>> Jul 11 18:31:00 truckin kernel: 187 pages swap cached
>>> Jul 11 18:31:00 truckin kernel: Out of memory: Killed process 29111, UID
>>> 500, (gnome-panel).
>>> Jul 11 18:31:38 truckin gconfd (pgaltieri-29082): Resolved address
>>> "xml:readwrite:/home/pgaltieri/.gconf" to a writable configuration source
>>> at
>>> position 0
>>> Jul 11 18:39:14 truckin kernel: Xorg invoked oom-killer: gfp_mask=0xd0,
>>> order=0, oomkilladj=0
>>> Jul 11 18:39:14 truckin kernel:
>>> Jul 11 18:39:14 truckin kernel: Call Trace:
>>> Jul 11 18:39:14 truckin kernel:  [<ffffffff800c7161>]
>>> out_of_memory+0x8e/0x2f3
>>> Jul 11 18:39:14 truckin kernel:  [<ffffffff8000f4b1>]
>>> __alloc_pages+0x27f/0x308
>>> Jul 11 18:39:14 truckin kernel:  [<ffffffff8002e105>] __wake_up+0x38/0x4f
>>> Jul 11 18:39:14 truckin kernel:  [<ffffffff8003c4b1>]
>>> __get_free_pages+0xe/0x71
>>> Jul 11 18:39:14 truckin kernel:  [<ffffffff8001eb0b>] __pollwait+0x58/0xe2
>>> Jul 11 18:39:14 truckin kernel:  [<ffffffff80030236>] unix_poll+0x18/0x99
>>> Jul 11 18:39:14 truckin kernel:  [<ffffffff80011880>]
>>> do_select+0x287/0x462
>>> Jul 11 18:39:14 truckin kernel:  [<ffffffff8001eab3>] __pollwait+0x0/0xe2
>>> Jul 11 18:39:14 truckin kernel:  [<ffffffff8008cfa1>]
>>> default_wake_function+0x0/0xe
>>> Jul 11 18:39:28 truckin last message repeated 9 times
>>> Jul 11 18:39:28 truckin kernel:  [<ffffffff800e91de>]
>>> core_sys_select+0x1bc/0x265
>>> Jul 11 18:39:28 truckin kernel:  [<ffffffff8000ce26>]
>>> do_sync_read+0xc7/0x104
>>> Jul 11 18:39:28 truckin kernel:  [<ffffffff800e06f4>]
>>> do_readv_writev+0x26e/0x291
>>> Jul 11 18:39:28 truckin kernel:  [<ffffffff80043aad>]
>>> sys_rt_sigreturn+0x323/0x356
>>> Jul 11 18:39:28 truckin kernel:  [<ffffffff80016854>]
>>> sys_select+0xc1/0x17c
>>> Jul 11 18:39:28 truckin kernel:  [<ffffffff8005d28d>] tracesys+0xd5/0xe0
>>> Jul 11 18:39:28 truckin kernel:
>>>
>>> There are lots of similar backtraces in /var/log/messages.  My system:
>>>
>>> [root at truckin ~]# cat /proc/cpuinfo
>>> processor    : 0
>>> vendor_id    : GenuineIntel
>>> cpu family    : 15
>>> model        : 4
>>> model name    :               Intel(R) Pentium(R) D CPU 3.20GHz
>>> stepping    : 4
>>> cpu MHz        : 3192.161
>>> cache size    : 1024 KB
>>> fpu        : yes
>>> fpu_exception    : yes
>>> cpuid level    : 3
>>> wp        : yes
>>> flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
>>> cmov
>>> pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall lm
>>> constant_tsc pni monitor ds_cpl est cid cx16 xtpr
>>> bogomips    : 6384.32
>>> clflush size    : 64
>>> cache_alignment    : 128
>>> address sizes    : 36 bits physical, 48 bits virtual
>>> power management:
>>>
>>> processor    : 1
>>> vendor_id    : GenuineIntel
>>> cpu family    : 15
>>> model        : 4
>>> model name    :               Intel(R) Pentium(R) D CPU 3.20GHz
>>> stepping    : 4
>>> cpu MHz        : 3192.161
>>> cache size    : 1024 KB
>>> physical id    : 0
>>> siblings    : 2
>>> core id        : 1
>>> cpu cores    : 2
>>> apicid        : 1
>>> fpu        : yes
>>> fpu_exception    : yes
>>> cpuid level    : 5
>>> wp        : yes
>>> flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
>>> cmov
>>> pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall lm
>>> constant_tsc pni monitor ds_cpl est cid cx16 xtpr
>>> bogomips    : 6384.23
>>> clflush size    : 64
>>> cache_alignment    : 128
>>> address sizes    : 36 bits physical, 48 bits virtual
>>> power management:
>>>
>>> [root at truckin ~]# cat /proc/meminfo
>>> MemTotal:      2058024 kB
>>> MemFree:        546028 kB
>>> Buffers:         26840 kB
>>> Cached:        1322780 kB
>>> SwapCached:      20704 kB
>>> Active:         332964 kB
>>> Inactive:      1087384 kB
>>> HighTotal:           0 kB
>>> HighFree:            0 kB
>>> LowTotal:      2058024 kB
>>> LowFree:        546028 kB
>>> SwapTotal:     2031608 kB
>>> SwapFree:      1947144 kB
>>> Dirty:            3292 kB
>>> Writeback:           0 kB
>>> AnonPages:       64512 kB
>>> Mapped:          17488 kB
>>> Slab:            59896 kB
>>> PageTables:       9112 kB
>>> NFS_Unstable:        0 kB
>>> Bounce:              0 kB
>>> CommitLimit:   3060620 kB
>>> Committed_AS:   420348 kB
>>> VmallocTotal: 34359738367 kB
>>> VmallocUsed:    264208 kB
>>> VmallocChunk: 34359473583 kB
>>> HugePages_Total:     0
>>> HugePages_Free:      0
>>> HugePages_Rsvd:      0
>>> Hugepagesize:     2048 kB
>>>
>>> During the time that I've noticed this I have been trying to download one
>>> of
>>> the iso images for RHEL6.0 Beta.  I've tried using firefox to download the
>>> image as well as wget.  I will start the download and come back sometime
>>> later and discover that I've been logged out.  I have also observed that
>>> running firefox 3.6.4 (firefox-3.6.4-8) will sometimes cause my X session
>>> to
>>> freeze up requiring me to login from another system and kill the X server
>>> in
>>> order to use it again.  This is a problem that I have only recently
>>> observed.
>>>
>>> Another thing I have noticed is that quite often while doing the download
>>> the destination file will stop getting updated.  The time stamp of the
>>> file
>>> will not change and the file size will not update even though when I run
>>> wireshark it shows the connection to ftp.redhat.com is still live and
>>> packets are being received and the sequence number in the packets are
>>> changing which implies data is arriving but not going to the file.
>>>
>>> I have been trying to download the RHEL 6 Beta iso images since last
>>> Thursday onto my RHEL 5.4 system and have not succeeded yet.
>>>
>>> Has anyone else seen this issue while trying to download a large file?
>>>
>>> Any help in trying to determine the cause and if possible a solution to
>>> the
>>> problem is greatly appreciated.
>>>
>>> Thanks,
>>> Paolo
>>>
>>>
>>>        
>>
>>      
I'm not running any IDS/IPS.  This system normally runs idle so I don't 
see how I can be running out of resources so quickly.  It started 
showing up when I tried to download the RHEL6 Beta iso images to this 
system.

Paolo