[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Bug in kernel 2.6.5-1.358smp



Hello,

I think I found a bug in the kernel.  I had 2 consecutive block ups. 
I'm running the system in a production environtment with a web
application in my company with lots of users.

I first installed FC2T3 to make some test, then I updated to FC2 Final
with yum and then applied the security updates available in:
[http-download.fedora.redhat.com-pub-fedora-linux-core-updates-2-i386-]
name=http-download.fedora.redhat.com-pub-fedora-linux-core-updates-2-i386-
baseurl=http://download.fedora.redhat.com/pub/fedora/linux/core/updates/2/i386/

I use Apache 2 (the one that comes with FC2), PHP 4.3.6 (compiled by
myself), TUX and Oracle 9i Client.

The portion of /var/log/messages is as follows:

Jun  2 08:48:52 appweb2-coomeva kernel: ------------[ cut here
]------------
Jun  2 08:48:52 appweb2-coomeva kernel: kernel BUG at
mm/page_alloc.c:786!
Jun  2 08:48:52 appweb2-coomeva kernel: invalid operand: 0000 [#1]
Jun  2 08:48:52 appweb2-coomeva kernel: SMP 
Jun  2 08:48:52 appweb2-coomeva kernel: CPU:    2
Jun  2 08:48:52 appweb2-coomeva kernel: EIP:    0060:[<02138aec>]    Not
tainted
Jun  2 08:48:52 appweb2-coomeva kernel: EFLAGS: 00010246  
(2.6.5-1.358smp) 
Jun  2 08:48:52 appweb2-coomeva kernel: EIP is at __free_pages+0x21/0x4c
Jun  2 08:48:52 appweb2-coomeva kernel: eax: 00000000   ebx: 00000000  
ecx: 03e128a0   edx: 03e128a0
Jun  2 08:48:52 appweb2-coomeva kernel: esi: 00000000   edi: 7c7c81f0  
ebp: 7c7c802c   esp: 7d09befc
Jun  2 08:48:52 appweb2-coomeva kernel: ds: 007b   es: 007b   ss: 0068
Jun  2 08:48:52 appweb2-coomeva kernel: Process tux (pid: 1526,
threadinfo=7d09b000 task=7dbbb410)
Jun  2 08:48:52 appweb2-coomeva kernel: Stack: 00000000 82947f00
00000000 7c7c802c 00000000 82953380 00000022 8293aff6 
Jun  2 08:48:52 appweb2-coomeva kernel:        7c7c802c 7c7c8054
8293bcd9 7d09bf40 82953380 00000000 00000005 fef73690 
Jun  2 08:48:52 appweb2-coomeva kernel:        829457fd 00000000
7d09b000 00000000 82947345 0212a0f0 82953380 000041ed 
Jun  2 08:48:52 appweb2-coomeva kernel: Call Trace:
Jun  2 08:48:53 appweb2-coomeva kernel:  [<82947f00>]
do_send_abuf+0x12c/0x147 [tux]
Jun  2 08:48:53 appweb2-coomeva kernel:  [<8293aff6>]
tux_schedule_atom+0x23/0x34 [tux]
Jun  2 08:48:53 appweb2-coomeva kernel:  [<8293bcd9>]
process_requests+0x90/0xa7 [tux]
Jun  2 08:48:53 appweb2-coomeva kernel:  [<829457fd>]
event_loop+0x75/0x178 [tux]
Jun  2 08:48:53 appweb2-coomeva kernel:  [<82947345>]
__sys_tux+0x343/0x77a [tux]
Jun  2 08:48:53 appweb2-coomeva kernel:  [<0212a0f0>]
in_group_p+0x31/0x58
Jun  2 08:48:53 appweb2-coomeva kernel:  [<021616a7>] dput+0x18/0x16c
Jun  2 08:48:53 appweb2-coomeva kernel:  [<02159672>]
path_release+0xa/0x26
Jun  2 08:48:54 appweb2-coomeva kernel:  [<0214da62>]
sys_chdir+0x57/0x5f
Jun  2 08:48:54 appweb2-coomeva kernel:  [<0224b124>] sys_tux+0x25/0xb2
Jun  2 08:48:54 appweb2-coomeva kernel: 
Jun  2 08:48:54 appweb2-coomeva kernel: Code: 0f 0b 12 03 c4 80 2b 02 f0
ff 49 04 0f 94 c0 84 c0 74 16 85 
Jun  2 08:48:54 appweb2-coomeva kernel:  <5>TUX: thread 0 stopping ...
Jun  2 08:50:07 appweb2-coomeva sshd(pam_unix)[1937]: session opened for
user nalwalovaton by (uid=500)
Jun  2 08:50:11 appweb2-coomeva su(pam_unix)[1965]: session opened for
user root by nalwalovaton(uid=500)
Jun  2 08:55:51 appweb2-coomeva kernel: Bad page state at
free_hot_cold_page (in process 'async IO 1/6', page 03dfa860)
Jun  2 08:55:51 appweb2-coomeva kernel: flags:0x20000000
mapping:00000000 mapped:0 count:0
Jun  2 08:55:51 appweb2-coomeva kernel: Backtrace:
Jun  2 08:55:51 appweb2-coomeva kernel: Call Trace:
Jun  2 08:55:51 appweb2-coomeva kernel:  [<02137f0d>] bad_page+0x5c/0x7f
Jun  2 08:55:51 appweb2-coomeva kernel:  [<02138593>]
free_hot_cold_page+0x67/0xe0
Jun  2 08:55:51 appweb2-coomeva kernel:  [<82947d46>]
get_abuf+0x9a/0x101 [tux]
Jun  2 08:55:51 appweb2-coomeva kernel:  [<82941008>]
http_pre_header+0x32/0x3ec [tux]
Jun  2 08:55:51 appweb2-coomeva kernel:  [<82940ebf>]
http_process_message+0x23a/0x266 [tux]
Jun  2 08:55:51 appweb2-coomeva kernel:  [<8293aff6>]
tux_schedule_atom+0x23/0x34 [tux]
Jun  2 08:55:51 appweb2-coomeva kernel:  [<8293c039>]
cachemiss_thread+0x128/0x1de [tux]
Jun  2 08:55:51 appweb2-coomeva kernel:  [<0211b419>]
default_wake_function+0x0/0xc
Jun  2 08:55:51 appweb2-coomeva kernel:  [<0211b419>]
default_wake_function+0x0/0xc
Jun  2 08:55:51 appweb2-coomeva kernel:  [<8293bf11>]
cachemiss_thread+0x0/0x1de [tux]
Jun  2 08:55:52 appweb2-coomeva kernel:  [<021041f1>]
kernel_thread_helper+0x5/0xb
Jun  2 08:55:52 appweb2-coomeva kernel: 
Jun  2 08:55:52 appweb2-coomeva kernel: Trying to fix it up, but a
reboot is needed
Jun  2 08:55:52 appweb2-coomeva kernel: Unable to handle kernel paging
request at virtual address 00100104
Jun  2 08:55:52 appweb2-coomeva kernel:  printing eip:
Jun  2 08:55:52 appweb2-coomeva kernel: 0213869f
Jun  2 08:55:52 appweb2-coomeva kernel: *pde = 00003001
Jun  2 08:55:52 appweb2-coomeva kernel: Oops: 0002 [#2]
Jun  2 08:55:53 appweb2-coomeva kernel: SMP 
Jun  2 08:55:53 appweb2-coomeva kernel: CPU:    0
Jun  2 08:55:53 appweb2-coomeva kernel: EIP:    0060:[<0213869f>]    Not
tainted
Jun  2 08:55:53 appweb2-coomeva kernel: EFLAGS: 00010002  
(2.6.5-1.358smp) 
Jun  2 08:55:53 appweb2-coomeva kernel: EIP is at
buffered_rmqueue+0x82/0x1a7
Jun  2 08:55:53 appweb2-coomeva kernel: eax: 03dfa878   ebx: 022f3780  
ecx: 00200200   edx: 00100100
Jun  2 08:55:53 appweb2-coomeva kernel: esi: 022f3b00   edi: 022f3780  
ebp: 00000246   esp: 7ac81e30
Jun  2 08:55:54 appweb2-coomeva kernel: ds: 007b   es: 007b   ss: 0068
Jun  2 08:55:54 appweb2-coomeva kernel: Process httpd (pid: 2060,
threadinfo=7ac81000 task=71b557b0)
Jun  2 08:55:54 appweb2-coomeva kernel: Stack: 03dfa860 000000d2
00000000 022f3780 00000000 022f5fa0 000000d2 02138861 
Jun  2 08:55:55 appweb2-coomeva kernel:        00000001 71b557b0
00000010 00000000 00000000 0038a025 00000000 7697c460 
Jun  2 08:55:55 appweb2-coomeva kernel:        0988c574 02141c86
00000000 00000000 732e56c4 7b709080 00000000 7697c460 
Jun  2 08:55:56 appweb2-coomeva kernel: Call Trace:
Jun  2 08:55:56 appweb2-coomeva kernel:  [<02138861>]
__alloc_pages+0x9d/0x284
Jun  2 08:55:56 appweb2-coomeva kernel:  [<02141c86>]
do_anonymous_page+0x6d/0x192
Jun  2 08:55:57 appweb2-coomeva kernel:  [<02141e14>]
do_no_page+0x69/0x2b0
Jun  2 08:55:58 appweb2-coomeva kernel:  [<021421ca>]
handle_mm_fault+0xc0/0x16e
Jun  2 08:55:58 appweb2-coomeva kernel:  [<02118c2b>]
do_page_fault+0x133/0x4b4
Jun  2 08:55:58 appweb2-coomeva kernel:  [<8289eafd>]
ext3_permission+0x0/0x153 [ext3]
Jun  2 08:55:59 appweb2-coomeva kernel:  [<021595ff>]
permission+0x41/0x46
Jun  2 08:56:00 appweb2-coomeva kernel:  [<02143043>]
vma_merge+0x85/0x237
Jun  2 08:56:00 appweb2-coomeva kernel:  [<02144121>] do_brk+0x127/0x1d5
Jun  2 08:56:00 appweb2-coomeva kernel:  [<02118af8>]
do_page_fault+0x0/0x4b4
Jun  2 08:56:00 appweb2-coomeva kernel: 
Jun  2 08:56:00 appweb2-coomeva kernel: Code: 89 4a 04 89 11 c7 40 04 00
02 20 00 c7 00 00 01 10 00 ff 8b 
Jun  2 09:00:02 appweb2-coomeva syslogd 1.4.1: restart.
Jun  2 09:00:02 appweb2-coomeva syslog: Iniciación de syslogd succeeded
Jun  2 09:00:03 appweb2-coomeva syslog: Iniciación de klogd succeeded
Jun  2 09:00:03 appweb2-coomeva kernel: klogd 1.4.1, log source =
/proc/kmsg started.
Jun  2 09:00:03 appweb2-coomeva kernel: Linux version 2.6.5-1.358smp
(bhcompile bugs build redhat com) (gcc version 3.3.3 20040412 (Red Hat
Linux 3.3.3-7)) #1 SMP Sat May 8 09:25:36 EDT 2004




Previously, I had another Lock up and I guess it is for the same
reason.  This is the log:

Jun  1 16:54:59 appweb2-coomeva kernel: ------------[ cut here
]------------
Jun  1 16:54:59 appweb2-coomeva kernel: kernel BUG at
mm/page_alloc.c:786!
Jun  1 16:54:59 appweb2-coomeva kernel: invalid operand: 0000 [#1]
Jun  1 16:54:59 appweb2-coomeva kernel: SMP 
Jun  1 16:54:59 appweb2-coomeva kernel: CPU:    0
Jun  1 16:54:59 appweb2-coomeva kernel: EIP:    0060:[<02138aec>]    Not
tainted
Jun  1 16:54:59 appweb2-coomeva kernel: EFLAGS: 00010246  
(2.6.5-1.358smp) 
Jun  1 16:54:59 appweb2-coomeva kernel: EIP is at __free_pages+0x21/0x4c
Jun  1 16:54:59 appweb2-coomeva kernel: eax: 00000000   ebx: 00000000  
ecx: 03d89180   edx: 03d89180
Jun  1 16:54:59 appweb2-coomeva kernel: esi: 00000fd9   edi: 7708597c  
ebp: 82953380   esp: 7d9b6f40
Jun  1 16:54:59 appweb2-coomeva kernel: ds: 007b   es: 007b   ss: 0068
Jun  1 16:54:59 appweb2-coomeva kernel: Process async IO 0/1 (pid: 2182,
threadinfo=7d9b6000 task=7c7b8750)
Jun  1 16:54:59 appweb2-coomeva kernel: Stack: 039dcf80 82947d46
7708597c 00000001 00000000 7708597c 82941008 00000000 
Jun  1 16:55:00 appweb2-coomeva kernel:        797e8054 00000000
00000000 7708597c 00000001 00000000 7d9b6000 82940ebf 
Jun  1 16:55:00 appweb2-coomeva kernel:        7708597c 00000001
80d769cc 8293aff6 80d769c4 7708597c 8293c039 00000001 
Jun  1 16:55:00 appweb2-coomeva kernel: Call Trace:
Jun  1 16:55:00 appweb2-coomeva kernel:  [<82947d46>]
get_abuf+0x9a/0x101 [tux]
Jun  1 16:55:00 appweb2-coomeva kernel:  [<82941008>]
http_pre_header+0x32/0x3ec [tux]
Jun  1 16:55:00 appweb2-coomeva kernel:  [<82940ebf>]
http_process_message+0x23a/0x266 [tux]
Jun  1 16:55:00 appweb2-coomeva kernel:  [<8293aff6>]
tux_schedule_atom+0x23/0x34 [tux]
Jun  1 16:55:00 appweb2-coomeva kernel:  [<8293c039>]
cachemiss_thread+0x128/0x1de [tux]
Jun  1 16:55:00 appweb2-coomeva kernel:  [<0211b419>]
default_wake_function+0x0/0xc
Jun  1 16:55:00 appweb2-coomeva kernel:  [<0211b419>]
default_wake_function+0x0/0xc
Jun  1 16:55:00 appweb2-coomeva kernel:  [<8293bf11>]
cachemiss_thread+0x0/0x1de [tux]
Jun  1 16:55:00 appweb2-coomeva kernel:  [<021041f1>]
kernel_thread_helper+0x5/0xb
Jun  1 16:55:01 appweb2-coomeva kernel: 
Jun  1 16:55:01 appweb2-coomeva kernel: Code: 0f 0b 12 03 c4 80 2b 02 f0
ff 49 04 0f 94 c0 84 c0 74 16 85 
Jun  1 17:04:55 appweb2-coomeva su(pam_unix)[7005]: session closed for
user root
Jun  1 17:04:57 appweb2-coomeva login(pam_unix)[6978]: session closed
for user nalwalovaton
Jun  2 08:38:42 appweb2-coomeva kernel:  <0>Bad page state at
free_hot_cold_page (in process 'async IO 0/2', page 0348b2a0)
Jun  2 08:38:42 appweb2-coomeva kernel: flags:0x20000000
mapping:00000000 mapped:0 count:0
Jun  2 08:38:42 appweb2-coomeva kernel: Backtrace:
Jun  2 08:38:42 appweb2-coomeva kernel: Call Trace:
Jun  2 08:38:42 appweb2-coomeva kernel:  [<02137f0d>] bad_page+0x5c/0x7f
Jun  2 08:38:42 appweb2-coomeva kernel:  [<02138593>]
free_hot_cold_page+0x67/0xe0
Jun  2 08:38:42 appweb2-coomeva kernel:  [<82947d46>]
get_abuf+0x9a/0x101 [tux]
Jun  2 08:38:42 appweb2-coomeva kernel:  [<82941008>]
http_pre_header+0x32/0x3ec [tux]
Jun  2 08:38:42 appweb2-coomeva kernel:  [<82940ebf>]
http_process_message+0x23a/0x266 [tux]
Jun  2 08:38:42 appweb2-coomeva kernel:  [<8293aff6>]
tux_schedule_atom+0x23/0x34 [tux]
Jun  2 08:38:42 appweb2-coomeva kernel:  [<8293c039>]
cachemiss_thread+0x128/0x1de [tux]
Jun  2 08:38:42 appweb2-coomeva kernel:  [<0211b419>]
default_wake_function+0x0/0xc
Jun  2 08:38:42 appweb2-coomeva kernel:  [<0211b419>]
default_wake_function+0x0/0xc
Jun  2 08:38:43 appweb2-coomeva kernel:  [<8293bf11>]
cachemiss_thread+0x0/0x1de [tux]
Jun  2 08:38:43 appweb2-coomeva kernel:  [<021041f1>]
kernel_thread_helper+0x5/0xb
Jun  2 08:38:43 appweb2-coomeva kernel: 
Jun  2 08:38:43 appweb2-coomeva kernel: Trying to fix it up, but a
reboot is needed
Jun  2 08:38:43 appweb2-coomeva kernel: Unable to handle kernel paging
request at virtual address 00100104
Jun  2 08:38:44 appweb2-coomeva kernel:  printing eip:
Jun  2 08:38:44 appweb2-coomeva kernel: 0213869f
Jun  2 08:38:44 appweb2-coomeva kernel: *pde = 00003001
Jun  2 08:38:44 appweb2-coomeva kernel: Oops: 0002 [#2]
Jun  2 08:38:44 appweb2-coomeva kernel: SMP 
Jun  2 08:38:44 appweb2-coomeva kernel: CPU:    2
Jun  2 08:38:44 appweb2-coomeva kernel: EIP:    0060:[<0213869f>]    Not
tainted
Jun  2 08:38:44 appweb2-coomeva kernel: EFLAGS: 00010002  
(2.6.5-1.358smp) 
Jun  2 08:38:44 appweb2-coomeva kernel: EIP is at
buffered_rmqueue+0x82/0x1a7
Jun  2 08:38:44 appweb2-coomeva kernel: eax: 0348b2b8   ebx: 022f3880  
ecx: 022f3c10   edx: 00100100
Jun  2 08:38:44 appweb2-coomeva kernel: esi: 022f3c00   edi: 022f3780  
ebp: 00000206   esp: 0ff05e30
Jun  2 08:38:44 appweb2-coomeva kernel: ds: 007b   es: 007b   ss: 0068
Jun  2 08:38:44 appweb2-coomeva kernel: Process httpd (pid: 7951,
threadinfo=0ff05000 task=6b6c80b0)
Jun  2 08:38:44 appweb2-coomeva kernel: Stack: 0348b2a0 000000d2
00000000 022f3780 00000000 022f5fa0 000000d2 02138861 
Jun  2 08:38:44 appweb2-coomeva kernel:        00000001 6b6c80b0
00000010 00000000 00000000 0038a025 00000000 07220720 
Jun  2 08:38:44 appweb2-coomeva kernel:        ebae4004 02141c86
00000000 00000000 249c4bc4 80b1f080 00000000 07220720 
Jun  2 08:38:45 appweb2-coomeva kernel: Call Trace:
Jun  2 08:38:45 appweb2-coomeva kernel:  [<02138861>]
__alloc_pages+0x9d/0x284
Jun  2 08:38:45 appweb2-coomeva kernel:  [<02141c86>]
do_anonymous_page+0x6d/0x192
Jun  2 08:38:45 appweb2-coomeva kernel:  [<02141e14>]
do_no_page+0x69/0x2b0
Jun  2 08:38:45 appweb2-coomeva kernel:  [<02140c53>]
follow_page+0x128/0x134
Jun  2 08:38:45 appweb2-coomeva kernel:  [<021421ca>]
handle_mm_fault+0xc0/0x16e
Jun  2 08:38:45 appweb2-coomeva kernel:  [<02118c2b>]
do_page_fault+0x133/0x4b4
Jun  2 08:38:45 appweb2-coomeva kernel:  [<02143576>]
do_mmap_pgoff+0x381/0x60c
Jun  2 08:38:45 appweb2-coomeva kernel:  [<0210b41f>]
sys_mmap2+0x77/0xab
Jun  2 08:38:45 appweb2-coomeva kernel:  [<02118af8>]
do_page_fault+0x0/0x4b4
Jun  2 08:38:45 appweb2-coomeva kernel: 
Jun  2 08:38:45 appweb2-coomeva kernel: Code: 89 4a 04 89 11 c7 40 04 00
02 20 00 c7 00 00 01 10 00 ff 8b 
Jun  2 08:38:53 appweb2-coomeva kernel:  <1>Unable to handle kernel
paging request at virtual address 00100104
Jun  2 08:38:53 appweb2-coomeva kernel:  printing eip:
Jun  2 08:38:53 appweb2-coomeva kernel: 021382ee
Jun  2 08:38:53 appweb2-coomeva kernel: *pde = 00003001
Jun  2 08:47:45 appweb2-coomeva syslogd 1.4.1: restart.



And 5 days ago I had the same problem... this time while I was executing
gkrellm in a remote desktop.  This is the log:

May 28 07:43:59 appweb2-coomeva kernel: Unable to handle kernel paging
request at virtual address 00100104
May 28 07:43:59 appweb2-coomeva kernel:  printing eip:
May 28 07:43:59 appweb2-coomeva kernel: 0213869f
May 28 07:43:59 appweb2-coomeva kernel: *pde = 00003001
May 28 07:43:59 appweb2-coomeva kernel: Oops: 0002 [#1]
May 28 07:43:59 appweb2-coomeva kernel: SMP 
May 28 07:43:59 appweb2-coomeva kernel: CPU:    0
May 28 07:43:59 appweb2-coomeva kernel: EIP:    0060:[<0213869f>]    Not
tainted
May 28 07:43:59 appweb2-coomeva kernel: EFLAGS: 00010002  
(2.6.5-1.358smp) 
May 28 07:43:59 appweb2-coomeva kernel: EIP is at
buffered_rmqueue+0x82/0x1a7
May 28 07:43:59 appweb2-coomeva kernel: eax: 03c900b8   ebx: 022f3780  
ecx: 022f3b10   edx: 00100100
May 28 07:43:59 appweb2-coomeva kernel: esi: 022f3b00   edi: 022f3780  
ebp: 00000246   esp: 6b09fe1c
May 28 07:43:59 appweb2-coomeva kernel: ds: 007b   es: 007b   ss: 0068
May 28 07:44:00 appweb2-coomeva kernel: Process gkrellm (pid: 2078,
threadinfo=6b09f000 task=691d4bf0)
May 28 07:44:00 appweb2-coomeva kernel: Stack: 03c900a0 000000d2
00000000 022f3780 00000000 022f5fa0 000000d2 02138861 
May 28 07:44:00 appweb2-coomeva kernel:        00000001 691d4bf0
00000010 00000000 6b09feb3 0038a025 00000000 66222490 
May 28 07:44:00 appweb2-coomeva kernel:        f3e92000 02141c86
00000000 00000000 6f3f1d04 68b5a800 00000000 66222490 
May 28 07:44:00 appweb2-coomeva kernel: Call Trace:
May 28 07:44:00 appweb2-coomeva kernel:  [<02138861>]
__alloc_pages+0x9d/0x284
May 28 07:48:50 appweb2-coomeva syslogd 1.4.1: restart.



Is this bug new?  Should I file it in bugzilla??

Any help about this will be highly appreaciated.  If you need more info
don't hesitate to ask me.

Thanks,


-William




[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]