[Linux-cluster] Need some help with an Oops

David Brieck Jr. dbrieck at gmail.com
Mon Sep 18 13:56:37 UTC 2006


I'm trying to figure out why I'm getting this Oops every time I copy
multiple large files to my shared storage. So far I'm just working on
this one node, but I'm going to try to see if it happens on other
nodes as well.

Basically we're using GNBD and GFS with multipath and CLVM from U4.
I'm trying to setup backups from the nodes to the shared storage for
tape backup at a later point. Basically just tar gzipping up each fs
directly to shared storage. It seems to work ok for smaller files
(such as the root and boot), but for larger files it hangs the node.
It then gets power fenced and rebooted.

There doesn't seem to be any related log files on the other nodes
(except on the GNBD server saying a gnbd process exited because of
signal 15) so I'm not sure what might be causing it and all I have are
the logs. Any help would be greatly appreciated. If I need to provide
more info please let me know.
-------------- next part --------------
Sep 18 09:21:11 http1 kernel: do_IRQ: stack overflow: 476
Sep 18 09:21:11 http1 kernel:  [<c010795b>] do_IRQ+0x49/0x1ae<1>Unable to handle kernel paging request at virtual address 98393c30
Sep 18 09:21:11 http1 kernel:  printing eip:
Sep 18 09:21:11 http1 kernel: c011aef5
Sep 18 09:21:11 http1 kernel: *pde = f6e22080
Sep 18 09:21:11 http1 kernel: Oops: 0000 [#1]
Sep 18 09:21:11 http1 kernel: SMP 
Sep 18 09:21:11 http1 kernel: Modules linked in: mptctl mptbase dell_rbu parport_pc lp parport autofs4 i2c_dev i2c_core lock_dlm(U) gfs(U) lock_harness(U) dm_round_robin gnbd(U) dlm(U) cman(U) sunrpc ipmi_devintf ipmi_si ipmi_msghandler md5 ipv6 dm_mirror dm_multipath dm_mod joydev button battery ac uhci_hcd ehci_hcd hw_random shpchp e1000 bonding(U) floppy sg ext3 jbd megaraid_mbox megaraid_mm sd_mod scsi_mod
Sep 18 09:21:11 http1 kernel: CPU:    75
Sep 18 09:21:11 http1 kernel: EIP:    0060:[<c011aef5>]    Not tainted VLI
Sep 18 09:21:11 http1 kernel: EFLAGS: 00010046   (2.6.9-42.0.2.ELsmp) 
Sep 18 09:21:11 http1 kernel: EIP is at do_page_fault+0xa0/0x5c6
Sep 18 09:21:11 http1 kernel: eax: ece09000   ebx: c0416632   ecx: ece09074   edx: ece09130
Sep 18 09:21:11 http1 kernel: esi: 00000000   edi: c011ae55   ebp: 98393bc0   esp: ece09064
Sep 18 09:21:11 http1 kernel: ds: 007b   es: 007b   ss: 0068
iso-8859-1Sep 18 09:21:11 http1 kernel: Process ???? (pid: 4194367, threadinfo=ece08000 task=efc7c000)
Sep 18 09:21:11 http1 kernel: Stack: 00000000 98393c30 00000000 00000000 ece09130 c02e5e84 00000000 0000000e 
Sep 18 09:21:11 http1 kernel:        0000000b 00000000 00000000 00000000 00000000 00000000 00030001 00000000 
Sep 18 09:21:11 http1 kernel:        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
Sep 18 09:21:11 http1 kernel: Call Trace:
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:11 http1 kernel:  =======================
Sep 18 09:21:11 http1 kernel:  [<f8840f01>] .text.lock.scsi+0x1a/0x35 [scsi_mod]
Sep 18 09:21:11 http1 kernel:  [<f8890424>] ext3_alloc_branch+0x194/0x260 [ext3]
Sep 18 09:21:11 http1 kernel:  =======================
Sep 18 09:21:11 http1 kernel: Unable to handle kernel paging request at virtual address 5e5b10c4
Sep 18 09:21:11 http1 kernel:  printing eip:
Sep 18 09:21:11 http1 kernel: c0105cd0
Sep 18 09:21:11 http1 kernel: *pde = 00000000
Sep 18 09:21:11 http1 kernel: Oops: 0000 [#2]
Sep 18 09:21:11 http1 kernel: SMP 
Sep 18 09:21:11 http1 kernel: Modules linked in: mptctl mptbase dell_rbu parport_pc lp parport autofs4 i2c_dev i2c_core lock_dlm(U) gfs(U) lock_harness(U) dm_round_robin gnbd(U) dlm(U) cman(U) sunrpc ipmi_devintf ipmi_si ipmi_msghandler md5 ipv6 dm_mirror dm_multipath dm_mod joydev button battery ac uhci_hcd ehci_hcd hw_random shpchp e1000 bonding(U) floppy sg ext3 jbd megaraid_mbox megaraid_mm sd_mod scsi_mod
Sep 18 09:21:11 http1 kernel: CPU:    75
Sep 18 09:21:11 http1 kernel: EIP:    0060:[<c0105cd0>]    Not tainted VLI
Sep 18 09:21:11 http1 kernel: EFLAGS: 00010093   (2.6.9-42.0.2.ELsmp) 
Sep 18 09:21:11 http1 kernel: EIP is at show_trace+0x11/0x6b
Sep 18 09:21:11 http1 kernel: eax: 5e5b1ffd   ebx: 5e5b10c4   ecx: ece08ed0   edx: c02de60c
Sep 18 09:21:11 http1 kernel: esi: 5e5b10c4   edi: 5e5b1000   ebp: 00000068   esp: ece08ed0
Sep 18 09:21:11 http1 kernel: ds: 007b   es: 007b   ss: 0068
iso-8859-1Sep 18 09:21:11 http1 kernel: Process ???? (pid: 4194367, threadinfo=ece08000 task=efc7c000)
Sep 18 09:21:11 http1 kernel: Stack: ece090c4 00000018 00000000 c0105d9d c02de636 ece09064 ece08000 ece09030 
Sep 18 09:21:11 http1 kernel:        00000000 c0105e9c c02de739 00000001 ece08000 ece09030 00000000 c02e5f8c 
Sep 18 09:21:11 http1 kernel:        c0106043 ece09030 c02e5f8c 00000000 000000ff 0000000b c01229e5 c02e5f23 
Sep 18 09:21:11 http1 kernel: Call Trace:
Sep 18 09:21:11 http1 kernel:  [<c0105d9d>] show_stack+0x73/0x79
Sep 18 09:21:11 http1 kernel:  [<c0105e9c>] show_registers+0xe6/0x14d
Sep 18 09:21:11 http1 kernel:  [<c0106043>] die+0xdb/0x16b
Sep 18 09:21:11 http1 kernel:  [<c01229e5>] vprintk+0x136/0x14a
Sep 18 09:21:11 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011b245>] do_page_fault+0x3f0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:11 http1 kernel:  [<c015c79d>] end_buffer_async_write+0x0/0xc0
Sep 18 09:21:11 http1 kernel:  [<c015c03d>] end_buffer_read_sync+0x0/0x1f
Sep 18 09:21:11 http1 kernel:  =======================
Sep 18 09:21:11 http1 kernel: Unable to handle kernel paging request at virtual address 00160015
Sep 18 09:21:11 http1 kernel:  printing eip:
Sep 18 09:21:11 http1 kernel: c0105cd0
Sep 18 09:21:11 http1 kernel: *pde = 2b5c5001
Sep 18 09:21:11 http1 kernel: Recursive die() failure, output suppressed
Sep 18 09:21:11 http1 kernel:  <1>Unable to handle kernel paging request at virtual address 0001006f
Sep 18 09:21:11 http1 kernel:  printing eip:
Sep 18 09:21:11 http1 kernel: c011aef5
Sep 18 09:21:11 http1 kernel: *pde = 2b5c5001
Sep 18 09:21:11 http1 kernel: Oops: 0000 [#3]
Sep 18 09:21:11 http1 kernel: SMP 
Sep 18 09:21:11 http1 kernel: Modules linked in: mptctl mptbase dell_rbu parport_pc lp parport autofs4 i2c_dev i2c_core lock_dlm(U) gfs(U) lock_harness(U) dm_round_robin gnbd(U) dlm(U) cman(U) sunrpc ipmi_devintf ipmi_si ipmi_msghandler md5 ipv6 dm_mirror dm_multipath dm_mod joydev button battery ac uhci_hcd ehci_hcd hw_random shpchp e1000 bonding(U) floppy sg ext3 jbd megaraid_mbox megaraid_mm sd_mod scsi_mod
Sep 18 09:21:11 http1 kernel: CPU:    31
Sep 18 09:21:11 http1 kernel: EIP:    0060:[<c011aef5>]    Not tainted VLI
Sep 18 09:21:11 http1 kernel: EFLAGS: 00010046   (2.6.9-42.0.2.ELsmp) 
Sep 18 09:21:11 http1 kernel: EIP is at do_page_fault+0xa0/0x5c6
Sep 18 09:21:11 http1 kernel: eax: f7f61000   ebx: 0000ffff   ecx: f7f610ec   edx: f7f611a8
Sep 18 09:21:11 http1 kernel: esi: 00000000   edi: c011ae55   ebp: 0000ffff   esp: f7f610dc
Sep 18 09:21:11 http1 kernel: ds: 007b   es: 007b   ss: 0068
Sep 18 09:21:11 http1 kernel: Process  (pid: -331851520, threadinfo=f7f60000 task=f7f76000)
Sep 18 09:21:12 http1 kernel: Stack: 00000000 0001006f 00000000 00000000 f7f611a8 c02e5e84 00000000 0000000e 
Sep 18 09:21:12 http1 kernel:        0000000b 00004000 00000000 00000000 00000000 00000000 00030001 00000000 
Sep 18 09:21:12 http1 kernel:        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000001 
Sep 18 09:21:12 http1 kernel: Call Trace:
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c01cc01b>] disable_msi_mode+0x27/0x9d
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:12 http1 kernel:  [<c012a312>] update_one_process+0x7/0xce
Sep 18 09:21:12 http1 kernel:  [<c012a3f7>] update_process_times+0x1e/0x2f
Sep 18 09:21:12 http1 kernel:  [<c0117462>] smp_apic_timer_interrupt+0x5e/0x9c
Sep 18 09:21:12 http1 kernel: Code: 81 7c 24 04 ff ff ff bf 8b 28 c7 44 24 38 01 00 03 00 76 13 f6 84 24 c8 00 00 00 05 0f 85 a4 01 00 00 e9 75 04 00 00 83 78 14 00 <8b> 5d 70 0f 85 92 01 00 00 85 db 0f 84 8a 01 00 00 8d 73 30 8b 
Sep 18 09:21:12 http1 kernel:  <1>Unable to handle kernel NULL pointer dereference at virtual address 00000000
Sep 18 09:21:12 http1 kernel:  printing eip:
Sep 18 09:21:12 http1 kernel: c011743e
Sep 18 09:21:12 http1 kernel: *pde = 2b5c5001
Sep 18 09:21:12 http1 kernel: Oops: 0002 [#4]
Sep 18 09:21:12 http1 kernel: SMP 
Sep 18 09:21:12 http1 kernel: Modules linked in: mptctl mptbase dell_rbu parport_pc lp parport autofs4 i2c_dev i2c_core lock_dlm(U) gfs(U) lock_harness(U) dm_round_robin gnbd(U) dlm(U) cman(U) sunrpc ipmi_devintf ipmi_si ipmi_msghandler md5 ipv6 dm_mirror dm_multipath dm_mod joydev button battery ac uhci_hcd ehci_hcd hw_random shpchp e1000 bonding(U) floppy sg ext3 jbd megaraid_mbox megaraid_mm sd_mod scsi_mod
Sep 18 09:21:12 http1 kernel: CPU:    31
Sep 18 09:21:12 http1 kernel: EIP:    0060:[<c011743e>]    Not tainted VLI
Sep 18 09:21:12 http1 kernel: EFLAGS: 00010086   (2.6.9-42.0.2.ELsmp) 
Sep 18 09:21:12 http1 kernel: EIP is at smp_apic_timer_interrupt+0x3a/0x9c
Sep 18 09:21:12 http1 kernel: eax: f7f76000   ebx: 00000ffc   ecx: f7f60000   edx: 00000000
Sep 18 09:21:12 http1 kernel: esi: f7f610a8   edi: 00000000   ebp: c02e5f8c   esp: f7f60f50
Sep 18 09:21:12 http1 kernel: ds: 007b   es: 007b   ss: 0068
Sep 18 09:21:12 http1 kernel: Process  (pid: -331851520, threadinfo=f7f60000 task=f7f76000)
Sep 18 09:21:12 http1 kernel: Stack: fffff000 c02d521a fffff000 f7f60f00 c0323e4c f7f610a8 00000000 c02e5f8c 
Sep 18 09:21:12 http1 kernel:        00000000 c030007b f7f6007b ffffffef c0106077 00000060 00000246 f7f610a8 
Sep 18 09:21:12 http1 kernel:        c02e5f8c 00000000 000000ff 0000000b c01229e5 c02e5f23 00000000 c02e5f23 
Sep 18 09:21:12 http1 kernel: Call Trace:
Sep 18 09:21:12 http1 kernel:  [<c02d521a>] apic_timer_interrupt+0x1a/0x20
Sep 18 09:21:12 http1 kernel:  [<c0106077>] die+0x10f/0x16b
Sep 18 09:21:12 http1 kernel:  [<c01229e5>] vprintk+0x136/0x14a
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011b245>] do_page_fault+0x3f0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:12 http1 kernel:  =======================
Sep 18 09:21:12 http1 kernel: Unable to handle kernel paging request at virtual address 00160015
Sep 18 09:21:12 http1 kernel:  printing eip:
Sep 18 09:21:12 http1 kernel: c0105cd0
Sep 18 09:21:12 http1 kernel: *pde = 2b5c5001
Sep 18 09:21:12 http1 kernel: Oops: 0000 [#5]
Sep 18 09:21:12 http1 kernel: SMP 
Sep 18 09:21:12 http1 kernel: Modules linked in: mptctl mptbase dell_rbu parport_pc lp parport autofs4 i2c_dev i2c_core lock_dlm(U) gfs(U) lock_harness(U) dm_round_robin gnbd(U) dlm(U) cman(U) sunrpc ipmi_devintf ipmi_si ipmi_msghandler md5 ipv6 dm_mirror dm_multipath dm_mod joydev button battery ac uhci_hcd ehci_hcd hw_random shpchp e1000 bonding(U) floppy sg ext3 jbd megaraid_mbox megaraid_mm sd_mod scsi_mod
Sep 18 09:21:12 http1 kernel: CPU:    31
Sep 18 09:21:12 http1 kernel: EIP:    0060:[<c0105cd0>]    Not tainted VLI
Sep 18 09:21:12 http1 kernel: EFLAGS: 00010097   (2.6.9-42.0.2.ELsmp) 
Sep 18 09:21:12 http1 kernel: EIP is at show_trace+0x11/0x6b
Sep 18 09:21:12 http1 kernel: eax: 00160ffd   ebx: 00160015   ecx: f7f60dbc   edx: c02de60c
Sep 18 09:21:12 http1 kernel: esi: 00160015   edi: 00160000   ebp: 00000068   esp: f7f60dbc
Sep 18 09:21:12 http1 kernel: ds: 007b   es: 007b   ss: 0068
Sep 18 09:21:12 http1 kernel: Process  (pid: -331851520, threadinfo=f7f60000 task=f7f76000)
Sep 18 09:21:12 http1 kernel: Stack: f7f60fb0 00000018 00000000 c0105d9d c02de636 f7f60f50 f7f60000 f7f60f1c 
Sep 18 09:21:12 http1 kernel:        00000000 c0105e9c c02de739 00000001 f7f60000 f7f60f1c 00000002 c02e5f8c 
Sep 18 09:21:12 http1 kernel:        c0106043 f7f60f1c c02e5f8c 00000002 000000ff 0000000b c01229e5 c02e5eef 
Sep 18 09:21:12 http1 kernel: Call Trace:
Sep 18 09:21:12 http1 kernel:  [<c0105d9d>] show_stack+0x73/0x79
Sep 18 09:21:12 http1 kernel:  [<c0105e9c>] show_registers+0xe6/0x14d
Sep 18 09:21:12 http1 kernel:  [<c0106043>] die+0xdb/0x16b
Sep 18 09:21:12 http1 kernel:  [<c01229e5>] vprintk+0x136/0x14a
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011b245>] do_page_fault+0x3f0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011743e>] smp_apic_timer_interrupt+0x3a/0x9c
Sep 18 09:21:12 http1 kernel:  [<c020b69e>] vt_console_print+0x65/0x2a5
Sep 18 09:21:12 http1 kernel:  [<c020b639>] vt_console_print+0x0/0x2a5
Sep 18 09:21:12 http1 kernel:  [<c01c27dd>] vsnprintf+0x448/0x488
Sep 18 09:21:12 http1 kernel:  [<c020b69e>] vt_console_print+0x65/0x2a5
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:12 http1 kernel:  [<c011743e>] smp_apic_timer_interrupt+0x3a/0x9c
Sep 18 09:21:12 http1 kernel:  [<c02d521a>] apic_timer_interrupt+0x1a/0x20
Sep 18 09:21:12 http1 kernel:  [<c0106077>] die+0x10f/0x16b
Sep 18 09:21:12 http1 kernel:  [<c01229e5>] vprintk+0x136/0x14a
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011b245>] do_page_fault+0x3f0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:12 http1 kernel:  =======================
Sep 18 09:21:12 http1 kernel: Unable to handle kernel paging request at virtual address 00160015
Sep 18 09:21:12 http1 kernel:  printing eip:
Sep 18 09:21:12 http1 kernel: c0105cd0
Sep 18 09:21:12 http1 kernel: *pde = 2b5c5001
Sep 18 09:21:12 http1 kernel: Recursive die() failure, output suppressed
Sep 18 09:21:12 http1 kernel:  <1>Unable to handle kernel NULL pointer dereference at virtual address 00000000
Sep 18 09:21:12 http1 kernel:  printing eip:
Sep 18 09:21:12 http1 kernel: c011743e
Sep 18 09:21:12 http1 kernel: *pde = 2b5c5001
Sep 18 09:21:12 http1 kernel: Oops: 0002 [#6]
Sep 18 09:21:12 http1 kernel: SMP 
Sep 18 09:21:12 http1 kernel: Modules linked in: mptctl mptbase dell_rbu parport_pc lp parport autofs4 i2c_dev i2c_core lock_dlm(U) gfs(U) lock_harness(U) dm_round_robin gnbd(U) dlm(U) cman(U) sunrpc ipmi_devintf ipmi_si ipmi_msghandler md5 ipv6 dm_mirror dm_multipath dm_mod joydev button battery ac uhci_hcd ehci_hcd hw_random shpchp e1000 bonding(U) floppy sg ext3 jbd megaraid_mbox megaraid_mm sd_mod scsi_mod
Sep 18 09:21:12 http1 kernel: CPU:    31
Sep 18 09:21:12 http1 kernel: EIP:    0060:[<c011743e>]    Not tainted VLI
Sep 18 09:21:12 http1 kernel: EFLAGS: 00010086   (2.6.9-42.0.2.ELsmp) 
Sep 18 09:21:12 http1 kernel: EIP is at smp_apic_timer_interrupt+0x3a/0x9c
Sep 18 09:21:12 http1 kernel: eax: f7f76000   ebx: 00000ffc   ecx: f7f60000   edx: 00000000
Sep 18 09:21:12 http1 kernel: esi: f7f60bf4   edi: c02de7d4   ebp: c02e5f8c   esp: f7f60a9c
Sep 18 09:21:12 http1 kernel: ds: 007b   es: 007b   ss: 0068
Sep 18 09:21:12 http1 kernel: Process  (pid: -331851520, threadinfo=f7f60000 task=f7f76000)
Sep 18 09:21:12 http1 kernel: Stack: fffff000 c02d521a fffff000 f7f60a00 c0323e4c f7f60bf4 c02de7d4 c02e5f8c 
Sep 18 09:21:12 http1 kernel:        00000000 c030007b f7f6007b ffffffef c0106077 00000060 00000246 0000a81b 
Sep 18 09:21:12 http1 kernel:        c0122acd c0416633 00000003 00000013 c01229e5 c02e5f23 00000000 c02e5f23 
Sep 18 09:21:12 http1 kernel: Call Trace:
Sep 18 09:21:12 http1 kernel:  [<c02d521a>] apic_timer_interrupt+0x1a/0x20
Sep 18 09:21:12 http1 kernel:  [<c0106077>] die+0x10f/0x16b
Sep 18 09:21:12 http1 kernel:  [<c0122acd>] release_console_sem+0x75/0xa9
Sep 18 09:21:12 http1 kernel:  [<c01229e5>] vprintk+0x136/0x14a
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011b245>] do_page_fault+0x3f0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c0105cd0>] show_trace+0x11/0x6b
Sep 18 09:21:12 http1 kernel:  [<c020b69e>] vt_console_print+0x65/0x2a5
Sep 18 09:21:12 http1 kernel:  [<c020b639>] vt_console_print+0x0/0x2a5
Sep 18 09:21:12 http1 kernel:  [<c012268f>] __call_console_drivers+0x36/0x40
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:12 http1 kernel:  [<c0105cd0>] show_trace+0x11/0x6b
Sep 18 09:21:12 http1 kernel:  [<c0105d9d>] show_stack+0x73/0x79
Sep 18 09:21:12 http1 kernel:  [<c0105e9c>] show_registers+0xe6/0x14d
Sep 18 09:21:12 http1 kernel:  [<c0106043>] die+0xdb/0x16b
Sep 18 09:21:12 http1 kernel:  [<c01229e5>] vprintk+0x136/0x14a
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011b245>] do_page_fault+0x3f0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c0105cd0>] show_trace+0x11/0x6b
Sep 18 09:21:12 http1 kernel:  [<c020b69e>] vt_console_print+0x65/0x2a5
Sep 18 09:21:12 http1 kernel:  [<c020b639>] vt_console_print+0x0/0x2a5
Sep 18 09:21:12 http1 kernel:  [<c012268f>] __call_console_drivers+0x36/0x40
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:12 http1 kernel:  [<c0105cd0>] show_trace+0x11/0x6b
Sep 18 09:21:12 http1 kernel:  [<c0105d9d>] show_stack+0x73/0x79
Sep 18 09:21:12 http1 kernel:  [<c0105e9c>] show_registers+0xe6/0x14d
Sep 18 09:21:12 http1 kernel:  [<c0106043>] die+0xdb/0x16b
Sep 18 09:21:12 http1 kernel:  [<c01229e5>] vprintk+0x136/0x14a
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011b245>] do_page_fault+0x3f0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011743e>] smp_apic_timer_interrupt+0x3a/0x9c
Sep 18 09:21:12 http1 kernel:  [<c020b69e>] vt_console_print+0x65/0x2a5
Sep 18 09:21:12 http1 kernel:  [<c020b639>] vt_console_print+0x0/0x2a5
Sep 18 09:21:12 http1 kernel:  [<c01c27dd>] vsnprintf+0x448/0x488
Sep 18 09:21:12 http1 kernel:  [<c020b69e>] vt_console_print+0x65/0x2a5
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c02d52b7>] error_code+0x2f/0x38
Sep 18 09:21:12 http1 kernel:  [<c011743e>] smp_apic_timer_interrupt+0x3a/0x9c
Sep 18 09:21:12 http1 kernel:  [<c02d521a>] apic_timer_interrupt+0x1a/0x20
Sep 18 09:21:12 http1 kernel:  [<c0106077>] die+0x10f/0x16b
Sep 18 09:21:12 http1 kernel:  [<c01229e5>] vprintk+0x136/0x14a
Sep 18 09:21:12 http1 kernel:  [<c011ae55>] do_page_fault+0x0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011b245>] do_page_fault+0x3f0/0x5c6
Sep 18 09:21:12 http1 kernel:  [<c011aef5>] do_page_fault+0xa0/0x5c6
Sep 18 09:21:12 http1 kernel:  =======================
Sep 18 09:21:12 http1 kernel: Unable to handle kernel paging request at virtual address 00160015


More information about the Linux-cluster mailing list