[Linux-cluster] GFS 6.0 crashing x86_64 machine
micah nerren
mnerren at paracel.com
Fri Aug 6 22:03:39 UTC 2004
On Fri, 2004-08-06 at 09:45, Michael Conrad Tadpol Tilstra wrote:
> On Thu, Aug 05, 2004 at 04:52:53PM -0700, micah nerren wrote:
> >
> > FYI, I tried this with a few different HBA's, that didn't work. I
> > thought perhaps it could be some funny interaction with the driver but
> > that doesn't seem to be the case.
> >
> > If there is anything I can do to help, please let me know! Up to and
> > including allowing access to the machines running the software if that
> > will help you debug it.
>
> well, at this point I'd try things without the hbas and without gulm.
> So first off, try mounting gfs using nolock instead of gulm on a single
> node.
> Then gets some space on a local drive to put gfs (without pool first)
> and use gulm to mount that. (kinda pointless other than just seeing if
> it does an oops.)
> If that works, put pool onto the local disk and try again.
>
> That should give us a good idea of what parts need to be involved to get
> the oops.
Alrighty, I thought I'd give you the latest on our efforts along these
lines. We are progressing down the paths you suggested, and wanted to
post a few results before the weekend.
We have used nolock instead of gulm, still on the pool device over the
HBA, and received a crash. Attached are two traces of the crashes. We
edited the code sprinkling printk's throughout to get some output.
Using lock_nolock instead of lock_gulm still crashes, but slightly
differently. See koops-nolock.txt
The tracing printk()'s added to lock_gulm and gfs don't show much, but
the crash is different yet again. See koops-gulm-traced.txt
The tracing messages use -> for enter, <- for leave and ?? for
"This function returns in far too many places to bother."
Later this evening or monday, I will attempt building a local file
system without a pool, then with a pool, to give you some more data.
Thanks,
Micah
-------------- next part --------------
Lock_Harness v6.0.0 (built Aug 6 2004 20:27:11) installed
Gulm v6.0.0 (built Aug 6 2004 20:27:09) installed
Debugging printks added at paracel.
GFS v6.0.0 (built Aug 6 2004 20:26:48) installed
->gfs_read_super(774e0000, 0, 0)
->gfs_mount_lockproto({proto="", table="", host=""}, 0)
->gulm_mount("hopkins:gfs02", "", a0128980, 1cf000, 32, 24f6b8)
->start_gulm_threads("hopkins", "")
->cm_login()
??lg_core_login(7768200, 1)
??xdr_enc_flush(776515c0)
??lg_core_handle_messages(7768200, a010ca00, 0)
??gulm_core_login_reply(0, 0, 0, -1, 3)
->lt_login()
??lg_lock_login(7768200, {71, 70, 83, 32})
Unable to handle kernel paging request at virtual address 0000000100000000
printing rip:
ffffffff802b5dd2
PML4 775d3067 PGD 0
Oops: 0000
CPU 0
Pid: 4026, comm: mount Not tainted
RIP: 0010:[<ffffffff802b5dd2>]{memcpy+18}
RSP: 0018:00000100775fb238 EFLAGS: 00010002
RAX: ffffffff805d3928 RBX: 00000100775fa760 RCX: 0000000000000001
RDX: 0000000000000080 RSI: 0000000100000000 RDI: ffffffff805d3928
RBP: 0000000000000000 R08: 00000000ffffffff R09: 00000100076bf840
R10: 0000002a95782200 R11: 0000000000000246 R12: 000001007bf46760
R13: 00000100775fa000 R14: 000001007bf46000 R15: ffffffff805d38c0
FS: 0000002a955764c0(0000) GS:ffffffff805d9840(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000100000000 CR3: 0000000000101000 CR4: 00000000000006e0
Call Trace: [<ffffffff8010ed13>]{__switch_to+499} [<ffffffff8011f8c2>]{thread_r
[<ffffffff80101000>]{init_level4_pgt+0} [<ffffffff8012f9b5>]{schedule_ti
[<ffffffff802b5915>]{do_softirq_thunk+53} [<ffffffff8028e0df>]{inet_wait
[<ffffffff8028e313>]{inet_stream_connect+339} [<ffffffffa0108f0c>]{:lock
[<ffffffffa010676c>]{:lock_gulm:xdr_connect+28} [<ffffffffa01035a5>]{:lo
[<ffffffffa01000bf>]{:lock_gulm:lt_login+63} [<ffffffffa00fc184>]{:lock_
[<ffffffffa010ca00>]{:lock_gulm:core_cb+0} [<ffffffffa0102202>]{:lock_gu
[<ffffffffa010284a>]{:lock_gulm:lg_core_login+346}
[<ffffffffa00fc568>]{:lock_gulm:cm_login+136} [<ffffffffa00fcc36>]{:lock
[<ffffffffa00fcfa9>]{:lock_gulm:gulm_mount+665} [<ffffffffa0128980>]{:gf
[<ffffffffa00fb3e3>]{:lock_harness:lm_mount_Rsmp_ad6c5c21+355}
[<ffffffffa0128980>]{:gfs:gfs_glock_cb+0} [<ffffffffa012e09e>]{:gfs:gfs_
[<ffffffff8013d8d2>]{do_anonymous_page+1234} [<ffffffff8013d94f>]{do_no_
[<ffffffff801a5103>]{do_page_fault+627} [<ffffffff801109d6>]{error_exit+
[<ffffffff801ebbe9>]{serial_in+41} [<ffffffff8011e20d>]{wake_up_cpu+29}
[<ffffffffa011939a>]{:gfs:gfs_read_super+1338} [<ffffffffa014dca0>]{:gfs
[<ffffffff80164c0c>]{get_sb_bdev+588} [<ffffffffa014dca0>]{:gfs:gfs_fs_t
[<ffffffff80164ec9>]{do_kern_mount+121} [<ffffffff8017baa1>]{do_add_moun
[<ffffffff8017bdb9>]{do_mount+345} [<ffffffff80154b40>]{__get_free_pages
[<ffffffff8017c1d5>]{sys_mount+197} [<ffffffff80110177>]{system_call+119
Process mount (pid: 4026, stackpage=100775fb000)
Stack: 00000100775fb238 0000000000000018 0000000000000040 00000100775fa760
ffffffff8010ed13 0000000000000006 00000100775fa000 000001007bf47ed8
000001007bf46000 ffffffff805e02c0 0000000000000000 0000000000000079
ffffffff8011f8c2 00000100775fb328 000001007ad88000 0000000000000020
0000000000000006 00000100775ffb40 0000000000000000 ffffffff80101000
000001007b571000 0000000000000069 0000000000000000 ffffffff805e02c0
00000100775fa000 00000100775fa000 000001007ad88000 0000000000000010
00000100775260c0 7fffffffffffffff 0000010077526108 00000100775fb3c8
0000000000000010 7fffffffffffffff ffffffff8012f9b5 00000100775fb3c8
ffffffff802b5915 0000000000000020 0000000000000006 000001000478929e
Call Trace: [<ffffffff8010ed13>]{__switch_to+499} [<ffffffff8011f8c2>]{thread_r
[<ffffffff80101000>]{init_level4_pgt+0} [<ffffffff8012f9b5>]{schedule_ti
[<ffffffff802b5915>]{do_softirq_thunk+53} [<ffffffff8028e0df>]{inet_wait
[<ffffffff8028e313>]{inet_stream_connect+339} [<ffffffffa0108f0c>]{:lock
[<ffffffffa010676c>]{:lock_gulm:xdr_connect+28} [<ffffffffa01035a5>]{:lo
[<ffffffffa01000bf>]{:lock_gulm:lt_login+63} [<ffffffffa00fc184>]{:lock_
[<ffffffffa010ca00>]{:lock_gulm:core_cb+0} [<ffffffffa0102202>]{:lock_gu
[<ffffffffa010284a>]{:lock_gulm:lg_core_login+346}
[<ffffffffa00fc568>]{:lock_gulm:cm_login+136} [<ffffffffa00fcc36>]{:lock
[<ffffffffa00fcfa9>]{:lock_gulm:gulm_mount+665} [<ffffffffa0128980>]{:gf
[<ffffffffa00fb3e3>]{:lock_harness:lm_mount_Rsmp_ad6c5c21+355}
[<ffffffffa0128980>]{:gfs:gfs_glock_cb+0} [<ffffffffa012e09e>]{:gfs:gfs_
[<ffffffff8013d8d2>]{do_anonymous_page+1234} [<ffffffff8013d94f>]{do_no_
[<ffffffff801a5103>]{do_page_fault+627} [<ffffffff801109d6>]{error_exit+
[<ffffffff801ebbe9>]{serial_in+41} [<ffffffff8011e20d>]{wake_up_cpu+29}
[<ffffffffa011939a>]{:gfs:gfs_read_super+1338} [<ffffffffa014dca0>]{:gfs
[<ffffffff80164c0c>]{get_sb_bdev+588} [<ffffffffa014dca0>]{:gfs:gfs_fs_t
[<ffffffff80164ec9>]{do_kern_mount+121} [<ffffffff8017baa1>]{do_add_moun
[<ffffffff8017bdb9>]{do_mount+345} [<ffffffff80154b40>]{__get_free_pages
[<ffffffff8017c1d5>]{sys_mount+197} [<ffffffff80110177>]{system_call+119
Code: 4c 8b 1e 4c 8b 46 08 4c 89 1f 4c 89 47 08 4c 8b 4e 10 4c 8b
Kernel panic: Fatal exception
NMI Watchdog detected LOCKUP on CPU0, eip ffffffff8012162f, registers:
CPU 0
Pid: 4026, comm: mount Not tainted
RIP: 0010:[<ffffffff8012162f>]{.text.lock.sched+131}
RSP: 0018:ffffffff805de5c0 EFLAGS: 00000086
RAX: 0000000000000000 RBX: 00000100775fa000 RCX: 00000000000a6040
RDX: ffffffff8049d6a0 RSI: ffffffff8049d6b0 RDI: 0000000000000000
RBP: ffffffff805de5f0 R08: 0000000000000000 R09: ffffffff8049d6a0
R10: ffffffff8049d690 R11: 00000100775fad28 R12: ffffffff805e02c0
R13: 000000000000000b R14: 0000000000000000 R15: 00000000000033c5
FS: 0000002a955764c0(0000) GS:ffffffff805d9840(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000100000000 CR3: 0000000000101000 CR4: 00000000000006e0
Call Trace: <EOE> [<ffffffff8011b263>]{smp_apic_timer_interrupt+291}
[<ffffffff801108dc>]{apic_timer_interrupt+64} [<ffffffff8011302e>]{handl
[<ffffffff801132e2>]{do_IRQ+274} [<ffffffff801106d7>]{common_interrupt+9
[<ffffffff8012a719>]{do_softirq+153} [<ffffffff80113323>]{do_IRQ+339}
[<ffffffff801106d7>]{common_interrupt+95} <EOI> [<ffffffff80267cf0>]{ip
[<ffffffff80249e55>]{dev_queue_xmit+453} [<ffffffff801f653d>]{__make_req
[<ffffffff801f64c7>]{__make_request+1159} [<ffffffff801f669b>]{generic_m
[<ffffffff801f6711>]{submit_bh_rsector+97} [<ffffffff8015eede>]{write_lo
[<ffffffff8015f064>]{write_some_buffers+372} [<ffffffff801248c9>]{printk
[<ffffffff8015f097>]{write_unlocked_buffers+23} [<ffffffff8015f1ae>]{syn
[<ffffffff8015f31a>]{fsync_dev+10} [<ffffffff8015f45b>]{sys_sync+11}
[<ffffffff80123d7e>]{panic+286} [<ffffffff8011129a>]{show_trace+666}
[<ffffffff801113bd>]{show_stack+205} [<ffffffff80111500>]{show_registers
[<ffffffff801116ac>]{die+268} [<ffffffff801a526d>]{do_page_fault+989}
[<ffffffff8027f492>]{tcp_v4_rcv+1330} [<ffffffff80262d70>]{ip_local_deli
[<ffffffff80262e64>]{ip_local_deliver_finish+244} [<ffffffff80252e51>]{n
[<ffffffff80262d70>]{ip_local_deliver_finish+0} [<ffffffff801109d6>]{err
[<ffffffff802b5dd2>]{memcpy+18} [<ffffffff8010ed13>]{__switch_to+499}
[<ffffffff8011f8c2>]{thread_return+0} [<ffffffff80101000>]{init_level4_p
[<ffffffff8012f9b5>]{schedule_timeout+37} [<ffffffff802b5915>]{do_softir
[<ffffffff8028e0df>]{inet_wait_for_connect+287} [<ffffffff8028e313>]{ine
[<ffffffffa0108f0c>]{:lock_gulm:.rodata.str1.1+583}
[<ffffffffa010676c>]{:lock_gulm:xdr_connect+28} [<ffffffffa01035a5>]{:lo
[<ffffffffa01000bf>]{:lock_gulm:lt_login+63} [<ffffffffa00fc184>]{:lock_
[<ffffffffa010ca00>]{:lock_gulm:core_cb+0} [<ffffffffa0102202>]{:lock_gu
[<ffffffffa010284a>]{:lock_gulm:lg_core_login+346}
[<ffffffffa00fc568>]{:lock_gulm:cm_login+136} [<ffffffffa00fcc36>]{:lock
[<ffffffffa00fcfa9>]{:lock_gulm:gulm_mount+665} [<ffffffffa0128980>]{:gf
[<ffffffffa00fb3e3>]{:lock_harness:lm_mount_Rsmp_ad6c5c21+355}
[<ffffffffa0128980>]{:gfs:gfs_glock_cb+0} [<ffffffffa012e09e>]{:gfs:gfs_
[<ffffffff8013d8d2>]{do_anonymous_page+1234} [<ffffffff8013d94f>]{do_no_
[<ffffffff801a5103>]{do_page_fault+627} [<ffffffff801109d6>]{error_exit+
[<ffffffff801ebbe9>]{serial_in+41} [<ffffffff8011e20d>]{wake_up_cpu+29}
[<ffffffffa011939a>]{:gfs:gfs_read_super+1338} [<ffffffffa014dca0>]{:gfs
[<ffffffff80164c0c>]{get_sb_bdev+588} [<ffffffffa014dca0>]{:gfs:gfs_fs_t
[<ffffffff80164ec9>]{do_kern_mount+121} [<ffffffff8017baa1>]{do_add_moun
[<ffffffff8017bdb9>]{do_mount+345} [<ffffffff80154b40>]{__get_free_pages
[<ffffffff8017c1d5>]{sys_mount+197} [<ffffffff80110177>]{system_call+119
Process mount (pid: 4026, stackpage=100775fb000)
Stack: ffffffff805de5c0 0000000000000018 0000000000100000 0000000000000000
00000100079c4c80 ffffffff803e89a0 0000000000000000 00000100000fdea0
ffffffff803e8d00 00000100079bf000 00000100079d6400 0000000000000042
00000100079de280 ffffff0000000000 000000fffffff000 0000000000000000
00000100079d7a80 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
00000100775fbc28 0000000000000000 00000000006d9994 0000000000000003
0000000000000000 0000000000000000 0000000100000000 ffffffffffffffff
ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff
ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff
Call Trace: <EOE> [<ffffffff8011b263>]{smp_apic_timer_interrupt+291}
[<ffffffff801108dc>]{apic_timer_interrupt+64} [<ffffffff8011302e>]{handl
[<ffffffff801132e2>]{do_IRQ+274} [<ffffffff801106d7>]{common_interrupt+9
[<ffffffff8012a719>]{do_softirq+153} [<ffffffff80113323>]{do_IRQ+339}
[<ffffffff801106d7>]{common_interrupt+95} <EOI> [<ffffffff80267cf0>]{ip
[<ffffffff80249e55>]{dev_queue_xmit+453} [<ffffffff801f653d>]{__make_req
[<ffffffff801f64c7>]{__make_request+1159} [<ffffffff801f669b>]{generic_m
[<ffffffff801f6711>]{submit_bh_rsector+97} [<ffffffff8015eede>]{write_lo
[<ffffffff8015f064>]{write_some_buffers+372} [<ffffffff801248c9>]{printk
[<ffffffff8015f097>]{write_unlocked_buffers+23} [<ffffffff8015f1ae>]{syn
[<ffffffff8015f31a>]{fsync_dev+10} [<ffffffff8015f45b>]{sys_sync+11}
[<ffffffff80123d7e>]{panic+286} [<ffffffff8011129a>]{show_trace+666}
[<ffffffff801113bd>]{show_stack+205} [<ffffffff80111500>]{show_registers
[<ffffffff801116ac>]{die+268} [<ffffffff801a526d>]{do_page_fault+989}
[<ffffffff8027f492>]{tcp_v4_rcv+1330} [<ffffffff80262d70>]{ip_local_deli
[<ffffffff80262e64>]{ip_local_deliver_finish+244} [<ffffffff80252e51>]{n
[<ffffffff80262d70>]{ip_local_deliver_finish+0} [<ffffffff801109d6>]{err
[<ffffffff802b5dd2>]{memcpy+18} [<ffffffff8010ed13>]{__switch_to+499}
[<ffffffff8011f8c2>]{thread_return+0} [<ffffffff80101000>]{init_level4_p
[<ffffffff8012f9b5>]{schedule_timeout+37} [<ffffffff802b5915>]{do_softir
[<ffffffff8028e0df>]{inet_wait_for_connect+287} [<ffffffff8028e313>]{ine
[<ffffffffa0108f0c>]{:lock_gulm:.rodata.str1.1+583}
[<ffffffffa010676c>]{:lock_gulm:xdr_connect+28} [<ffffffffa01035a5>]{:lo
[<ffffffffa01000bf>]{:lock_gulm:lt_login+63} [<ffffffffa00fc184>]{:lock_
[<ffffffffa010ca00>]{:lock_gulm:core_cb+0} [<ffffffffa0102202>]{:lock_gu
[<ffffffffa010284a>]{:lock_gulm:lg_core_login+346}
[<ffffffffa00fc568>]{:lock_gulm:cm_login+136} [<ffffffffa00fcc36>]{:lock
[<ffffffffa00fcfa9>]{:lock_gulm:gulm_mount+665} [<ffffffffa0128980>]{:gf
[<ffffffffa00fb3e3>]{:lock_harness:lm_mount_Rsmp_ad6c5c21+355}
[<ffffffffa0128980>]{:gfs:gfs_glock_cb+0} [<ffffffffa012e09e>]{:gfs:gfs_
[<ffffffff8013d8d2>]{do_anonymous_page+1234} [<ffffffff8013d94f>]{do_no_
[<ffffffff801a5103>]{do_page_fault+627} [<ffffffff801109d6>]{error_exit+
[<ffffffff801ebbe9>]{serial_in+41} [<ffffffff8011e20d>]{wake_up_cpu+29}
[<ffffffffa011939a>]{:gfs:gfs_read_super+1338} [<ffffffffa014dca0>]{:gfs
[<ffffffff80164c0c>]{get_sb_bdev+588} [<ffffffffa014dca0>]{:gfs:gfs_fs_t
[<ffffffff80164ec9>]{do_kern_mount+121} [<ffffffff8017baa1>]{do_add_moun
[<ffffffff8017bdb9>]{do_mount+345} [<ffffffff80154b40>]{__get_free_pages
[<ffffffff8017c1d5>]{sys_mount+197} [<ffffffff80110177>]{system_call+119
Code: f3 90 7e f7 e9 0b db ff ff 80 ba c0 02 5e 80 00 f3 90 7e f5
console shuts up ...
-------------- next part --------------
Lock_Harness v6.0.0 (built Aug 6 2004 20:27:11) installed
Lock_Nolock v6.0.0 (built Aug 6 2004 20:27:12) installed
GFS v6.0.0 (built Aug 6 2004 20:26:48) installed
->gfs_read_super(78a4d000, 0, 0)
->gfs_mount_lockproto({proto="", table="", host=""}, 0)
Gulm v6.0.0 (built Aug 5 2004 16:27:11) installed
Unable to handle kernel NULL pointer dereference at virtual address 000000000000
printing rip:
ffffffff8024a875
PML4 77ae7067 PGD 7798c067 PMD 0
Oops: 0002
CPU 0
Pid: 4027, comm: mount Not tainted
RIP: 0010:[<ffffffff8024a875>]{net_rx_action+213}
RSP: 0018:0000010077605048 EFLAGS: 00010046
RAX: 0000000000000000 RBX: ffffffff806077e8 RCX: ffffffff80607988
RDX: ffffffff806077e8 RSI: 0000010077b27080 RDI: ffffffff806077d0
RBP: ffffffff80607668 R08: 0000000080a56a9c R09: 0000000000a580a5
R10: 000000000100007f R11: 0000000000000000 R12: ffffffff806077e8
R13: ffffffff806077c0 R14: 00000000000046e6 R15: 0000000000000000
FS: 0000002a955764c0(0000) GS:ffffffff805d9840(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000000101000 CR4: 00000000000006e0
Call Trace: [<ffffffff8024a84d>]{net_rx_action+173}
[<ffffffff8012a72e>]{do_softirq+174} [<ffffffff80267cf0>]{ip_finish_outp
[<ffffffff80267cc0>]{dst_output+0} [<ffffffff802b5915>]{do_softirq_thunk
[<ffffffff802533a7>]{.text.lock.netfilter+165} [<ffffffff80267cc0>]{dst_
[<ffffffff80265fbb>]{ip_queue_xmit+1019} [<ffffffff80262ee0>]{ip_rcv_fin
[<ffffffff802630f0>]{ip_rcv_finish+528} [<ffffffff80252e51>]{nf_hook_slo
[<ffffffff80262ee0>]{ip_rcv_finish+0} [<ffffffff80277faf>]{tcp_transmit_
[<ffffffff80278ac6>]{tcp_write_xmit+198} [<ffffffff8026de83>]{tcp_sendms
[<ffffffff8028e795>]{inet_sendmsg+69} [<ffffffff802407ae>]{sock_sendmsg+
[<ffffffffa01484b1>]{:lock_gulm:do_tfer+369} [<ffffffffa014abd4>]{:lock_
[<ffffffffa0148595>]{:lock_gulm:xdr_send+37} [<ffffffffa0147498>]{:lock_
[<ffffffffa014551d>]{:lock_gulm:lg_lock_login+301}
[<ffffffffa0141ff9>]{:lock_gulm:lt_login+57} [<ffffffffa013e164>]{:lock_
[<ffffffffa014e6a0>]{:lock_gulm:core_cb+0} [<ffffffffa01440eb>]{:lock_gu
[<ffffffffa0144713>]{:lock_gulm:lg_core_login+323}
[<ffffffffa013e53a>]{:lock_gulm:cm_login+122} [<ffffffffa013ebde>]{:lock
[<ffffffffa013ef08>]{:lock_gulm:gulm_mount+616} [<ffffffffa0117980>]{:gf
[<ffffffff801277bb>]{release_task+763} [<ffffffffa00fb3e3>]{:lock_harnes
[<ffffffffa0117980>]{:gfs:gfs_glock_cb+0} [<ffffffffa011d09e>]{:gfs:gfs_
[<ffffffff8013d8d2>]{do_anonymous_page+1234} [<ffffffff8013d94f>]{do_no_
[<ffffffff801a5103>]{do_page_fault+627} [<ffffffff801109d6>]{error_exit+
[<ffffffff801ebbe9>]{serial_in+41} [<ffffffff8011e20d>]{wake_up_cpu+29}
[<ffffffffa010839a>]{:gfs:gfs_read_super+1338} [<ffffffffa013cca0>]{:gfs
[<ffffffff80164c0c>]{get_sb_bdev+588} [<ffffffffa013cca0>]{:gfs:gfs_fs_t
[<ffffffff80164ec9>]{do_kern_mount+121} [<ffffffff8017baa1>]{do_add_moun
[<ffffffff8017bdb9>]{do_mount+345} [<ffffffff80154b40>]{__get_free_pages
[<ffffffff8017c1d5>]{sys_mount+197} [<ffffffff80110177>]{system_call+119
Process mount (pid: 4027, stackpage=10077605000)
Stack: 0000010077605048 0000000000000018 ffffffff8024a84d 0000012a80445d20
0000000000000001 ffffffff80606c60 0000000000000000 000000000000000a
0000000000000000 0000000000000002 ffffffff8012a72e ffffffff80267cf0
0000000000000246 0000000000000000 0000000000000003 ffffffff80445d20
ffffffff80267cc0 0000000000000000 ffffffff802b5915 0000000000000043
0000000000000006 000001007a56a09e 0000010077a32d80 0000000000000000
0000000000000000 ffffffff8049c648 0000000000000000 ffffffff806077c0
ffffffff802533a7 ffffffff80267cc0 ffffffff80445d20 0000000000000002
0000010077a32d80 ffffffff805abcd0 000001007a56a0ac 0000010077a32d80
0000010077b27080 0000000000000000 0000010077b27080 0000010077a32de8
Call Trace: [<ffffffff8024a84d>]{net_rx_action+173}
[<ffffffff8012a72e>]{do_softirq+174} [<ffffffff80267cf0>]{ip_finish_outp
[<ffffffff80267cc0>]{dst_output+0} [<ffffffff802b5915>]{do_softirq_thunk
[<ffffffff802533a7>]{.text.lock.netfilter+165} [<ffffffff80267cc0>]{dst_
[<ffffffff80265fbb>]{ip_queue_xmit+1019} [<ffffffff80262ee0>]{ip_rcv_fin
[<ffffffff802630f0>]{ip_rcv_finish+528} [<ffffffff80252e51>]{nf_hook_slo
[<ffffffff80262ee0>]{ip_rcv_finish+0} [<ffffffff80277faf>]{tcp_transmit_
[<ffffffff80278ac6>]{tcp_write_xmit+198} [<ffffffff8026de83>]{tcp_sendms
[<ffffffff8028e795>]{inet_sendmsg+69} [<ffffffff802407ae>]{sock_sendmsg+
[<ffffffffa01484b1>]{:lock_gulm:do_tfer+369} [<ffffffffa014abd4>]{:lock_
[<ffffffffa0148595>]{:lock_gulm:xdr_send+37} [<ffffffffa0147498>]{:lock_
[<ffffffffa014551d>]{:lock_gulm:lg_lock_login+301}
[<ffffffffa0141ff9>]{:lock_gulm:lt_login+57} [<ffffffffa013e164>]{:lock_
[<ffffffffa014e6a0>]{:lock_gulm:core_cb+0} [<ffffffffa01440eb>]{:lock_gu
[<ffffffffa0144713>]{:lock_gulm:lg_core_login+323}
[<ffffffffa013e53a>]{:lock_gulm:cm_login+122} [<ffffffffa013ebde>]{:lock
[<ffffffffa013ef08>]{:lock_gulm:gulm_mount+616} [<ffffffffa0117980>]{:gf
[<ffffffff801277bb>]{release_task+763} [<ffffffffa00fb3e3>]{:lock_harnes
[<ffffffffa0117980>]{:gfs:gfs_glock_cb+0} [<ffffffffa011d09e>]{:gfs:gfs_
[<ffffffff8013d8d2>]{do_anonymous_page+1234} [<ffffffff8013d94f>]{do_no_
[<ffffffff801a5103>]{do_page_fault+627} [<ffffffff801109d6>]{error_exit+
[<ffffffff801ebbe9>]{serial_in+41} [<ffffffff8011e20d>]{wake_up_cpu+29}
[<ffffffffa010839a>]{:gfs:gfs_read_super+1338} [<ffffffffa013cca0>]{:gfs
[<ffffffff80164c0c>]{get_sb_bdev+588} [<ffffffffa013cca0>]{:gfs:gfs_fs_t
[<ffffffff80164ec9>]{do_kern_mount+121} [<ffffffff8017baa1>]{do_add_moun
[<ffffffff8017bdb9>]{do_mount+345} [<ffffffff80154b40>]{__get_free_pages
[<ffffffff8017c1d5>]{sys_mount+197} [<ffffffff80110177>]{system_call+119
Code: 48 89 18 48 89 43 08 8b 85 90 01 00 00 85 c0 79 08 03 85 94
Kernel panic: Fatal exception
In interrupt handler - not syncing
NMI Watchdog detected LOCKUP on CPU1, eip ffffffff801a5419, registers:
CPU 1
Pid: 3534, comm: lock_gulmd Not tainted
RIP: 0010:[<ffffffff801a5419>]{.text.lock.fault+7}
RSP: 0018:000001007adc1978 EFLAGS: 00000086
RAX: 000000000000000f RBX: ffffffff80607ae8 RCX: 0000000000000000
RDX: ffffffff803042e0 RSI: ffffffff803042e0 RDI: ffffffff8024a875
RBP: ffffffff80607968 R08: ffffffff803042d0 R09: 0000000000a580a5
R10: 000000000100007f R11: 0000000000000000 R12: 0000010007a0e9c0
R13: 0000000000000000 R14: 0000000000000002 R15: 000001007adc1a58
FS: 0000002a95576ce0(0000) GS:ffffffff805d98c0(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 00000000079d2000 CR4: 00000000000006e0
Call Trace:
Process lock_gulmd (pid: 3534, stackpage=1007adc1000)
Stack: 000001007adc1978 0000000000000018 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
Call Trace:
Code: f3 90 7e f5 e9 c8 fd ff ff 90 90 90 90 90 90 90 90 90 90 90
console shuts up ...
More information about the Linux-cluster
mailing list