[Linux-cluster] another cman/sm problem

[Daniel McNeil]

cl030:
CMAN: node cl031a is not responding - removing from the cluster
dlm: closing connection to node 1
dlm: closing connection to node 2
SM: 00000001 sm_stop: SG still joined
SM: 01000932 sm_stop: SG still joined
SM: 02000933 sm_stop: SG still joined
Unable to handle kernel NULL pointer dereference at virtual address 00000004
 printing eip:
c0119677
*pde = 00000000
Oops: 0000 [#1]
PREEMPT SMP
Modules linked in: lock_nolock lock_dlm dlm qla2200 qla2xxx gfs lock_harness cman dm_mod
CPU:    1
EIP:    0060:[<c0119677>]    Not tainted VLI
EFLAGS: 00010096   (2.6.9)
EIP is at task_rq_lock+0x27/0x70
eax: ea000f2c   ebx: c052e000   ecx: 00000000   edx: 00000000
esi: c0533020   edi: c052e000   ebp: ea000ef4   esp: ea000ee8
ds: 007b   es: 007b   ss: 0068
Process cman_comms (pid: 3739, threadinfo=ea000000 task=e9de98f0)
Stack: c1b037b0 e617daf4 f781243c ea000f3c c0119d92 00000000 ea000f2c c014a82f
       c181f040 00000020 f7810750 f8d04755 ffffff95 02000933 00000000 ea000f50
       f8d047b5 00000296 c1b037b0 e617daf4 f781243c ea000f50 c011a02e 00000000
Call Trace:
 [<c010626f>] show_stack+0x7f/0xa0
 [<c010641e>] show_registers+0x15e/0x1d0
 [<c010663e>] die+0xfe/0x190
 [<c0118683>] do_page_fault+0x293/0x7c1
 [<c0105e59>] error_code+0x2d/0x38
 [<c0119d92>] try_to_wake_up+0x22/0x2a0
 [<c011a02e>] wake_up_process+0x1e/0x30
 [<f8d048b0>] callback_startdone_barrier+0x20/0x30 [cman]
 [<f8cfc641>] node_shutdown+0x291/0x3c0 [cman]
 [<f8cf847a>] cluster_kthread+0x2aa/0x350 [cman]
 [<c0103325>] kernel_thread_helper+0x5/0x10
SM: 00000001 sm_stop: SG still joined
SM: 01000932 sm_stop: SG still joined
SM: 02000933 sm_stop: SG still joined
Unable to handle kernel NULL pointer dereference at virtual address 00000004
 printing eip:
c0119677
*pde = 00000000
Oops: 0000 [#1]
PREEMPT SMP

cl031:
dlm: closing connection to node 1
dlm: closing connection to node 2
SM: 00000001 sm_stop: SG still joined
SM: 01000932 sm_stop: SG still joined
SM: 02000933 sm_stop: SG still joined
Unable to handle kernel NULL pointer dereference at virtual address 00000004
 printing eip:
c0119677
*pde = 00000000
Oops: 0000 [#1]
PREEMPT SMP
Modules linked in: lock_dlm dlm qla2200 qla2xxx gfs lock_harness cman dm_mod
CPU:    1
EIP:    0060:[<c0119677>]    Not tainted VLI
EFLAGS: 00010096   (2.6.9)
EIP is at task_rq_lock+0x27/0x70
eax: eacc0f2c   ebx: c052e000   ecx: 00000000   edx: 00000000
esi: c0533020   edi: c052e000   ebp: eacc0ef4   esp: eacc0ee8
ds: 007b   es: 007b   ss: 0068
Process cman_comms (pid: 2876, threadinfo=eacc0000 task=eae75a30)
Stack: ea3a285c caeb7da4 f502e1a8 eacc0f3c c0119d92 00000000 eacc0f2c c014a82f
       c181f040 00000020 f7d35f38 f8d04755 ffffff95 02000933 00000000 eacc0f50
       f8d047b5 00000296 ea3a285c caeb7da4 f502e1a8 eacc0f50 c011a02e 00000000
Call Trace:
 [<c010626f>] show_stack+0x7f/0xa0
 [<c010641e>] show_registers+0x15e/0x1d0
 [<c010663e>] die+0xfe/0x190
 [<c0118683>] do_page_fault+0x293/0x7c1
 [<c0105e59>] error_code+0x2d/0x38
 [<c0119d92>] try_to_wake_up+0x22/0x2a0
 [<c011a02e>] wake_up_process+0x1e/0x30
 [<f8d04880>] callback_startdone_barrier_new+0x20/0x30 [cman]
 [<f8cfc641>] node_shutdown+0x291/0x3c0 [cman]
 [<f8cf847a>] cluster_kthread+0x2aa/0x350 [cman]
 [<c0103325>] kernel_thread_helper+0x5/0x10
Code: 00 00 00 00 55 89 e5 83 ec 0c 89 1c 24 89 74 24 04 89 7c 24 08
 8b 45 0c 9c 8f 00 fa be 20 30 53 c0 bb 00 e0 52 c0 8b 55 08 89 df
 <8b> 42 04 8b 40 10 8b 0c

cl032:
Dec 22 05:29:19 cl032 sshd(pam_unix)[18296]: session closed for user root
Dec 22 05:42:14 cl032 kernel: CMAN: bad generation number 15 in HELLO message, expected 14
Dec 22 05:42:17 cl032 kernel: CMAN: Node cl030a is leaving the cluster, ShutdownDec 22 05:42:17 cl032 kernel: CMAN: quorum lost, blocking activity

My test is doing a lot of mounting and umounting.  Wouldn't that
stress SM code a lot.  Is SM causing the problem?
http://developer.osdl.org/daniel/GFS/cman.21dec2004/

Daniel