[Linux-cluster] Kernel Panic (cman) after reboot

Patrick Caulfield pcaulfie at redhat.com
Tue May 23 07:18:22 UTC 2006 

thorsten.henrici at gfd.de wrote:
> 
> Hello,
> I'm experiencing the following reproducable problem:
> 
> I have two nodes A and B. I reboot node B and get the following syslog
> on node B:
> 
> Shutdown Node B - Syslog
> ----------------------------------------------------------------------
> May 20 13:27:24 sdhhdewer38b shutdown: shutting down for system reboot
> May 20 13:27:24 sdhhdewer38b init: Switching to runlevel: 6
> (...)
> May 20 13:27:47 sdhhdewer38b rgmanager: [24641]: <notice> Shutting down
> Cluster Service Manager...
> May 20 13:27:47 sdhhdewer38b clurgmgrd[31332]: <notice> Shutting down
> May 20 13:27:47 sdhhdewer38b clurgmgrd[31332]: <notice> Stopping service
> s_ndb_mgmd_ip
> May 20 13:27:47 sdhhdewer38b clurgmgrd: [31332]: <info> Removing IPv4
> address 10.112.24.20 from eth0
> May 20 13:27:57 sdhhdewer38b clurgmgrd[31332]: <notice> Service
> s_ndb_mgmd_ip is stopped
> May 20 13:27:59 sdhhdewer38b clurgmgrd[31332]: <notice> Shutdown
> complete, exiting
> May 20 13:28:00 sdhhdewer38b rgmanager: [24641]: <notice> Cluster
> Service Manager is stopped.
> (...)
> May 20 13:28:21 sdhhdewer38b fenced: Stopping fence domain:
> May 20 13:28:21 sdhhdewer38b fenced: shutdown succeeded
> May 20 13:28:21 sdhhdewer38b fenced:
> May 20 13:28:21 sdhhdewer38b fenced:
> May 20 13:28:21 sdhhdewer38b rc: Stopping fenced:  succeeded
> May 20 13:28:21 sdhhdewer38b lock_gulmd: Stopping lock_gulmd:
> May 20 13:28:21 sdhhdewer38b lock_gulmd: shutdown succeeded
> May 20 13:28:21 sdhhdewer38b lock_gulmd:                   [  
> May 20 13:28:21 sdhhdewer38b lock_gulmd:
> May 20 13:28:21 sdhhdewer38b rc: Stopping lock_gulmd:  succeeded
> May 20 13:28:21 sdhhdewer38b cman: Stopping cman:
> May 20 13:28:24 sdhhdewer38b cman: failed to stop cman failed
> May 20 13:28:24 sdhhdewer38b cman:                         [
> May 20 13:28:24 sdhhdewer38b cman:
> May 20 13:28:24 sdhhdewer38b rc: Stopping cman:  failed
> May 20 13:28:24 sdhhdewer38b ccsd: Stopping ccsd:
> May 20 13:28:24 sdhhdewer38b ccsd[2564]: Stopping ccsd, SIGTERM received.
> May 20 13:28:25 sdhhdewer38b ccsd: shutdown succeeded
> May 20 13:28:25 sdhhdewer38b ccsd:
> May 20 13:28:25 sdhhdewer38b ccsd:
> May 20 13:28:25 sdhhdewer38b rc: Stopping ccsd:  succeeded
> ----------------------------------------------------------
> 
> Rebooting Node B crashes (kernel panic) Node B while starting the cman
> service (loading the cman module)
> That being already prettey bad, it even becomes worse. Node A leaves the
> cluster - which brings all services running on it to a halt.
> 
> I assume, that this behavior won't occur if I manually remove node B
> from the cluster before rebooting. (I haven't tested yet, but will do as
> soon as I have the chance to).
> Nevertheless I think this behavior is a much too risky thing to have in
> a production environment. Is this already known and is there any save
> way to fix this?
> 
> Syslogs of Node A and B during reboot:
> 
> I'm runnig a self-written daemon-process that checks
> /proc/cluster/status for the node's membership state. If the cluster
> hasn't the status 'Member' for more than 60 seconds, I'm halting the
> system to get into a consistent state. Call it something like self-fencing.
> 
> Startup Node B - Syslog
> --------------------------------------------------------------------------------------
> 
> May 20 13:34:19 sdhhdewer38b ccsd[2633]: Connected to cluster
> infrastruture via: CMAN/SM Plugin v1.1.5
> May 20 13:34:19 sdhhdewer38b ccsd[2633]: Initial status:: Inquorate
> May 20 13:34:19 sdhhdewer38b kernel: CMAN: sending membership request
> May 20 13:34:19 sdhhdewer38b kernel: CMAN: sending membership request
> May 20 13:34:19 sdhhdewer38b kernel: CMAN: got node sdhhdewer38a
> May 20 13:36:13 sdhhdewer38b kernel: CMAN: removing node sdhhdewer38a
> from the cluster : No response to messages
> May 20 13:36:13 sdhhdewer38b kernel: ------------[ cut here ]------------
> May 20 13:36:13 sdhhdewer38b kernel: kernel BUG at
> /usr/src/build/714635-i686/BUILD/cman-kernel-2.6.9-43/smp/src/membership.c:3150!
> 
> May 20 13:36:13 sdhhdewer38b kernel: invalid operand: 0000 [#1]
> May 20 13:36:13 sdhhdewer38b kernel: SMP
> May 20 13:36:13 sdhhdewer38b kernel: Modules linked in: cman(U) sunrpc
> md5 ipv6 dm_multipath button battery ac uhci_hcd ehci_hcd hw_random
> bcm5700(U) floppy dm_snapshot dm_zero d
> m_mirror ext3 jbd dm_mod qla6312(U) qla2400(U) qla2300(U) qla2xxx(U)
> qla2xxx_conf(U) cciss sd_mod scsi_mod
> May 20 13:36:13 sdhhdewer38b kernel: CPU:    2
> May 20 13:36:13 sdhhdewer38b kernel: EIP:    0060:[<f8ae1e2a>]    Not
> tainted VLI
> May 20 13:36:13 sdhhdewer38b kernel: EFLAGS: 00010246   (2.6.9-34.ELsmp)
> May 20 13:36:13 sdhhdewer38b kernel: EIP is at elect_master+0x2e/0x3a
> [cman]
> May 20 13:36:13 sdhhdewer38b kernel: eax: 00000000   ebx: f77c7fa0  
> ecx: 00000080   edx: 00000080
> May 20 13:36:13 sdhhdewer38b kernel: esi: f8af5044   edi: f77c7fd8  
> ebp: 00000000   esp: f77c7f98
> May 20 13:36:13 sdhhdewer38b kernel: ds: 007b   es: 007b   ss: 0068
> May 20 13:36:13 sdhhdewer38b kernel: Process cman_memb (pid: 2658,
> threadinfo=f77c7000 task=f7638730)
> May 20 13:36:13 sdhhdewer38b kernel: Stack: f8af4f08 f8adf8d1 c364eb00
> f6b23320 f8addeb7 f7638730 f7638730 f8ade09a
> May 20 13:36:13 sdhhdewer38b kernel:        0000001f 00000000 f705e6b0
> 00000000 f7638730 c011e71b 00100100 00200200
> May 20 13:36:13 sdhhdewer38b kernel:        00000000 00000000 0000007b
> f8added8 00000000 00000000 c01041f5 00000000
> May 20 13:36:13 sdhhdewer38b kernel: Call Trace:
> May 20 13:36:13 sdhhdewer38b kernel:  [<f8adf8d1>]
> a_node_just_died+0x13a/0x199 [cman]
> May 20 13:36:13 sdhhdewer38b kernel:  [<f8addeb7>]
> process_dead_nodes+0x4e/0x6f [cman]
> May 20 13:36:13 sdhhdewer38b kernel:  [<f8ade09a>]
> membership_kthread+0x1c2/0x39d [cman]
> May 20 13:36:13 sdhhdewer38b kernel:  [<c011e71b>]
> default_wake_function+0x0/0xc
> May 20 13:36:13 sdhhdewer38b kernel:  [<f8added8>]
> membership_kthread+0x0/0x39d [cman]
> May 20 13:36:13 sdhhdewer38b kernel:  [<c01041f5>]
> kernel_thread_helper+0x5/0xb
> May 20 13:36:13 sdhhdewer38b kernel: Code: 28 5e af f8 89 c3 ba 01 00 00
> 00 39 ca 7d 1c a1 2c 5e af f8 8b 04 90 85 c0 74 0d 83 78 1c 02 75 07 89
> 03 8b 40 14 eb 0d 42 eb e0 <0f> 0
> b 4e 0c 73 8d ae f8 31 c0 5b c3 a1 2c 5e af f8 e8 79 10 67
> May 20 13:36:13 sdhhdewer38b kernel:  <0>Fatal exception: panic in 5
> seconds
> May 20 13:36:18 sdhhdewer38b cman: Timed-out waiting for cluster failed
> May 20 13:36:18 sdhhdewer38b lock_gulmd: no <gulm> section detected in
> /etc/cluster/cluster.conf succeeded
> May 20 13:39:26 sdhhdewer38b syslogd 1.4.1: restart.
> --------------------------------------------------------------------------------------
> 
> 
> Node A - Syslog
> --------------------------------------------------------------------------------------
> 
> May 20 13:27:57 sdhhdewer38a clurgmgrd[31341]: <info> Magma Event:
> Membership Change
> May 20 13:27:57 sdhhdewer38a clurgmgrd[31341]: <info> State change:
> sdhhdewer38b DOWN
> May 20 13:28:00 sdhhdewer38a clurgmgrd[31341]: <notice> Starting stopped
> service s_ndb_mgmd_ip
> May 20 13:28:00 sdhhdewer38a clurgmgrd: [31341]: <info> Adding IPv4
> address 10.112.24.20 to eth0
> May 20 13:28:01 sdhhdewer38a clurgmgrd[31341]: <notice> Service
> s_ndb_mgmd_ip started
> May 20 13:34:19 sdhhdewer38a kernel: CMAN: node sdhhdewer38b rejoining
> May 20 13:34:27 sdhhdewer38a logger: Node is not a member of the
> Cluster. Membership state: Transition-Master
> May 20 13:34:27 sdhhdewer38a logger: Node will be shut down in 50 seconds
> May 20 13:34:37 sdhhdewer38a logger: Node is not a member of the
> Cluster. Membership state: Transition-Master
> May 20 13:34:37 sdhhdewer38a logger: Node will be shut down in 40 seconds
> May 20 13:34:40 sdhhdewer38a clurgmgrd: [31341]: <info> Executing
> /etc/init.d/httpd status
> May 20 13:34:47 sdhhdewer38a logger: Node is not a member of the
> Cluster. Membership state: Transition-Master
> May 20 13:34:47 sdhhdewer38a logger: Node will be shut down in 30 seconds
> May 20 13:34:57 sdhhdewer38a logger: Node is not a member of the
> Cluster. Membership state: Transition-Master
> May 20 13:34:57 sdhhdewer38a logger: Node will be shut down in 20 seconds
> May 20 13:35:07 sdhhdewer38a logger: Node is not a member of the
> Cluster. Membership state: Transition-Master
> May 20 13:35:07 sdhhdewer38a logger: Node will be shut down in 10 seconds
> May 20 13:35:11 sdhhdewer38a clurgmgrd: [31341]: <info> Executing
> /etc/init.d/httpd status
> May 20 13:35:18 sdhhdewer38a logger: Node is not a member of the
> Cluster. Membership state: Transition-Master
> May 20 13:35:18 sdhhdewer38a logger: Node will be shut down in 0 seconds
> May 20 13:35:18 sdhhdewer38a logger: sdhhdewer38a is currently not a
> cluster member. Shutting down to get into a consistent state !
> May 20 13:35:18 sdhhdewer38a logger: Killing the following processes
> before shutdown: 31341  2742  2745  2743  2744
> May 20 13:35:18 sdhhdewer38a shutdown: shutting down for system reboot
> --------------------------------------------------------------------------------------
>

This is in bugzilla:

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=187777

a fix is in CVS.

-- 

patrick

More information about the Linux-cluster mailing list