[Linux-cluster] CMAN: got WAIT barrier not in phase 1 TRANSITION.96 (2)

[Tom Mornini]

Hello all.

We're getting problems when adding cluster nodes to our cluster.

Everything has been reasonable and stable until this point.

I posted these logs earlier with a different subject. I had
misunderstood the warning message and posted the wrong error
message in the subject.

Here's the syslog with a great deal of context:

Oct 13 04:08:53 ey00-s00017 kernel: VFS: Mounted root (reiserfs  
filesystem) read only.
Oct 13 04:08:53 ey00-s00017 kernel: Adding 262136k swap on /dev/ 
sda2.  Priority: -1 extents:1 across:262136k
Oct 13 04:09:04 ey00-s00017 kernel: CMAN 1.03.00 (built Sep  8 2006  
03:49:59) installed
Oct 13 04:09:04 ey00-s00017 kernel: NET: Registered protocol family 30
Oct 13 04:09:04 ey00-s00017 kernel: CMAN: Waiting to join or form a  
Linux-cluster
Oct 13 04:09:05 ey00-s00017 kernel: CMAN: sending membership request
Oct 13 04:09:06 ey00-s00017 kernel: CMAN: got node ey00-s00025
Oct 13 04:09:06 ey00-s00017 kernel: CMAN: got node ey00-s00019
Oct 13 04:09:06 ey00-s00017 kernel: CMAN: got node ey00-s00030
Oct 13 04:09:06 ey00-s00017 kernel: CMAN: got node ey00-s00024
Oct 13 04:09:06 ey00-s00017 kernel: CMAN: got node ey00-s00010
Oct 13 04:09:06 ey00-s00017 kernel: CMAN: got node ey00-s00016
Oct 13 04:09:06 ey00-s00017 kernel: CMAN: got node ey00-s00004
Oct 13 04:09:06 ey00-s00017 kernel: CMAN: got node ey00-s00011
Oct 13 04:09:06 ey00-s00017 kernel: CMAN: got node ey00-s00005
Oct 13 04:09:06 ey00-s00017 kernel: CMAN: got node ey00-s00009
Oct 13 04:09:06 ey00-s00017 kernel: CMAN: got node ey00-s00002
Oct 13 04:09:06 ey00-s00017 kernel: CMAN: got node ey00-s00015
Oct 13 04:09:06 ey00-s00017 kernel: CMAN: got node ey00-s00014
Oct 13 04:09:06 ey00-s00017 kernel: CMAN: got node ey00-s00008
Oct 13 04:09:06 ey00-s00017 kernel: CMAN: got node ey00-s00003
Oct 13 04:09:06 ey00-s00017 kernel: CMAN: got node ey00-s00006
Oct 13 04:09:06 ey00-s00017 kernel: CMAN: got node ey00-s00012
Oct 13 04:09:06 ey00-s00017 kernel: CMAN: got node ey00-s00013
Oct 13 04:09:06 ey00-s00017 kernel: CMAN: got node ey00-s00007
Oct 13 04:09:06 ey00-s00017 kernel: CMAN: got node ey00-s00001
Oct 13 04:09:06 ey00-s00017 kernel: CMAN: got node ey00-s00000
Oct 13 04:09:06 ey00-s00017 kernel: CMAN: got node ey00-04
Oct 13 04:09:06 ey00-s00017 kernel: CMAN: got node ey00-05
Oct 13 04:09:06 ey00-s00017 kernel: CMAN: got node ey00-03
Oct 13 04:09:06 ey00-s00017 kernel: CMAN: got node ey00-00
Oct 13 04:09:06 ey00-s00017 kernel: CMAN: got node ey00-01
Oct 13 04:09:06 ey00-s00017 kernel: CMAN: got node ey00-02
Oct 13 04:09:06 ey00-s00017 kernel: dlm: no version for  
"kcl_register_service" found: kernel tainted.
Oct 13 04:09:06 ey00-s00017 kernel: DLM 1.03.00 (built Sep  8 2006  
03:50:23) installed
Oct 13 04:09:57 ey00-s00017 kernel: CMAN: node ey00-s00018 rejoining
Oct 13 04:17:18 ey00-s00017 kernel: CMAN: got WAIT barrier not in  
phase 1 TRANSITION.96 (2)

Same thing on a different slice:

Oct 13 04:08:36 ey00-s00018 kernel: VFS: Mounted root (reiserfs  
filesystem) readonly.
Oct 13 04:08:36 ey00-s00018 kernel: Adding 262136k swap on /dev/ 
sda2.  Priority:-1 extents:1 across:262136k
Oct 13 04:08:48 ey00-s00018 kernel: CMAN 1.03.00 (built Sep  8 2006  
03:49:59) installed
Oct 13 04:08:48 ey00-s00018 kernel: NET: Registered protocol family 30
Oct 13 04:08:48 ey00-s00018 kernel: CMAN: Waiting to join or form a  
Linux-cluster
Oct 13 04:08:49 ey00-s00018 kernel: CMAN: sending membership request
Oct 13 04:08:50 ey00-s00018 kernel: CMAN: got node ey00-s00025
Oct 13 04:08:50 ey00-s00018 kernel: CMAN: got node ey00-s00019
Oct 13 04:08:50 ey00-s00018 kernel: CMAN: got node ey00-s00030
Oct 13 04:08:50 ey00-s00018 kernel: CMAN: got node ey00-s00024
Oct 13 04:08:50 ey00-s00018 kernel: CMAN: got node ey00-s00010
Oct 13 04:08:50 ey00-s00018 kernel: CMAN: got node ey00-s00016
Oct 13 04:08:50 ey00-s00018 kernel: CMAN: got node ey00-02
Oct 13 04:08:50 ey00-s00018 kernel: CMAN: got node ey00-01
Oct 13 04:08:50 ey00-s00018 kernel: CMAN: got node ey00-00
Oct 13 04:08:50 ey00-s00018 kernel: CMAN: got node ey00-03
Oct 13 04:08:50 ey00-s00018 kernel: CMAN: got node ey00-05
Oct 13 04:08:50 ey00-s00018 kernel: CMAN: got node ey00-04
Oct 13 04:08:50 ey00-s00018 kernel: CMAN: got node ey00-s00000
Oct 13 04:08:50 ey00-s00018 kernel: CMAN: got node ey00-s00001
Oct 13 04:08:50 ey00-s00018 kernel: CMAN: got node ey00-s00007
Oct 13 04:08:50 ey00-s00018 kernel: CMAN: got node ey00-s00013
Oct 13 04:08:50 ey00-s00018 kernel: CMAN: got node ey00-s00012
Oct 13 04:08:50 ey00-s00018 kernel: CMAN: got node ey00-s00006
Oct 13 04:08:50 ey00-s00018 kernel: CMAN: got node ey00-s00003
Oct 13 04:08:50 ey00-s00018 kernel: CMAN: got node ey00-s00008
Oct 13 04:08:50 ey00-s00018 kernel: CMAN: got node ey00-s00014
Oct 13 04:08:50 ey00-s00018 kernel: CMAN: got node ey00-s00015
Oct 13 04:08:50 ey00-s00018 kernel: CMAN: got node ey00-s00002
Oct 13 04:08:50 ey00-s00018 kernel: CMAN: got node ey00-s00009
Oct 13 04:08:50 ey00-s00018 kernel: CMAN: got node ey00-s00005
Oct 13 04:08:50 ey00-s00018 kernel: CMAN: got node ey00-s00011
Oct 13 04:08:50 ey00-s00018 kernel: CMAN: got node ey00-s00017
Oct 13 04:08:50 ey00-s00018 kernel: CMAN: got node ey00-s00004
Oct 13 04:08:50 ey00-s00018 kernel: dlm: no version for  
"kcl_register_service" found: kernel tainted.
Oct 13 04:08:50 ey00-s00018 kernel: DLM 1.03.00 (built Sep  8 2006  
03:50:23) installed
Oct 13 04:16:10 ey00-s00018 kernel: CMAN: got WAIT barrier not in  
phase 1 TRANSITION.96 (2)

-- 
-- Tom Mornini, CTO
-- Engine Yard, Ruby on Rails Hosting
-- Reliability, Ease of Use, Scalability
-- (866) 518-YARD (9273)