[Linux-cluster] Adding third node to existing cluster problem

[Rajkumar, Anoop]

Hi

I have a two node cluster and I was adding third node to it but its
failing with below error message.

Jul 14 10:44:56 usrylxap235 dlm_controld[5472]: connect to ccs error
-111, check ccsd or cluster status
Jul 14 10:44:56 usrylxap235 ccsd[5758]: Initial status:: Inquorate
Jul 14 10:44:56 usrylxap235 openais[5764]: [TOTEM] HeartBeat is
Disabled. To enable set heartbeat_failures_allowed > 0
Jul 14 10:44:56 usrylxap235 ccsd[5758]: Cluster is not quorate.
Refusing connection.
Jul 14 10:44:56 usrylxap235 openais[5764]: [TOTEM] Receive multicast
socket recv buffer size (288000 bytes).
Jul 14 10:44:56 usrylxap235 ccsd[5758]: Error while processing connect:
Connection refused
Jul 14 10:44:56 usrylxap235 openais[5764]: [TOTEM] Transmit multicast
socket send buffer size (288000 bytes).
Jul 14 10:44:56 usrylxap235 openais[5764]: [TOTEM] The network interface
[192.168.0.7] is now up.
Jul 14 10:44:56 usrylxap235 gfs_controld[5478]: connect to ccs error
-111, check ccsd or cluster status
Jul 14 10:44:56 usrylxap235 openais[5764]: [TOTEM] Created or loaded
sequence id 68.192.168.0.7 for this ring.
Jul 14 10:44:56 usrylxap235 ccsd[5758]: Cluster is not quorate.
Refusing connection.
Jul 14 10:44:56 usrylxap235 openais[5764]: [TOTEM] entering GATHER state
from 15.
Jul 14 10:44:56 usrylxap235 ccsd[5758]: Error while processing connect:
Connection refused

Clustat output from other two nodes shows third node as offline.

[root at usrylxap237 ~]# clustat
Cluster Status for cluster1 @ Wed Jul 14 10:50:57 2010
Member Status: Quorate

 Member Name                             ID   Status
 ------ ----                             ---- ------
 usrylxap237-1.merck.com                     1 Online, Local, rgmanager
 usrylxap238-1.merck.com                     2 Online, rgmanager
 usrylxap235-1.merck.com                     3 Offline

 Service Name                   Owner (Last)                   State
 ------- ----                   ----- ------                   -----
 service:http-service           usrylxap237-1.merck.com        started
 service:mysql                  usrylxap237-1.merck.com        started


Clustat output from third node shows other two nodes offline.

[root at usrylxap235 sysconfig]# clustat
Cluster Status for cluster1 @ Wed Jul 14 10:52:47 2010
Member Status: Inquorate

 Member Name                             ID   Status
 ------ ----                             ---- ------
 usrylxap237-1.merck.com                     1 Offline
 usrylxap238-1.merck.com                     2 Offline
 usrylxap235-1.merck.com                     3 Online, Local


I will appreciate any help to resolv this.

Thanks
Anoop

Notice:  This e-mail message, together with any attachments, contains
information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station,
New Jersey, USA 08889), and/or its affiliates Direct contact information
for affiliates is available at 
http://www.merck.com/contact/contacts.html) that may be confidential,
proprietary copyrighted and/or legally privileged. It is intended solely
for the use of the individual or entity named on this message. If you are
not the intended recipient, and have received this message in error,
please notify us immediately by reply e-mail and then delete it from 
your system.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100714/d99e8086/attachment.htm>