[Linux-cluster] cluster is not quorate. refusing connection

Fri Dec 10 23:05:21 UTC 2004

                                                                                                        On Fri, 2004-12-10 at 15:36, Jonathan E Brassow wrote:
> starting ccsd with the -v flag should print to the log the plugin it is 
> using to connect to the cluster infrastructure.  Something on the order 
> of:
> "Connected to cluster infrastructure via:  CMAN/SM Plugin v1.1"
> 
> This will at least tell you that ccsd is able to connect to the cluster 
> manager - and therefore, should know whether the cluster is quorate or 
> not.
> 
>   brassow
> 
> On Dec 10, 2004, at 4:16 PM, Matthew B. Brookover wrote:
> 
> > On Fri, 2004-12-10 at 00:17, David Teigland wrote:
> >> This sounds similar to a problem I have if I run fence_tool without 
> >> ccsd
> >> running.
> >>
> >> Check /proc/cluster/status while it's waiting to see if the cluster
> >> actually has quorum or not.  Also, I've added some extra checking and
> >> debugging to fence_tool that should help narrow down where things are
> >> stuck.  Please update from cvs and rebuild at least the stuff in
> >> cluster/fence; then use "fence_tool join -D".
> >>
> >> Usually things get stuck talking to ccs when ccs/magma libraries are 
> >> out
> >> of sync, but this case sounds different.
> >

I tried -v, and the cluster is quorate.  Only the fencing is not
starting up correctly.

Here are the logs with ccsd -v:
Added -v flag to ccsd, producing this log entry on fouroften:
Dec 10 15:50:00 fouroften kernel: Lock_Harness <CVS> (built Dec 10 2004
09:14:45) installed
Dec 10 15:50:00 fouroften kernel: GFS <CVS> (built Dec 10 2004 09:14:04)
installed
Dec 10 15:50:00 fouroften kernel: CMAN <CVS> (built Dec 10 2004
09:51:59) installed
Dec 10 15:50:00 fouroften kernel: NET: Registered protocol family 30
Dec 10 15:50:00 fouroften kernel: DLM <CVS> (built Dec 10 2004 09:52:25)
installed
Dec 10 15:50:00 fouroften kernel: Lock_DLM (built Dec 10 2004 09:14:25)
installed
Dec 10 15:50:00 fouroften ccsd[3379]: Starting ccsd DEVEL.1102700899:
Dec 10 15:50:01 fouroften ccsd[3379]:  Built: Dec 10 2004 10:50:57
Dec 10 15:50:01 fouroften ccsd[3379]:  Copyright (C) Red Hat, Inc. 
2004  All rights reserved.
Dec 10 15:50:01 fouroften ccsd[3379]:   Verbose Flag:: SET
Dec 10 15:50:01 fouroften ccsd[3379]: cluster.conf (cluster name =
CSMTEST, version = 6) found.
Dec 10 15:50:01 fouroften kernel: CMAN: Waiting to join or form a
Linux-cluster
Dec 10 15:50:01 fouroften crond(pam_unix)[3388]: session opened for user
root by (uid=0)
Dec 10 15:50:01 fouroften crond(pam_unix)[3387]: session opened for user
root by (uid=0)
Dec 10 15:50:02 fouroften ccsd[3379]: Connected to cluster infrastruture
via: CMAN/SM Plugin v1.1
Dec 10 15:50:02 fouroften ccsd[3379]: Initial status:: Inquorate
Dec 10 15:50:02 fouroften crond(pam_unix)[3388]: session closed for user
root
Dec 10 15:50:03 fouroften crond(pam_unix)[3387]: session closed for user
root
Dec 10 15:50:33 fouroften kernel: CMAN: forming a new cluster
Dec 10 15:50:33 fouroften kernel: CMAN: quorum regained, resuming
activity
Dec 10 15:50:33 fouroften kernel: CMAN: got node fiveoften

Logs on fiveoften:
Dec 10 15:50:07 fiveoften kernel: Lock_Harness <CVS> (built Dec 10 2004
09:14:45) installed
Dec 10 15:50:07 fiveoften kernel: GFS <CVS> (built Dec 10 2004 09:14:04)
installed
Dec 10 15:50:07 fiveoften crond(pam_unix)[3354]: session closed for user
root
Dec 10 15:50:07 fiveoften kernel: CMAN <CVS> (built Dec 10 2004
09:51:59) installed
Dec 10 15:50:07 fiveoften kernel: NET: Registered protocol family 30
Dec 10 15:50:07 fiveoften kernel: DLM <CVS> (built Dec 10 2004 09:52:25)
installed
Dec 10 15:50:07 fiveoften kernel: Lock_DLM (built Dec 10 2004 09:14:25)
installed
Dec 10 15:50:08 fiveoften ccsd[3381]: Starting ccsd DEVEL.1102700899:
Dec 10 15:50:08 fiveoften ccsd[3381]:  Built: Dec 10 2004 10:50:57
Dec 10 15:50:08 fiveoften ccsd[3381]:  Copyright (C) Red Hat, Inc. 
2004  All rights reserved.
Dec 10 15:50:08 fiveoften ccsd[3381]:   Verbose Flag:: SET
Dec 10 15:50:08 fiveoften ccsd[3381]: cluster.conf (cluster name =
CSMTEST, version = 6) found.
Dec 10 15:50:08 fiveoften kernel: CMAN: Waiting to join or form a
Linux-cluster
Dec 10 15:50:35 fiveoften kernel: CMAN: sending membership request
Dec 10 15:50:35 fiveoften kernel: CMAN: got node fouroften
Dec 10 15:50:35 fiveoften kernel: CMAN: quorum regained, resuming
activity
Dec 10 15:50:35 fiveoften ccsd[3381]: Cluster is not quorate.  Refusing
connection.
Dec 10 15:50:35 fiveoften ccsd[3381]: Error while processing connect:
Connection refused
Dec 10 15:50:36 fiveoften ccsd[3381]: Cluster is not quorate.  Refusing
connection.
Dec 10 15:50:36 fiveoften ccsd[3381]: Error while processing connect:
Connection refused
Dec 10 15:50:37 fiveoften ccsd[3381]: Cluster is not quorate.  Refusing
connection.
Dec 10 15:50:37 fiveoften ccsd[3381]: Error while processing connect:
Connection refused
Dec 10 15:50:38 fiveoften ccsd[3381]: Cluster is not quorate.  Refusing
connection.
Dec 10 15:50:38 fiveoften ccsd[3381]: Error while processing connect:
Connection refused
Dec 10 15:50:39 fiveoften ccsd[3381]: Cluster is not quorate.  Refusing
connection.
Dec 10 15:50:39 fiveoften ccsd[3381]: Error while processing connect:
Connection refused
Dec 10 15:50:40 fiveoften ccsd[3381]: Cluster is not quorate.  Refusing
connection.
Dec 10 15:50:40 fiveoften ccsd[3381]: Error while processing connect:
Connection refused
Dec 10 15:50:41 fiveoften ccsd[3381]: Cluster is not quorate.  Refusing
connection.
Dec 10 15:50:41 fiveoften ccsd[3381]: Error while processing connect:
Connection refused
Dec 10 15:50:42 fiveoften ccsd[3381]: Cluster is not quorate.  Refusing
connection.
Dec 10 15:50:42 fiveoften ccsd[3381]: Error while processing connect:
Connection refused
Dec 10 15:50:43 fiveoften ccsd[3381]: Cluster is not quorate.  Refusing
connection.
Dec 10 15:50:43 fiveoften ccsd[3381]: Error while processing connect:
Connection refused
Dec 10 15:50:44 fiveoften ccsd[3381]: Cluster is not quorate.  Refusing
connection.
Dec 10 15:50:44 fiveoften ccsd[3381]: Error while processing connect:
Connection refused
Dec 10 15:53:26 fiveoften ccsd[3381]: Connected to cluster infrastruture
via: CMAN/SM Plugin v1.1
Dec 10 15:53:26 fiveoften ccsd[3381]: Initial status:: Quorate

Matt
mbrookov at mines.edu