[Linux-cluster] cluster is not quorate. refusing connection

Fri Dec 10 23:50:41 UTC 2004

on fouroften, ccsd starts up and reports it's initial status 
(inquorate), this happens after it connects to the cluster 
infrastructure.  A few moments later, CMAN reports that the cluster is 
quorate.

On fiveoften, ccsd starts up, but doesn't report its initial status (or 
connect to the cluster infrastructure) until after the error messages - 
3min and 18sec after it has started up...  At this point, I'm not sure 
what the hold-up is.

Once ccsd has reported that it is connected to the cluster 
infrastructure, everything is fine right?

I'll try to find out what the hold up is.

  brassow

On Dec 10, 2004, at 5:05 PM, Matthew B. Brookover wrote:

>                                                                        
>                                  On Fri, 2004-12-10 at 15:36, Jonathan 
> E Brassow wrote:
>> starting ccsd with the -v flag should print to the log the plugin it 
>> is
>> using to connect to the cluster infrastructure.  Something on the 
>> order
>> of:
>> "Connected to cluster infrastructure via:  CMAN/SM Plugin v1.1"
>>
>> This will at least tell you that ccsd is able to connect to the 
>> cluster
>> manager - and therefore, should know whether the cluster is quorate or
>> not.
>>
>>   brassow
>>
>> On Dec 10, 2004, at 4:16 PM, Matthew B. Brookover wrote:
>>
>>> On Fri, 2004-12-10 at 00:17, David Teigland wrote:
>>>> This sounds similar to a problem I have if I run fence_tool without
>>>> ccsd
>>>> running.
>>>>
>>>> Check /proc/cluster/status while it's waiting to see if the cluster
>>>> actually has quorum or not.  Also, I've added some extra checking 
>>>> and
>>>> debugging to fence_tool that should help narrow down where things 
>>>> are
>>>> stuck.  Please update from cvs and rebuild at least the stuff in
>>>> cluster/fence; then use "fence_tool join -D".
>>>>
>>>> Usually things get stuck talking to ccs when ccs/magma libraries are
>>>> out
>>>> of sync, but this case sounds different.
>>>
>
> I tried -v, and the cluster is quorate.  Only the fencing is not
> starting up correctly.
>
> Here are the logs with ccsd -v:
> Added -v flag to ccsd, producing this log entry on fouroften:
> Dec 10 15:50:00 fouroften kernel: Lock_Harness <CVS> (built Dec 10 2004
> 09:14:45) installed
> Dec 10 15:50:00 fouroften kernel: GFS <CVS> (built Dec 10 2004 
> 09:14:04)
> installed
> Dec 10 15:50:00 fouroften kernel: CMAN <CVS> (built Dec 10 2004
> 09:51:59) installed
> Dec 10 15:50:00 fouroften kernel: NET: Registered protocol family 30
> Dec 10 15:50:00 fouroften kernel: DLM <CVS> (built Dec 10 2004 
> 09:52:25)
> installed
> Dec 10 15:50:00 fouroften kernel: Lock_DLM (built Dec 10 2004 09:14:25)
> installed
> Dec 10 15:50:00 fouroften ccsd[3379]: Starting ccsd DEVEL.1102700899:
> Dec 10 15:50:01 fouroften ccsd[3379]:  Built: Dec 10 2004 10:50:57
> Dec 10 15:50:01 fouroften ccsd[3379]:  Copyright (C) Red Hat, Inc.
> 2004  All rights reserved.
> Dec 10 15:50:01 fouroften ccsd[3379]:   Verbose Flag:: SET
> Dec 10 15:50:01 fouroften ccsd[3379]: cluster.conf (cluster name =
> CSMTEST, version = 6) found.
> Dec 10 15:50:01 fouroften kernel: CMAN: Waiting to join or form a
> Linux-cluster
> Dec 10 15:50:01 fouroften crond(pam_unix)[3388]: session opened for 
> user
> root by (uid=0)
> Dec 10 15:50:01 fouroften crond(pam_unix)[3387]: session opened for 
> user
> root by (uid=0)
> Dec 10 15:50:02 fouroften ccsd[3379]: Connected to cluster 
> infrastruture
> via: CMAN/SM Plugin v1.1
> Dec 10 15:50:02 fouroften ccsd[3379]: Initial status:: Inquorate
> Dec 10 15:50:02 fouroften crond(pam_unix)[3388]: session closed for 
> user
> root
> Dec 10 15:50:03 fouroften crond(pam_unix)[3387]: session closed for 
> user
> root
> Dec 10 15:50:33 fouroften kernel: CMAN: forming a new cluster
> Dec 10 15:50:33 fouroften kernel: CMAN: quorum regained, resuming
> activity
> Dec 10 15:50:33 fouroften kernel: CMAN: got node fiveoften
>
> Logs on fiveoften:
> Dec 10 15:50:07 fiveoften kernel: Lock_Harness <CVS> (built Dec 10 2004
> 09:14:45) installed
> Dec 10 15:50:07 fiveoften kernel: GFS <CVS> (built Dec 10 2004 
> 09:14:04)
> installed
> Dec 10 15:50:07 fiveoften crond(pam_unix)[3354]: session closed for 
> user
> root
> Dec 10 15:50:07 fiveoften kernel: CMAN <CVS> (built Dec 10 2004
> 09:51:59) installed
> Dec 10 15:50:07 fiveoften kernel: NET: Registered protocol family 30
> Dec 10 15:50:07 fiveoften kernel: DLM <CVS> (built Dec 10 2004 
> 09:52:25)
> installed
> Dec 10 15:50:07 fiveoften kernel: Lock_DLM (built Dec 10 2004 09:14:25)
> installed
> Dec 10 15:50:08 fiveoften ccsd[3381]: Starting ccsd DEVEL.1102700899:
> Dec 10 15:50:08 fiveoften ccsd[3381]:  Built: Dec 10 2004 10:50:57
> Dec 10 15:50:08 fiveoften ccsd[3381]:  Copyright (C) Red Hat, Inc.
> 2004  All rights reserved.
> Dec 10 15:50:08 fiveoften ccsd[3381]:   Verbose Flag:: SET
> Dec 10 15:50:08 fiveoften ccsd[3381]: cluster.conf (cluster name =
> CSMTEST, version = 6) found.
> Dec 10 15:50:08 fiveoften kernel: CMAN: Waiting to join or form a
> Linux-cluster
> Dec 10 15:50:35 fiveoften kernel: CMAN: sending membership request
> Dec 10 15:50:35 fiveoften kernel: CMAN: got node fouroften
> Dec 10 15:50:35 fiveoften kernel: CMAN: quorum regained, resuming
> activity
> Dec 10 15:50:35 fiveoften ccsd[3381]: Cluster is not quorate.  Refusing
> connection.
> Dec 10 15:50:35 fiveoften ccsd[3381]: Error while processing connect:
> Connection refused
> Dec 10 15:50:36 fiveoften ccsd[3381]: Cluster is not quorate.  Refusing
> connection.
> Dec 10 15:50:36 fiveoften ccsd[3381]: Error while processing connect:
> Connection refused
> Dec 10 15:50:37 fiveoften ccsd[3381]: Cluster is not quorate.  Refusing
> connection.
> Dec 10 15:50:37 fiveoften ccsd[3381]: Error while processing connect:
> Connection refused
> Dec 10 15:50:38 fiveoften ccsd[3381]: Cluster is not quorate.  Refusing
> connection.
> Dec 10 15:50:38 fiveoften ccsd[3381]: Error while processing connect:
> Connection refused
> Dec 10 15:50:39 fiveoften ccsd[3381]: Cluster is not quorate.  Refusing
> connection.
> Dec 10 15:50:39 fiveoften ccsd[3381]: Error while processing connect:
> Connection refused
> Dec 10 15:50:40 fiveoften ccsd[3381]: Cluster is not quorate.  Refusing
> connection.
> Dec 10 15:50:40 fiveoften ccsd[3381]: Error while processing connect:
> Connection refused
> Dec 10 15:50:41 fiveoften ccsd[3381]: Cluster is not quorate.  Refusing
> connection.
> Dec 10 15:50:41 fiveoften ccsd[3381]: Error while processing connect:
> Connection refused
> Dec 10 15:50:42 fiveoften ccsd[3381]: Cluster is not quorate.  Refusing
> connection.
> Dec 10 15:50:42 fiveoften ccsd[3381]: Error while processing connect:
> Connection refused
> Dec 10 15:50:43 fiveoften ccsd[3381]: Cluster is not quorate.  Refusing
> connection.
> Dec 10 15:50:43 fiveoften ccsd[3381]: Error while processing connect:
> Connection refused
> Dec 10 15:50:44 fiveoften ccsd[3381]: Cluster is not quorate.  Refusing
> connection.
> Dec 10 15:50:44 fiveoften ccsd[3381]: Error while processing connect:
> Connection refused
> Dec 10 15:53:26 fiveoften ccsd[3381]: Connected to cluster 
> infrastruture
> via: CMAN/SM Plugin v1.1
> Dec 10 15:53:26 fiveoften ccsd[3381]: Initial status:: Quorate
>
> Matt
> mbrookov at mines.edu
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> http://www.redhat.com/mailman/listinfo/linux-cluster
>