hi, maybe u have your cluster have Votes problem, when u use "cman_tool status" and if u see like a "Quorum: 2 Activity Blocked" message, then use "cman_tool expected -e 1". Again use "cman_tool status", u will see removed "Activity Blocked" on "Quorum:". This command pull down Qourum of cluster and cluster will runing (expected vote: n/2+1). This solution a tepmrolary solution for inadequate cluster qourum. (before check config_version="{ver.num}" , u know, must be same number in all nodes) i hope server for your problem (yep, i know.. my englsh are ultra-professionel-imba :) ) <div class="gmail_quote">2010/1/22 Terry <<a href="mailto:td3201@gmail.com">td3201@gmail.com</a>> <blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div><div></div><div class="h5">On Fri, Jan 22, 2010 at 9:00 AM, King, Adam <<a href="mailto:adam.king@intechnology.com">adam.king@intechnology.com</a>> wrote: > I'm assuming you have read this? <a href="http://sources.redhat.com/cluster/wiki/FAQ/CMAN#cman_2to3" target="_blank">http://sources.redhat.com/cluster/wiki/FAQ/CMAN#cman_2to3</a> > > > > > Adam King > Systems Administrator > <a href="mailto:adam.king@intechnology.com">adam.king@intechnology.com</a> > > > InTechnology plc > Support 0845 120 7070 > Telephone 01423 850000 > Facsimile 01423 858866 > <a href="http://www.intechnology.com" target="_blank">www.intechnology.com</a> > > > -----Original Message----- > > From: <a href="mailto:linux-cluster-bounces@redhat.com">linux-cluster-bounces@redhat.com</a> [mailto:<a href="mailto:linux-cluster-bounces@redhat.com">linux-cluster-bounces@redhat.com</a>] On Behalf Of Terry > Sent: 22 January 2010 14:45 > To: linux clustering > Subject: Re: [Linux-cluster] cannot add 3rd node to running cluster > > On Mon, Jan 4, 2010 at 1:34 PM, Abraham Alawi <<a href="mailto:a.alawi@auckland.ac.nz">a.alawi@auckland.ac.nz</a>> wrote: >> >> On 1/01/2010, at 5:13 AM, Terry wrote: >> >>> On Wed, Dec 30, 2009 at 10:13 AM, Terry <<a href="mailto:td3201@gmail.com">td3201@gmail.com</a>> wrote: >>>> On Tue, Dec 29, 2009 at 5:20 PM, Jason W. <<a href="mailto:jwellband@gmail.com">jwellband@gmail.com</a>> wrote: >>>>> On Tue, Dec 29, 2009 at 2:30 PM, Terry <<a href="mailto:td3201@gmail.com">td3201@gmail.com</a>> wrote: >>>>>> Hello, >>>>>> >>>>>> I have a working 2 node cluster that I am trying to add a third node >>>>>> to. I am trying to use Red Hat's conga (luci) to add the node in but >>>>> >>>>> If you have two node cluster with two_node=1 in cluster.conf - such as >>>>> two nodes with no quorum device to break a tie - you'll need to bring >>>>> the cluster down, change two_node to 0 on both nodes (and rev the >>>>> cluster version at the top of cluster.conf), bring the cluster up and >>>>> then add the third node. >>>>> >>>>> For troubleshooting any cluster issue, take a look at syslog >>>>> (/var/log/messages by default). It can help to watch it on a >>>>> centralized syslog server that all of your nodes forward logs to. >>>>> >>>>> -- >>>>> HTH, YMMV, HANW :) >>>>> >>>>> Jason >>>>> >>>>> The path to enlightenment is /usr/bin/enlightenment. >>>> >>>> Thank you for the response. /var/log/messages doesn't have any >>>> errors. It says cman started then says can't connect to cluster >>>> infrastructure after a few seconds. My cluster does not have the >>>> two_node=1 config now. Conga took that out for me. That bit me last >>>> night because I needed to put that back in. >>>> >>> >>> CMAN still will not start and gives no debug information. Anyone know >>> why cman_tool -d join would not print any output at all? >>> Troubleshooting this is kind of a nightmare. I verified that two_node >>> is not in play. >>> >>> -- >>> Linux-cluster mailing list >>> <a href="mailto:Linux-cluster@redhat.com">Linux-cluster@redhat.com</a> >>> <a href="https://www.redhat.com/mailman/listinfo/linux-cluster" target="_blank">https://www.redhat.com/mailman/listinfo/linux-cluster</a> >> >> >> Try this line in your cluster.conf file: >> <logging debug="on" logfile="/var/log/rhcs.log" to_file="yes"/> >> >> Also, if you are sure your cluster.conf is correct then copy it manually to all the nodes and add clean_start="1" to the fence_daemon line in cluster.conf and run 'service cman start' simultaneously on all the nodes (probably a good idea to do that from runlevel 1 but make sure you have the network up first) >> >> Cheers, >> >> -- Abraham >> >> '''''''''''''''''''''''''''''''''''''''''''''''''''''' >> Abraham Alawi >> >> Unix/Linux Systems Administrator >> Science IT >> University of Auckland >> e: <a href="mailto:a.alawi@auckland.ac.nz">a.alawi@auckland.ac.nz</a> >> p: +64-9-373 7599, ext#: 87572 >> >> '''''''''''''''''''''''''''''''''''''''''''''''''''''' >> >> > > I am still battling this. I stopped the cluster completely, modified > the config and then started it, but that didn't work either. Same > issue. I noticed clurgmgrd wasn't staying running so I then tried > this: > > [root@omadvnfs01c ~]# clurgmgrd -d -f > [7014] notice: Waiting for CMAN to start > > Then in another window I issued: > [root@omadvnfs01c ~]# cman_tool join > > > Then back in the other window below "[7014] notice: Waiting for CMAN > to start", I got: > failed acquiring lockspace: Transport endpoint is not connected > Locks not working! > > Anyone know what could be going on? > > -- > Linux-cluster mailing list > <a href="mailto:Linux-cluster@redhat.com">Linux-cluster@redhat.com</a> > <a href="https://www.redhat.com/mailman/listinfo/linux-cluster" target="_blank">https://www.redhat.com/mailman/listinfo/linux-cluster</a> > This is an email from InTechnology plc, Central House, Beckwith Knowle, Harrogate, UK, HG3 1UG. > Registered in England 3916586. > > The contents of this message may be privileged and confidential. If you have received this message in error, you may not use, > > disclose, copy or distribute its content in any way. Please notify the sender immediately. All messages are scanned for all viruses. > > -- </div></div>I didn't but I performed those steps anyways. As it sits, I have a three node cluster with only two nodes in it. Which is bad too but it is what it is until I figure this out. Here's my cluster.conf just for completeness: <cluster alias="omadvnfs01" config_version="53" name="omadvnfs01"> <fence_daemon clean_start="0" post_fail_delay="0" post_join_delay="3"/> <clusternodes> <clusternode name="<a href="http://omadvnfs01a.sec.jel.lc" target="_blank">omadvnfs01a.sec.jel.lc</a>" nodeid="1" votes="1"> <fence> <method name="1"> <device name="omadvnfs01a-drac"/> </method> </fence> </clusternode> <clusternode name="<a href="http://omadvnfs01b.sec.jel.lc" target="_blank">omadvnfs01b.sec.jel.lc</a>" nodeid="2" votes="1"> <fence> <method name="1"> <device name="omadvnfs01b-drac"/> </method> </fence> </clusternode> <clusternode name="<a href="http://omadvnfs01c.sec.jel.lc" target="_blank">omadvnfs01c.sec.jel.lc</a>" nodeid="3" votes="1"> <fence> <method name="1"> <device name="omadvnfs01c-drac"/> </method> </fence> </clusternode> </clusternodes> <cman/> <fencedevices> <fencedevice agent="fence_drac" ipaddr="10.98.1.211" login="root" name="omadvnfs01a-drac" passwd="foo"/> <fencedevice agent="fence_drac" ipaddr="10.98.1.212" login="root" name="omadvnfs01b-drac" passwd="foo"/> <fencedevice agent="fence_drac" ipaddr="10.98.1.213" login="root" name="omadvnfs01c-drac" passwd="foo"/> </fencedevices> <rm> <failoverdomains> <failoverdomain name="fd_omadvnfs01a-nfs" nofailback="1" ordered="1" restricted="0"> <failoverdomainnode name="<a href="http://omadvnfs01a.sec.jel.lc" target="_blank">omadvnfs01a.sec.jel.lc</a>" priority="1"/> </failoverdomain> <failoverdomain name="fd_omadvnfs01b-nfs" nofailback="1" ordered="1" restricted="0"> <failoverdomainnode name="<a href="http://omadvnfs01b.sec.jel.lc" target="_blank">omadvnfs01b.sec.jel.lc</a>" priority="2"/> </failoverdomain> <failoverdomain name="fd_omadvnfs01c-nfs" nofailback="1" ordered="1" restricted="0"> <failoverdomainnode name="<a href="http://omadvnfs01c.sec.jel.lc" target="_blank">omadvnfs01c.sec.jel.lc</a>" priority="1"/> </failoverdomain> </failoverdomains> I am not sure if I did a restart after I did the work though. When it says "shutdown cluster software" that is simply a 'service cman stop' on redhat, right? Want to make sure I don't need to kill any other components before updating the configuration manually. I appreciate the help. I am probably going to try it again this afternoon to double check my work. <div><div></div><div class="h5"> -- Linux-cluster mailing list <a href="mailto:Linux-cluster@redhat.com">Linux-cluster@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/linux-cluster" target="_blank">https://www.redhat.com/mailman/listinfo/linux-cluster</a> </div></div></blockquote></div>