<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN"> <HTML> <HEAD> <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"> <META NAME="Generator" CONTENT="MS Exchange Server version 6.5.7652.24"> <TITLE>RE: [Linux-cluster] 3 node cluster problems</TITLE> </HEAD> <BODY>  If you are configuring your cluster by system-config-cluster then no need to run ricci/luci. Ricci/luci needed for configuring the cluster using conga. You can configure in either ways. On seeing your clustat command outputs, it seems cluster is partitioned (spilt brain) into 2 sub clusters [Sub1- (csarcsys1-eth0, csarcsys2-eth0) 2- csarcsys3-eth0]. Without a quorum device you can more often face this situation. To avoid this you can configure a quorum device with a heuristic like ping message. Use the link (<A HREF="http://www.redhatmagazine.com/2007/12/19/enhancing-cluster-quorum-with-qdisk/">http://www.redhatmagazine.com/2007/12/19/enhancing-cluster-quorum-with-qdisk/</A>) for configuring a quorum disk in RHCS. Thanks, S -----Original Message----- From: linux-cluster-bounces@redhat.com [<A HREF="mailto:linux-cluster-bounces@redhat.com">mailto:linux-cluster-bounces@redhat.com</A>] On Behalf Of Dalton, Maurice Sent: Tuesday, March 25, 2008 5:18 PM To: linux clustering Subject: RE: [Linux-cluster] 3 node cluster problems Still no change. Same as below. I completely rebuilt the cluster using system-config-cluster The Cluster software was installed from rhn, luci and ricci are running. This is the new config file and it has been copied to the 2 other systems [root@csarcsys1-eth0 cluster]# more cluster.conf <?xml version="1.0"?> <cluster config_version="5" name="csarcsys5"> <fence_daemon post_fail_delay="0" post_join_delay="3"/> <clusternodes> <clusternode name="csarcsys1-eth0" nodeid="1" votes="1"> <fence/> </clusternode> <clusternode name="csarcsys2-eth0" nodeid="2" votes="1"> <fence/> </clusternode> <clusternode name="csarcsys3-eth0" nodeid="3" votes="1"> <fence/> </clusternode> </clusternodes> <cman/> <fencedevices/> <rm> <failoverdomains> <failoverdomain name="csarcsysfo" ordered="0" restricted="1"> <failoverdomainnode name="csarcsys1-eth0" priority="1"/> <failoverdomainnode name="csarcsys2-eth0" priority="1"/> <failoverdomainnode name="csarcsys3-eth0" priority="1"/> </failoverdomain> </failoverdomains> <resources> <ip address="172.xx.xx.xxx" monitor_link="1"/> <fs device="/dev/sdc1" force_fsck="0" force_unmount="1" fsid="57739" fstype="ext3" mountpo int="/csarc-test" name="csarcsys-fs" options="rw" self_fence="0"/> </resources> </rm> </cluster> -----Original Message----- From: linux-cluster-bounces@redhat.com [<A HREF="mailto:linux-cluster-bounces@redhat.com">mailto:linux-cluster-bounces@redhat.com</A>] On Behalf Of Bennie Thomas Sent: Monday, March 24, 2008 4:17 PM To: linux clustering Subject: Re: [Linux-cluster] 3 node cluster problems Did you load the Cluster software via Conga or manually ? You would have had to load luci on one node and ricci on all three. Try copying the modified /etc/cluster/cluster.conf from csarcsys1 to the other two nodes. Make sure you can ping the private interface to/from all nodes and reboot. If this does not work post your /etc/cluster/cluster.conf file again. Dalton, Maurice wrote: > Yes > I also rebooted again just now to be sure. > > > -----Original Message----- > From: linux-cluster-bounces@redhat.com > [<A HREF="mailto:linux-cluster-bounces@redhat.com">mailto:linux-cluster-bounces@redhat.com</A>] On Behalf Of Bennie Thomas > Sent: Monday, March 24, 2008 3:33 PM > To: linux clustering > Subject: Re: [Linux-cluster] 3 node cluster problems > > When you changed the nodenames in the /etc/lcuster/cluster.conf and made > > sure the /etc/hosts > file had the correct nodenames (Ie. 10.0.0.100 csarcsys1-eth0 > csarcsys1-eth0.xxxx.xxxx.xxx.) > Did you reboot all the nodes at the sametime ? > > Dalton, Maurice wrote: > >> No luck. It seems as if csarcsys3 thinks its in his own cluster >> I renamed all config files and rebuilt from system-config-cluster >> >> Clustat command from csarcsys3 >> >> >> [root@csarcsys3-eth0 cluster]# clustat >> msg_open: No such file or directory >> Member Status: Inquorate >> >> Member Name ID Status >> ------ ---- ---- ------ >> csarcsys1-eth0 1 Offline >> csarcsys2-eth0 2 Offline >> csarcsys3-eth0 3 Online, Local >> >> clustat command from csarcsys2 >> >> [root@csarcsys2-eth0 cluster]# clustat >> msg_open: No such file or directory >> Member Status: Quorate >> >> Member Name ID Status >> ------ ---- ---- ------ >> csarcsys1-eth0 1 Online >> csarcsys2-eth0 2 Online, Local >> csarcsys3-eth0 3 Offline >> >> >> -----Original Message----- >> From: linux-cluster-bounces@redhat.com >> [<A HREF="mailto:linux-cluster-bounces@redhat.com">mailto:linux-cluster-bounces@redhat.com</A>] On Behalf Of Bennie Thomas >> Sent: Monday, March 24, 2008 2:25 PM >> To: linux clustering >> Subject: Re: [Linux-cluster] 3 node cluster problems >> >> You will also, need to make sure the clustered nodenames are in your >> /etc/hosts file. >> Also, make sure your cluster network interface is up on all nodes and >> that the >> /etc/cluster/cluster.conf are the same on all nodes. >> >> >> >> Dalton, Maurice wrote: >> >> >>> The last post is incorrect. >>> >>> Fence is still hanging at start up. >>> >>> Here's another log message. >>> >>> Mar 24 19:03:14 csarcsys3-eth0 ccsd[6425]: Error while processing >>> connect: Connection refused >>> >>> Mar 24 19:03:15 csarcsys3-eth0 dlm_controld[6453]: connect to ccs >>> error -111, check ccsd or cluster status >>> >>> *From:* linux-cluster-bounces@redhat.com >>> [<A HREF="mailto:linux-cluster-bounces@redhat.com">mailto:linux-cluster-bounces@redhat.com</A>] *On Behalf Of *Bennie >>> > Thomas > >>> *Sent:* Monday, March 24, 2008 11:22 AM >>> *To:* linux clustering >>> *Subject:* Re: [Linux-cluster] 3 node cluster problems >>> >>> try removing the fully qualified hostname from the cluster.conf file. >>> >>> >>> Dalton, Maurice wrote: >>> >>> I have NO fencing equipment >>> >>> I have been task to setup a 3 node cluster >>> >>> Currently I have having problems getting cman(fence) to start >>> >>> Fence will try to start up during cman start up but will fail >>> >>> I tried to run /sbin/fenced -D - I get the following >>> >>> 1206373475 cman_init error 0 111 >>> >>> Here's my cluster.conf file >>> >>> <?xml version="1.0"?> >>> >>> <cluster alias="csarcsys51" config_version="26" name="csarcsys51"> >>> >>> <fence_daemon clean_start="0" post_fail_delay="0" >>> >>> >> post_join_delay="3"/> >> >> >>> <clusternodes> >>> >>> <clusternode name="csarcsys1-eth0.xxx.xxxx.nasa.gov" nodeid="1" >>> >>> >> votes="1"> >> >> >>> <fence/> >>> >>> </clusternode> >>> >>> <clusternode name="csarcsys2-eth0.xxx.xxxx.nasa.gov" nodeid="2" >>> >>> >> votes="1"> >> >> >>> <fence/> >>> >>> </clusternode> >>> >>> <clusternode name="csarcsys3-eth0.xxx.xxxxnasa.gov" nodeid="3" >>> >>> >> votes="1"> >> >> >>> <fence/> >>> >>> </clusternode> >>> >>> </clusternodes> >>> >>> <cman/> >>> >>> <fencedevices/> >>> >>> <rm> >>> >>> <failoverdomains> >>> >>> <failoverdomain name="csarcsys-fo" ordered="1" restricted="0"> >>> >>> <failoverdomainnode name="csarcsys1-eth0.xxx.xxxx.nasa.gov" >>> >>> >> priority="1"/> >> >> >>> <failoverdomainnode name="csarcsys2-eth0.xxx.xxxx.nasa.gov" >>> >>> >> priority="1"/> >> >> >>> <failoverdomainnode name="csarcsys2-eth0.xxx.xxxx.nasa.gov" >>> >>> >> priority="1"/> >> >> >>> </failoverdomain> >>> >>> </failoverdomains> >>> >>> <resources> >>> >>> <ip address="xxx.xxx.xxx.xxx" monitor_link="1"/> >>> >>> <fs device="/dev/sdc1" force_fsck="0" force_unmount="1" fsid="57739" >>> fstype="ext3" mountpo >>> >>> int="/csarc-test" name="csarcsys-fs" options="rw" self_fence="0"/> >>> >>> <nfsexport name="csarcsys-export"/> >>> >>> <nfsclient name="csarcsys-nfs-client" options="no_root_squash,rw" >>> path="/csarc-test" targe >>> >>> t="xxx.xxx.xxx.*"/> >>> >>> </resources> >>> >>> </rm> >>> >>> </cluster> >>> >>> Messages from the logs >>> >>> ar 24 13:24:19 csarcsys2-eth0 ccsd[24888]: Cluster is not quorate. >>> Refusing connection. >>> >>> Mar 24 13:24:19 csarcsys2-eth0 ccsd[24888]: Error while processing >>> connect: Connection refused >>> >>> Mar 24 13:24:20 csarcsys2-eth0 ccsd[24888]: Cluster is not quorate. >>> Refusing connection. >>> >>> Mar 24 13:24:20 csarcsys2-eth0 ccsd[24888]: Error while processing >>> connect: Connection refused >>> >>> Mar 24 13:24:21 csarcsys2-eth0 ccsd[24888]: Cluster is not quorate. >>> Refusing connection. >>> >>> Mar 24 13:24:21 csarcsys2-eth0 ccsd[24888]: Error while processing >>> connect: Connection refused >>> >>> Mar 24 13:24:22 csarcsys2-eth0 ccsd[24888]: Cluster is not quorate. >>> Refusing connection. >>> >>> Mar 24 13:24:22 csarcsys2-eth0 ccsd[24888]: Error while processing >>> connect: Connection refused >>> >>> Mar 24 13:24:23 csarcsys2-eth0 ccsd[24888]: Cluster is not quorate. >>> Refusing connection. >>> >>> Mar 24 13:24:23 csarcsys2-eth0 ccsd[24888]: Error while processing >>> connect: Connection refused >>> >>> >>> >>> >>> > ------------------------------------------------------------------------ > >> >> >>> >>> >>> -- >>> Linux-cluster mailing list >>> Linux-cluster@redhat.com <<A HREF="mailto:Linux-cluster@redhat.com">mailto:Linux-cluster@redhat.com</A>> >>> <A HREF="https://www.redhat.com/mailman/listinfo/linux-cluster">https://www.redhat.com/mailman/listinfo/linux-cluster</A> >>> >>> >>> >>> > ------------------------------------------------------------------------ > >> >> >>> -- >>> Linux-cluster mailing list >>> Linux-cluster@redhat.com >>> <A HREF="https://www.redhat.com/mailman/listinfo/linux-cluster">https://www.redhat.com/mailman/listinfo/linux-cluster</A> >>> >>> >> >> -- >> Linux-cluster mailing list >> Linux-cluster@redhat.com >> <A HREF="https://www.redhat.com/mailman/listinfo/linux-cluster">https://www.redhat.com/mailman/listinfo/linux-cluster</A> >> >> -- >> Linux-cluster mailing list >> Linux-cluster@redhat.com >> <A HREF="https://www.redhat.com/mailman/listinfo/linux-cluster">https://www.redhat.com/mailman/listinfo/linux-cluster</A> >> >> > > > -- > Linux-cluster mailing list > Linux-cluster@redhat.com > <A HREF="https://www.redhat.com/mailman/listinfo/linux-cluster">https://www.redhat.com/mailman/listinfo/linux-cluster</A> > > -- > Linux-cluster mailing list > Linux-cluster@redhat.com > <A HREF="https://www.redhat.com/mailman/listinfo/linux-cluster">https://www.redhat.com/mailman/listinfo/linux-cluster</A> > -- Linux-cluster mailing list Linux-cluster@redhat.com <A HREF="https://www.redhat.com/mailman/listinfo/linux-cluster">https://www.redhat.com/mailman/listinfo/linux-cluster</A> -- Linux-cluster mailing list Linux-cluster@redhat.com <A HREF="https://www.redhat.com/mailman/listinfo/linux-cluster">https://www.redhat.com/mailman/listinfo/linux-cluster</A> </BODY> </HTML>