[Linux-cluster] Node can't join already quorated cluster
Javier Vela
jvdiago at gmail.com
Wed Jun 20 21:54:30 UTC 2012
Hi, Finally I solved the problem. First, the qdisk over lvm does not work
very well. Switching to a plain device works better. And, as you stated,
without a fence device is not possible to get the cluster to work well, so
I`m going to push VMWare admins and use vmware fencing.
I'm very grateful, I've been working on this problem 3 days without
understanding what was happening, and with only a few emails the problem is
solved. The only thing that bothers me is why cman doesn't advise you that
without a proper fencing the cluster won't work. Moreover I haven't found
in the Red Hat documentatión a statement telling what I've readed in the
link you pasted:
Fencing is a absolutely critical part of clustering. Without fully working
> fence devices, your cluster will fail.
>
I'm a bit sorry, but now I have another problem. With the cluster quorate
and the two nodes online + qdisk, when I start rgmanager on one node,
everything works ok, an the service starts. Then I start rgmanager in the
other node, but in the second node clustat doesn't show the service:
node2 (with the service working):
[root at node2 ~]# clustat
Cluster Status for test_cluster @ Wed Jun 20 16:21:19 2012
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
node1-hb 1 Online
node2-hb 2 Online, Local, rgmanager
/dev/disk/by-path/pci-0000:02:01.0-scsi- 0 Online, Quorum Disk
Service Name Owner (Last) State
------- ---- ----- ------ -----
service:postgres node2-hb started
node1 (doesn't see the service)
[root at node1 ~]# clustat
Cluster Status for test_cluster @ Wed Jun 20 16:21:15 2012
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
node1-hb 1 Online, Local
node2-hb 2 Online
/dev/disk/by-path/pci-0000:02:01.0-scsi- 0 Online, Quorum Disk
In the /var/log/messages I don't see errors, only this:
last message repeated X times
¿What I'm missing? As far I can see, rgmanager doesn't appear in node1, but:
[root at node1 ~]# service rgmanager status
Se está ejecutando clurgmgrd (pid 8254)...
The cluster conf:
<?xml version="1.0"?>
<cluster alias="test_cluster" config_version="15" name="test_cluster">
<fence_daemon clean_start="0" post_fail_delay="0"
post_join_delay="6"/>
<clusternodes>
<clusternode name="node1-hb" nodeid="1" votes="1">
<fence>
<method name="manual">
<device name="fence_manual"
nodename="node1-hb"/>
</method>
</fence>
</clusternode>
<clusternode name="node2-hb" nodeid="2" votes="1">
<fence>
<method name="manual">
<device name="fence_manual"
nodename="node2-hb"/>
</method>
</fence>
</clusternode>
</clusternodes>
<cman two_node="0" expected_votes="3"/>
<fencedevices>
<fencedevice agent="fence_manual" name="fence_manual"/>
</fencedevices>
<rm log_facility="local4" log_level="7">
<failoverdomains>
<failoverdomain name="etest_cluster_fo"
nofailback="1" ordered="1" restricted="1">
<failoverdomainnode name="node1-hb" priority="1"/>
<failoverdomainnode name="node2-hb"
priority="2"/>
</failoverdomain>
</failoverdomains>
<resources/>
<service autostart="1" domain="test_cluster_fo" exclusive="0"
name="postgres" recovery="relocate">
<ip address="172.24.119.44" monitor_link="1"/>
<lvm name="vg_postgres" vg_name="vg_postgres"
lv_name="postgres"/>
<fs device="/dev/vg_postgres/postgres" force_fsck="1"
force_unmount="1" fstype="ext3" mountpoint="/var/lib/pgsql" name="postgres"
self_fence="0"/>
<script file="/etc/init.d/postgresql" name="postgres">
</script>
</service>
</rm>
<totem consensus="4000" join="60" token="20000"
token_retransmits_before_loss_const="20"/>
<quorumd interval="1" label="cluster_qdisk" tko="10" votes="1">
<heuristic program="/usr/share/cluster/check_eth_link.sh
eth0" score="1" interval="2" tko="3"/>
</quorumd>
</cluster>
Regards, Javi.
2012/6/20 Digimer <lists at alteeve.ca>
> It's worth re-stating;
>
> You are running an unsupported configuration. Please try to have the
> VMWare admins enable fence calls against your nodes and setup fencing.
> Until and unless you do, you will almost certainly run into problems, up to
> and including corrupting your data.
>
> Please take a minute to read this:
>
> https://alteeve.com/w/2-Node_**Red_Hat_KVM_Cluster_Tutorial#**
> Concept.3B_Fencing<https://alteeve.com/w/2-Node_Red_Hat_KVM_Cluster_Tutorial#Concept.3B_Fencing>
>
> Digimer
>
>
> On 06/20/2012 11:22 AM, emmanuel segura wrote:
>
>> Ok Javier
>>
>> So now i know you don't wanna the fencing and the reason :-)
>>
>> <fence_daemon clean_start="1" post_fail_delay="0" post_join_delay="-1"/>
>>
>> and use the fence_manual
>>
>>
>>
>> 2012/6/20 Javier Vela <jvdiago at gmail.com <mailto:jvdiago at gmail.com>>
>>
>>
>> I don't use fencing because with ha-lvm I thought that I dind't need
>> it. But also because both nodes are VMs in VMWare. I know that there
>> is a module to do fencing with vmware but I prefer to avoid it. I'm
>> not in control of the VMWare infraestructure and probably VMWare
>> admins won't give me the tools to use this module.
>>
>> Regards, Javi
>>
>> Fencing is critical, and running a cluster without fencing, even
>> with
>>
>>
>> qdisk, is not supported. Manual fencing is also not supported. The
>> *only* way to have a reliable cluster, testing or production, is
>> to use
>> fencing.
>>
>> Why do you not wish to use it?
>>
>> On 06/20/2012 09:43 AM, Javier Vela wrote:
>>
>>
>> > As I readed, if you use HA-LVM you don't need fencing because of
>> vg
>> > tagging. Is It absolutely mandatory to use fencing with qdisk?
>> >
>> > If it is, i supose i can use manual_fence, but in production I
>> also
>>
>>
>> > won't use fencing.
>> >
>> > Regards, Javi.
>> >
>> > Date: Wed, 20 Jun 2012 14:45:28 +0200
>> > From:emi2fast at gmail.com <mailto:emi2fast at gmail.com> <mailto:
>> emi2fast at gmail.com <mailto:emi2fast at gmail.com>>
>>
>>
>> > To:linux-cluster at redhat.com <mailto:linux-cluster at redhat.**com<linux-cluster at redhat.com>>
>> <mailto:linux-cluster at redhat.**com <linux-cluster at redhat.com> <mailto:
>> linux-cluster at redhat.**com <linux-cluster at redhat.com>>>
>>
>> > Subject: Re: [Linux-cluster] Node can't join already quorated
>> cluster
>>
>>
>> >
>> > If you don't wanna use a real fence divice, because you only do
>> some
>> > test, you have to use fence_manual agent
>> >
>> > 2012/6/20 Javier Vela <jvdiago at gmail.com <mailto:
>> jvdiago at gmail.com> <mailto:jvdiago at gmail.com <mailto:jvdiago at gmail.com
>> >>>
>>
>>
>>
>> >
>> > Hi, I have a very strange problem, and after searching
>> through lot
>> > of forums, I haven't found the solution. This is the
>> scenario:
>> >
>> > Two node cluster with Red Hat 5.7, HA-LVM, no fencing and
>> quorum
>>
>>
>> > disk. I start qdiskd, cman and rgmanager on one node. After 5
>> > minutes, finally the fencing finishes and cluster get
>> quorate with 2
>> > votes:
>> >
>> > [root at node2 ~]# clustat
>> > Cluster Status for test_cluster @ Wed Jun 20 05:56:39 2012
>>
>>
>> > Member Status: Quorate
>> >
>> > Member Name ID Status
>> > ------ ---- ---- ------
>> > node1-hb 1 Offline
>>
>>
>> > node2-hb 2 Online, Local,
>> rgmanager
>> > /dev/mapper/vg_qdisk-lv_qdisk 0 Online,
>> Quorum Disk
>> >
>> > Service Name Owner (Last)
>> State
>>
>>
>> > ------- ---- ----- ------
>> -----
>> > service:postgres node2
>> started
>> >
>> > Now, I start the second node. When cman reaches fencing, it
>> hangs
>>
>>
>> > for 5 minutes aprox, and finally fails. clustat says:
>> >
>> > root at node1 ~]# clustat
>> > Cluster Status for test_cluster @ Wed Jun 20 06:01:12 2012
>> > Member Status: Inquorate
>> >
>>
>>
>> > Member Name ID Status
>> > ------ ---- ---- ------
>> > node1-hb 1 Online, Local
>> > node2-hb 2 Offline
>>
>>
>> > /dev/mapper/vg_qdisk-lv_qdisk 0 Offline
>> >
>> > And in /var/log/messages I can see this errors:
>> >
>> > Jun 20 06:02:12 node1 openais[6098]: [TOTEM] entering
>> OPERATIONAL state.
>>
>>
>> > Jun 20 06:02:12 node1 openais[6098]: [CLM ] got nodejoin
>> message
>> > 15.15.2.10
>> > Jun 20 06:02:13 node1 dlm_controld[5386]: connect to ccs
>> error -111,
>> > check ccsd or cluster status
>>
>>
>> > Jun 20 06:02:13 node1 ccsd[6090]: Cluster is not quorate.
>> Refusing
>> > connection.
>> > Jun 20 06:02:13 node1 ccsd[6090]: Error while processing
>> connect:
>> > Connection refused
>> > Jun 20 06:02:13 node1 ccsd[6090]: Initial status:: Inquorate
>>
>>
>> > Jun 20 06:02:13 node1 gfs_controld[5392]: connect to ccs
>> error -111,
>> > check ccsd or cluster status
>> > Jun 20 06:02:13 node1 ccsd[6090]: Cluster is not quorate.
>> Refusing
>> > connection.
>>
>>
>> > Jun 20 06:02:13 node1 ccsd[6090]: Error while processing
>> connect:
>> > Connection refused
>> > Jun 20 06:02:14 node1 openais[6098]: [TOTEM] entering GATHER
>> state
>> > from 9.
>> > Jun 20 06:02:14 node1 ccsd[6090]: Cluster is not quorate.
>> Refusing
>>
>>
>> > connection.
>> > Jun 20 06:02:14 node1 ccsd[6090]: Error while processing
>> connect:
>> > Connection refused
>> > Jun 20 06:02:14 node1 ccsd[6090]: Cluster is not quorate.
>> Refusing
>> > connection.
>>
>>
>> > Jun 20 06:02:14 node1 ccsd[6090]: Error while processing
>> connect:
>> > Connection refused
>> > Jun 20 06:02:15 node1 ccsd[6090]: Cluster is not quorate.
>> Refusing
>> > connection.
>> > Jun 20 06:02:15 node1 ccsd[6090]: Error while processing
>> connect:
>>
>>
>> > Connection refused
>> > Jun 20 06:02:15 node1 ccsd[6090]: Cluster is not quorate.
>> Refusing
>> > connection.
>> > Jun 20 06:02:15 node1 ccsd[6090]: Error while processing
>> connect:
>> > Connection refused
>>
>>
>> > Jun 20 06:02:15 node1 ccsd[6090]: Cluster is not quorate.
>> Refusing
>> > connection.
>> > Jun 20 06:02:15 node1 ccsd[6090]: Error while processing
>> connect:
>> > Connection refused
>> > Jun 20 06:02:16 node1 ccsd[6090]: Cluster is not quorate.
>> Refusing
>>
>>
>> > connection.
>> > Jun 20 06:02:16 node1 ccsd[6090]: Error while processing
>> connect:
>> > Connection refused
>> > Jun 20 06:02:16 node1 ccsd[6090]: Cluster is not quorate.
>> Refusing
>> > connection.
>>
>>
>> > Jun 20 06:02:16 node1 ccsd[6090]: Error while processing
>> connect:
>> > Connection refused
>> > Jun 20 06:02:17 node1 ccsd[6090]: Cluster is not quorate.
>> Refusing
>> > connection.
>> > Jun 20 06:02:17 node1 ccsd[6090]: Error while processing
>> connect:
>>
>>
>> > Connection refused
>> > Jun 20 06:02:17 node1 ccsd[6090]: Cluster is not quorate.
>> Refusing
>> > connection.
>> > Jun 20 06:02:17 node1 ccsd[6090]: Error while processing
>> connect:
>> > Connection refused
>>
>>
>> > Jun 20 06:02:18 node1 openais[6098]: [TOTEM] entering GATHER
>> state
>> > from 0.
>> > Jun 20 06:02:18 node1 openais[6098]: [TOTEM] Creating commit
>> token
>> > because I am the rep.
>> > Jun 20 06:02:18 node1 openais[6098]: [TOTEM] Storing new
>> sequence id
>>
>>
>> > for ring 15c
>> > Jun 20 06:02:18 node1 openais[6098]: [TOTEM] entering COMMIT
>> state.
>> > Jun 20 06:02:18 node1 openais[6098]: [TOTEM] entering
>> RECOVERY state.
>> > Jun 20 06:02:18 node1 openais[6098]: [TOTEM] position [0]
>> member
>>
>>
>> > 15.15.2.10 <http://15.15.2.10>:
>> > Jun 20 06:02:18 node1 openais[6098]: [TOTEM] previous ring
>> seq 344
>> > rep 15.15.2.10
>> > Jun 20 06:02:18 node1 openais[6098]: [TOTEM] aru e high
>> delivered e
>>
>>
>> > received flag 1
>> > Jun 20 06:02:18 node1 openais[6098]: [TOTEM] Did not need to
>> > originate any messages in recovery.
>> > Jun 20 06:02:18 node1 openais[6098]: [TOTEM] Sending initial
>> ORF token
>>
>>
>> > Jun 20 06:02:18 node1 openais[6098]: [TOTEM] entering
>> OPERATIONAL state.
>> > Jun 20 06:02:18 node1 ccsd[6090]: Cluster is not quorate.
>> Refusing
>> > connection.
>> > Jun 20 06:02:18 node1 ccsd[6090]: Error while processing
>> connect:
>>
>>
>> > Connection refused
>> > Jun 20 06:02:18 node1 openais[6098]: [TOTEM] entering GATHER
>> state
>> > from 9.
>> > Jun 20 06:02:18 node1 ccsd[6090]: Cluster is not quorate.
>> Refusing
>> > connection.
>>
>>
>> >
>> > And the quorum disk:
>> >
>> > [root at node2 ~]# mkqdisk -L -d
>> > kqdisk v0.6.0
>> > /dev/mapper/vg_qdisk-lv_qdisk:
>> > /dev/vg_qdisk/lv_qdisk:
>> > Magic: eb7a62c2
>>
>>
>> > Label: cluster_qdisk
>> > Created: Thu Jun 7 09:23:34 2012
>> > Host: node1
>> > Kernel Sector Size: 512
>>
>>
>> > Recorded Sector Size: 512
>> >
>> > Status block for node 1
>> > Last updated by node 2
>> > Last updated on Wed Jun 20 06:17:23 2012
>> > State: Evicted
>>
>>
>> > Flags: 0000
>> > Score: 0/0
>> > Average Cycle speed: 0.000500 seconds
>> > Last Cycle speed: 0.000000 seconds
>> > Incarnation: 4fe1a06c4fe1a06c
>>
>>
>> > Status block for node 2
>> > Last updated by node 2
>> > Last updated on Wed Jun 20 07:09:38 2012
>> > State: Master
>> > Flags: 0000
>> > Score: 0/0
>>
>>
>> > Average Cycle speed: 0.001000 seconds
>> > Last Cycle speed: 0.000000 seconds
>> > Incarnation: 4fe1a06c4fe1a06c
>> >
>> >
>> > In the other node I don't see any errors in
>> /var/log/messages. One
>>
>>
>> > strange thing is that if I start cman on both nodes at the
>> same
>> > time, everything works fine and both nodes quorate (until I
>> reboot
>> > one node and the problem appears). I've checked that
>> multicast is
>>
>>
>> > working properly. With iperf I can send a receive multicast
>> paquets.
>> > Moreover I've seen with tcpdump the paquets that openais
>> send when
>> > cman is trying to start. I've readed about a bug in RH 5.3
>> with the
>>
>>
>> > same behaviour, but it is solved in RH 5.4.
>> >
>> > I don't have Selinux enabled, and Iptables are also
>> disabled. Here
>> > is the cluster.conf simplified (with less services and
>> resources). I
>>
>>
>> > want to point out one thing. I have allow_kill="0" in order
>> to avoid
>> > fencing errors when quorum tries to fence a failed node. As
>> <fence/>
>> > is empty, before this stanza I got a lot of messages in
>>
>>
>> > /var/log/messages with failed fencing.
>> >
>> > <?xml version="1.0"?>
>> > <cluster alias="test_cluster" config_version="15"
>> name="test_cluster">
>>
>>
>> > <fence_daemon clean_start="0" post_fail_delay="0"
>> > post_join_delay="-1"/>
>> > <clusternodes>
>> > <clusternode name="node1-hb" nodeid="1"
>> votes="1">
>>
>>
>> > <fence/>
>> > </clusternode>
>> > <clusternode name="node2-hb" nodeid="2"
>> votes="1">
>> > <fence/>
>>
>>
>> > </clusternode>
>> > </clusternodes>
>> > <cman two_node="0" expected_votes="3"/>
>> > <fencedevices/>
>>
>>
>> >
>> > <rm log_facility="local4" log_level="7">
>> > <failoverdomains>
>> > <failoverdomain
>> name="etest_cluster_fo"
>>
>>
>> > nofailback="1" ordered="1" restricted="1">
>> > <failoverdomainnode
>> name="node1-hb"
>> > priority="1"/>
>>
>>
>> > <failoverdomainnode
>> name="node2-hb"
>> > priority="2"/>
>> > </failoverdomain>
>> > </failoverdomains>
>>
>>
>> > <resources/>
>> > <service autostart="1" domain="test_cluster_fo"
>> > exclusive="0" name="postgres" recovery="relocate">
>>
>>
>> > <ip address="172.24.119.44"
>> monitor_link="1"/>
>> > <lvm name="vg_postgres"
>> vg_name="vg_postgres"
>> > lv_name="postgres"/>
>>
>>
>> >
>> > <fs device="/dev/vg_postgres/**postgres"
>> > force_fsck="1" force_unmount="1" fstype="ext3"
>> > mountpoint="/var/lib/pgsql" name="postgres" self_fence="0"/>
>>
>>
>> >
>> > <script file="/etc/init.d/postgresql"
>> name="postgres">
>> > </script>
>> > </service>
>> > </rm>
>>
>>
>> > <totem consensus="4000" join="60" token="20000"
>> > token_retransmits_before_loss_**const="20"/>
>> > <quorumd allow_kill="0" interval="1"
>> label="cluster_qdisk"
>>
>>
>> > tko="10" votes="1">
>> > <heuristic
>> > program="/usr/share/cluster/**check_eth_link.sh eth0"
>> score="1"
>> > interval="2" tko="3"/>
>>
>>
>> > </quorumd>
>> > </cluster>
>> >
>> >
>> > The /etc/hosts:
>> > 172.24.119.10 node1
>> > 172.24.119.34 node2
>> > 15.15.2.10 node1-hb node1-hb.localdomain
>>
>>
>> > 15.15.2.11 node2-hb node2-hb.localdomain
>> >
>> > And the versions:
>> > Red Hat Enterprise Linux Server release 5.7 (Tikanga)
>> > cman-2.0.115-85.el5
>> > rgmanager-2.0.52-21.el5
>>
>>
>> > openais-0.80.6-30.el5
>> >
>> > I don't know what else I should try, so if you can give me
>> some
>> > ideas, I will be very pleased.
>> >
>> > Regards, Javi.
>> >
>> > --
>>
>>
>> > Linux-cluster mailing list
>> >Linux-cluster at redhat.com <mailto:Linux-cluster at redhat.**com<Linux-cluster at redhat.com>>
>> <mailto:Linux-cluster at redhat.**com <Linux-cluster at redhat.com> <mailto:
>> Linux-cluster at redhat.**com <Linux-cluster at redhat.com>>>
>>
>>
>> >https://www.redhat.com/**mailman/listinfo/linux-cluster<https://www.redhat.com/mailman/listinfo/linux-cluster>
>>
>> >
>> >
>> >
>> >
>> > --
>> > esta es mi vida e me la vivo hasta que dios quiera
>> >
>> > -- Linux-cluster mailing listLinux-cluster at redhat.com <mailto:
>> Linux-cluster at redhat.**com <Linux-cluster at redhat.com>>
>>
>> > <mailto:Linux-cluster at redhat.**com <Linux-cluster at redhat.com> <mailto:
>> Linux-cluster at redhat.**com <Linux-cluster at redhat.com>>>
>>
>>
>> >https://www.redhat.com/**mailman/listinfo/linux-cluster<https://www.redhat.com/mailman/listinfo/linux-cluster>
>> >
>> >
>> > --
>> > Linux-cluster mailing list
>> >Linux-cluster at redhat.com <mailto:Linux-cluster at redhat.**com<Linux-cluster at redhat.com>
>> >
>>
>>
>> >https://www.redhat.com/**mailman/listinfo/linux-cluster<https://www.redhat.com/mailman/listinfo/linux-cluster>
>> >
>>
>>
>> --
>> Digimer
>>
>> Papers and Projects:https://alteeve.com
>>
>>
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com <mailto:Linux-cluster at redhat.**com<Linux-cluster at redhat.com>
>> >
>> https://www.redhat.com/**mailman/listinfo/linux-cluster<https://www.redhat.com/mailman/listinfo/linux-cluster>
>>
>>
>>
>>
>> --
>> esta es mi vida e me la vivo hasta que dios quiera
>>
>>
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/**mailman/listinfo/linux-cluster<https://www.redhat.com/mailman/listinfo/linux-cluster>
>>
>>
>
> --
> Digimer
> Papers and Projects: https://alteeve.com
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/**mailman/listinfo/linux-cluster<https://www.redhat.com/mailman/listinfo/linux-cluster>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20120620/bc7583be/attachment.htm>
More information about the Linux-cluster
mailing list