<div dir="ltr"><div>Thanks Fabio for replying may request.</div><div><br></div>I'm using stock CentOS 6.4 versions and no rm, just clvmd and dlm.<div><br></div><div><div>Name : cman Relocations: (not relocatable)</div>
<div>Version : 3.0.12.1 Vendor: CentOS</div><div>Release : 49.el6_4.2 Build Date: Tue 03 Sep 2013 02:18:10 AM PDT</div><div><br></div><div><div>Name : lvm2-cluster Relocations: (not relocatable)</div>
<div>Version : 2.02.98 Vendor: CentOS</div><div>Release : 9.el6_4.3 Build Date: Tue 05 Nov 2013 07:36:18 AM PST</div></div><div><br></div><div><div>Name : corosync Relocations: (not relocatable)</div>
<div>Version : 1.4.1 Vendor: CentOS</div><div>Release : 15.el6_4.1 Build Date: Tue 14 May 2013 02:09:27 PM PDT</div></div><div><br></div><div><br></div><div>My question is based off this problem I have till January:</div>
<div><br></div><div><font face="courier new, monospace"><br>When ever I add a new node (</font><span style="font-family:'courier new',monospace">I put into the cluster.conf and reloaded with cman_tool version -r -S) </span><font face="courier new, monospace"> I end up with situations like the new node wants to gain the quorum and starts to fence the existing pool master and appears to generate some sort of split cluster. Does it work at all, corosync and dlm do not know about the recently added node ?</font><div>
<font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">New Node </font></div><div><font face="courier new, monospace">==========<br><br><div>Node Sts Inc Joined Name</div>
<div> 1 X 0 hv-1</div><div> 2 X 0 hv-2</div><div> 3 X 0 hv-3</div><div> 4 X 0 hv-4</div><div>
5 X 0 hv-5</div><div> 6 M 80 2014-01-07 21:37:42 hv-6<--- host added</div><div><br></div><br><div>Jan 7 21:37:42 hv-1 corosync[12564]: [TOTEM ] The network interface [10.14.18.77] is now up.</div>
<div>Jan 7 21:37:42 hv-1 corosync[12564]: [QUORUM] Using quorum provider quorum_cman</div><div>Jan 7 21:37:42 hv-1 corosync[12564]: [SERV ] Service engine loaded: corosync cluster quorum service v0.1</div><div>Jan 7 21:37:42 hv-1 corosync[12564]: [CMAN ] CMAN 3.0.12.1 (built Sep 3 2013 09:17:34) started</div>
<div>Jan 7 21:37:42 hv-1 corosync[12564]: [SERV ] Service engine loaded: corosync CMAN membership service 2.90</div><div>Jan 7 21:37:42 hv-1 corosync[12564]: [SERV ] Service engine loaded: openais checkpoint service B.01.01</div>
<div>Jan 7 21:37:42 hv-1 corosync[12564]: [SERV ] Service engine loaded: corosync extended virtual synchrony service</div><div>Jan 7 21:37:42 hv-1 corosync[12564]: [SERV ] Service engine loaded: corosync configuration service</div>
<div>Jan 7 21:37:42 hv-1 corosync[12564]: [SERV ] Service engine loaded: corosync cluster closed process group service v1.01</div><div>Jan 7 21:37:42 hv-1 corosync[12564]: [SERV ] Service engine loaded: corosync cluster config database access v1.01</div>
<div>Jan 7 21:37:42 hv-1 corosync[12564]: [SERV ] Service engine loaded: corosync profile loading service</div><div>Jan 7 21:37:42 hv-1 corosync[12564]: [QUORUM] Using quorum provider quorum_cman</div><div>Jan 7 21:37:42 hv-1 corosync[12564]: [SERV ] Service engine loaded: corosync cluster quorum service v0.1</div>
<div>Jan 7 21:37:42 hv-1 corosync[12564]: [MAIN ] Compatibility mode set to whitetank. Using V1 and V2 of the synchronization engine.</div><div>Jan 7 21:37:42 hv-1 corosync[12564]: [TOTEM ] adding new UDPU member {10.14.18.65}</div>
<div>Jan 7 21:37:42 hv-1 corosync[12564]: [TOTEM ] adding new UDPU member {10.14.18.67}</div><div>Jan 7 21:37:42 hv-1 corosync[12564]: [TOTEM ] adding new UDPU member {10.14.18.68}</div><div>Jan 7 21:37:42 hv-1 corosync[12564]: [TOTEM ] adding new UDPU member {10.14.18.70}</div>
<div>Jan 7 21:37:42 hv-1 corosync[12564]: [TOTEM ] adding new UDPU member {10.14.18.66}</div><div>Jan 7 21:37:42 hv-1 corosync[12564]: [TOTEM ] adding new UDPU member {10.14.18.77}</div><div>Jan 7 21:37:42 hv-1 corosync[12564]: [TOTEM ] A processor joined or left the membership and a new membership was formed.</div>
<div>Jan 7 21:37:42 hv-1 corosync[12564]: [CMAN ] quorum regained, resuming activity</div><div>Jan 7 21:37:42 hv-1 corosync[12564]: [QUORUM] This node is within the primary component and will provide service.</div>
<div>Jan 7 21:37:42 hv-1 corosync[12564]: [QUORUM] Members[1]: 6</div><div>Jan 7 21:37:42 hv-1 corosync[12564]: [QUORUM] Members[1]: 6</div><div>Jan 7 21:37:42 hv-1 corosync[12564]: [CPG ] chosen downlist: sender r(0) ip(10.14.18.77) ; members(old:0 left:0)</div>
<div>Jan 7 21:37:42 hv-1 corosync[12564]: [MAIN ] Completed service synchronization, ready to provide service.</div><div>Jan 7 21:37:46 hv-1 fenced[12620]: fenced 3.0.12.1 started</div><div>Jan 7 21:37:46 hv-1 dlm_controld[12643]: dlm_controld 3.0.12.1 started</div>
<div>Jan 7 21:37:47 hv-1 gfs_controld[12695]: gfs_controld 3.0.12.1 started</div><div>Jan 7 21:37:54 hv-1 fenced[12620]: fencing node hv-b1clcy1</div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><div>
sudo -i corosync-objctl |grep member</div><div><br></div><div>totem.interface.member.memberaddr=hv-1</div><div>totem.interface.member.memberaddr=hv-2</div><div>totem.interface.member.memberaddr=hv-3</div><div>totem.interface.member.memberaddr=hv-4</div>
<div>totem.interface.member.memberaddr=hv-5</div><div>totem.interface.member.memberaddr=hv-6</div><div>runtime.totem.pg.mrp.srp.members.6.ip=r(0) ip(10.14.18.77)</div><div>runtime.totem.pg.mrp.srp.members.6.join_count=1</div>
<div>runtime.totem.pg.mrp.srp.members.6.status=joined</div><div><br></div></font></div><div><font face="courier new, monospace"><br></font></div><div><div style="font-family:arial"><font face="courier new, monospace">Existing Node </font></div>
<div style="font-family:arial"><font face="courier new, monospace">=============</font></div></div><div style="font-family:arial"><font face="courier new, monospace"><br></font></div><div style="font-family:arial"><font face="courier new, monospace">member 6 has not been added to the quorum list :</font></div>
<div style="font-family:arial"><font face="courier new, monospace"><br></font></div></font><div><span style="font-family:'courier new',monospace">Jan 7 21:36:28 hv-1 corosync[7769]: [QUORUM] Members[4]: 1 2 3 5</span><br>
</div><div><font face="courier new, monospace"><div>Jan 7 21:37:54 hv-1 corosync[7769]: [TOTEM ] A processor joined or left the membership and a new membership was formed.</div><div>Jan 7 21:37:54 hv-1 corosync[7769]: [CPG ] chosen downlist: sender r(0) ip(10.14.18.65) ; members(old:4 left:0)</div>
<div><br></div></font><div><font face="courier new, monospace"><br>Node Sts Inc Joined Name<br> 1 M 4468 2013-12-10 14:33:27 hv-1<br> 2 M 4468 2013-12-10 14:33:27 hv-2<br> 3 M 5036 2014-01-07 17:51:26 hv-3<br>
4 X 4468 hv-4(dead at the moment)<br> 5 M 4468 2013-12-10 14:33:27 hv-5<br> 6 X 0 hv-6<--- added</font><br></div></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><div style="color:rgb(0,0,0)">Jan 7 21:36:28 hv-1 corosync[7769]: [QUORUM] Members[4]: 1 2 3 5</div><div style="color:rgb(0,0,0)">
Jan 7 21:37:54 hv-1 corosync[7769]: [TOTEM ] A processor joined or left the membership and a new membership was formed.</div><div style="color:rgb(0,0,0)">Jan 7 21:37:54 hv-1 corosync[7769]: [CPG ] chosen downlist: sender r(0) ip(10.14.18.65) ; members(old:4 left:0)</div>
<div style="color:rgb(0,0,0)">Jan 7 21:37:54 hv-1 corosync[7769]: [MAIN ] Completed service synchronization, ready to provide service.</div><div style="color:rgb(0,0,0)"><br></div><div style="color:rgb(0,0,0)"><br></div>
<div style="color:rgb(0,0,0)"><div>totem.interface.member.memberaddr=hv-1</div><div>totem.interface.member.memberaddr=hv-2</div><div>totem.interface.member.memberaddr=hv-3</div><div>totem.interface.member.memberaddr=hv-4</div>
<div>totem.interface.member.memberaddr=hv-5.</div><div>runtime.totem.pg.mrp.srp.members.1.ip=r(0) ip(10.14.18.65)</div><div>runtime.totem.pg.mrp.srp.members.1.join_count=1</div><div>runtime.totem.pg.mrp.srp.members.1.status=joined</div>
<div>runtime.totem.pg.mrp.srp.members.2.ip=r(0) ip(10.14.18.66)</div><div>runtime.totem.pg.mrp.srp.members.2.join_count=1</div><div>runtime.totem.pg.mrp.srp.members.2.status=joined</div><div>runtime.totem.pg.mrp.srp.members.4.ip=r(0) ip(10.14.18.68)</div>
<div>runtime.totem.pg.mrp.srp.members.4.join_count=1</div><div>runtime.totem.pg.mrp.srp.members.4.status=left</div><div>runtime.totem.pg.mrp.srp.members.5.ip=r(0) ip(10.14.18.70)</div><div>runtime.totem.pg.mrp.srp.members.5.join_count=1</div>
<div>runtime.totem.pg.mrp.srp.members.5.status=joined</div><div>runtime.totem.pg.mrp.srp.members.3.ip=r(0) ip(10.14.18.67)</div><div>runtime.totem.pg.mrp.srp.members.3.join_count=3</div><div>runtime.totem.pg.mrp.srp.members.3.status=joined</div>
</div><div style="color:rgb(0,0,0)"><br></div><div style="color:rgb(0,0,0)"><br></div><div style="color:rgb(0,0,0)">cluster.conf:</div><div style="color:rgb(0,0,0)"><br></div><div style="color:rgb(0,0,0)"><div><?xml version="1.0"?></div>
<div><cluster config_version="32" name="hv-1618-110-1"></div><div> <fence_daemon clean_start="0"/></div><div> <cman transport="udpu" expected_votes="1"/></div>
<div> <logging debug="off"/></div><div> <clusternodes></div><div> <clusternode name="hv-1" votes="1" nodeid="1"><fence><method name="single"><device name="human"/></method></fence></clusternode></div>
<div> <clusternode name="hv-2" votes="1" nodeid="3"><fence><method name="single"><device name="human"/></method></fence></clusternode></div>
<div> <clusternode name="hv-3" votes="1" nodeid="4"><fence><method name="single"><device name="human"/></method></fence></clusternode></div>
<div> <clusternode name="hv-4" votes="1" nodeid="5"><fence><method name="single"><device name="human"/></method></fence></clusternode></div>
<div> <clusternode name="hv-5" votes="1" nodeid="2"><fence><method name="single"><device name="human"/></method></fence></clusternode></div>
<div> <clusternode name="hv-6" votes="1" nodeid="6"><fence><method name="single"><device name="human"/></method></fence></clusternode></div>
<div> </clusternodes></div><div> <fencedevices></div><div> <fencedevice name="human" agent="manual"/></fencedevices></div><div> <rm/></div><div></cluster></div></div>
<div style="color:rgb(0,0,0)"><br></div><div style="color:rgb(0,0,0)">(manual fencing just for testing)</div><div style="color:rgb(0,0,0)"><br></div><div style="color:rgb(0,0,0)"><br></div><div style="color:rgb(0,0,0)">corosync.conf:</div>
<div style="color:rgb(0,0,0)"><br></div><div style="color:rgb(0,0,0)"><div>compatibility: whitetank</div><div>totem {</div><div> version: 2</div><div> secauth: off</div><div> threads: 0</div><div> # fail_recv_const: 5000</div>
<div> interface {</div><div> ringnumber: 0</div><div> bindnetaddr: 10.14.18.0</div><div> mcastaddr: 239.0.0.4</div><div> mcastport: 5405</div><div> }</div><div>}</div><div>logging {</div><div> fileline: off</div>
<div> to_stderr: no</div><div> to_logfile: yes</div><div> to_syslog: yes</div><div> # the pathname of the log file</div><div> logfile: /var/log/cluster/corosync.log</div><div> debug: off</div><div> timestamp: on</div>
<div> logger_subsys {</div><div> subsys: AMF</div><div> debug: off</div><div> }</div><div>}</div><div><br></div><div>amf {</div><div> mode: disabled</div><div>}</div></div><div style="color:rgb(0,0,0);font-size:medium">
<br></div></font></div></div></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, Feb 22, 2014 at 5:54 AM, Fabio M. Di Nitto <span dir="ltr"><<a href="mailto:fdinitto@redhat.com" target="_blank">fdinitto@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="">On 02/22/2014 10:33 AM, emmanuel segura wrote:<br>
> I know if you need to modify anything outside <rm>... </rm>{used by<br>
> rgmanager} tag in the cluster.conf file, you need to restart the whole<br>
> cluster stack, with cman+rgmanager i have never seen how to add a node<br>
> and remove a node from cluster without restart cman.<br>
<br>
</div>It depends on the version. RHEL5 that's correct, RHEL6 it works also for<br>
outside of <rm> but there are some limitations as some parameters just<br>
can't be changed runtime.<br>
<br>
Fabio<br>
<div class=""><br>
><br>
><br>
><br>
><br>
> 2014-02-22 6:21 GMT+01:00 Bjoern Teipel<br>
> <<a href="mailto:bjoern.teipel@internetbrands.com">bjoern.teipel@internetbrands.com</a><br>
> <mailto:<a href="mailto:bjoern.teipel@internetbrands.com">bjoern.teipel@internetbrands.com</a>>>:<br>
><br>
</div><div><div class="h5">> Hi all,<br>
><br>
> who's using CLVM with CMAN in a cluster with more than 2 nodes in<br>
> production ?<br>
> Did you guys got it to manage to live add a new node to the cluster<br>
> while everything is running ?<br>
> I'm only able to add nodes while the cluster stack is shutdown.<br>
> That's certainly not a good idea when you have to run CLVM on<br>
> hypervisors and you need to shutdown all VMs to add a new box.<br>
> Would be also good if you paste some of your configs using IPMI fencing<br>
><br>
> Thanks in advance,<br>
> Bjoern<br>
><br>
</div></div>> --<br>
> Linux-cluster mailing list<br>
<div class="im HOEnZb">> <a href="mailto:Linux-cluster@redhat.com">Linux-cluster@redhat.com</a> <mailto:<a href="mailto:Linux-cluster@redhat.com">Linux-cluster@redhat.com</a>><br>
> <a href="https://www.redhat.com/mailman/listinfo/linux-cluster" target="_blank">https://www.redhat.com/mailman/listinfo/linux-cluster</a><br>
><br>
><br>
><br>
><br>
> --<br>
> esta es mi vida e me la vivo hasta que dios quiera<br>
><br>
><br>
<br>
</div><div class="HOEnZb"><div class="h5">--<br>
Linux-cluster mailing list<br>
<a href="mailto:Linux-cluster@redhat.com">Linux-cluster@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/linux-cluster" target="_blank">https://www.redhat.com/mailman/listinfo/linux-cluster</a><br>
</div></div></blockquote></div><br></div>