<div dir="ltr"><br><font face="courier new, monospace">I'm trying to join a new node into an existing 5 node CLVM cluster but I just can't get it to work.<br><br>When ever I add a new node (</font><span style="font-family:'courier new',monospace">I put into the cluster.conf and reloaded with cman_tool version -r -S) </span><font face="courier new, monospace"> I end up with situations like the new node wants to gain the quorum and starts to fence the existing pool master and appears to generate some sort of split cluster. Does it work at all, corosync and dlm do not know about the recently added node ?</font><div>
<font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">New Node </font></div><div><font face="courier new, monospace">==========<br><br><div>Node Sts Inc Joined Name</div>
<div> 1 X 0 hv-b1clcy1</div><div> 2 X 0 hv-b1flcy1</div><div> 3 X 0 hv-b1fmcy1</div><div> 4 X 0 hv-b1dmcy1</div>
<div> 5 X 0 hv-b1fkcy1</div><div> 6 M 80 2014-01-07 21:37:42 hv-b1dkcy1 <--- host added</div><div><br></div><br><div>Jan 7 21:37:42 hv-b1dkcy1 corosync[12564]: [TOTEM ] The network interface [10.14.18.77] is now up.</div>
<div>Jan 7 21:37:42 hv-b1dkcy1 corosync[12564]: [QUORUM] Using quorum provider quorum_cman</div><div>Jan 7 21:37:42 hv-b1dkcy1 corosync[12564]: [SERV ] Service engine loaded: corosync cluster quorum service v0.1</div>
<div>Jan 7 21:37:42 hv-b1dkcy1 corosync[12564]: [CMAN ] CMAN 3.0.12.1 (built Sep 3 2013 09:17:34) started</div><div>Jan 7 21:37:42 hv-b1dkcy1 corosync[12564]: [SERV ] Service engine loaded: corosync CMAN membership service 2.90</div>
<div>Jan 7 21:37:42 hv-b1dkcy1 corosync[12564]: [SERV ] Service engine loaded: openais checkpoint service B.01.01</div><div>Jan 7 21:37:42 hv-b1dkcy1 corosync[12564]: [SERV ] Service engine loaded: corosync extended virtual synchrony service</div>
<div>Jan 7 21:37:42 hv-b1dkcy1 corosync[12564]: [SERV ] Service engine loaded: corosync configuration service</div><div>Jan 7 21:37:42 hv-b1dkcy1 corosync[12564]: [SERV ] Service engine loaded: corosync cluster closed process group service v1.01</div>
<div>Jan 7 21:37:42 hv-b1dkcy1 corosync[12564]: [SERV ] Service engine loaded: corosync cluster config database access v1.01</div><div>Jan 7 21:37:42 hv-b1dkcy1 corosync[12564]: [SERV ] Service engine loaded: corosync profile loading service</div>
<div>Jan 7 21:37:42 hv-b1dkcy1 corosync[12564]: [QUORUM] Using quorum provider quorum_cman</div><div>Jan 7 21:37:42 hv-b1dkcy1 corosync[12564]: [SERV ] Service engine loaded: corosync cluster quorum service v0.1</div>
<div>Jan 7 21:37:42 hv-b1dkcy1 corosync[12564]: [MAIN ] Compatibility mode set to whitetank. Using V1 and V2 of the synchronization engine.</div><div>Jan 7 21:37:42 hv-b1dkcy1 corosync[12564]: [TOTEM ] adding new UDPU member {10.14.18.65}</div>
<div>Jan 7 21:37:42 hv-b1dkcy1 corosync[12564]: [TOTEM ] adding new UDPU member {10.14.18.67}</div><div>Jan 7 21:37:42 hv-b1dkcy1 corosync[12564]: [TOTEM ] adding new UDPU member {10.14.18.68}</div><div>Jan 7 21:37:42 hv-b1dkcy1 corosync[12564]: [TOTEM ] adding new UDPU member {10.14.18.70}</div>
<div>Jan 7 21:37:42 hv-b1dkcy1 corosync[12564]: [TOTEM ] adding new UDPU member {10.14.18.66}</div><div>Jan 7 21:37:42 hv-b1dkcy1 corosync[12564]: [TOTEM ] adding new UDPU member {10.14.18.77}</div><div>Jan 7 21:37:42 hv-b1dkcy1 corosync[12564]: [TOTEM ] A processor joined or left the membership and a new membership was formed.</div>
<div>Jan 7 21:37:42 hv-b1dkcy1 corosync[12564]: [CMAN ] quorum regained, resuming activity</div><div>Jan 7 21:37:42 hv-b1dkcy1 corosync[12564]: [QUORUM] This node is within the primary component and will provide service.</div>
<div>Jan 7 21:37:42 hv-b1dkcy1 corosync[12564]: [QUORUM] Members[1]: 6</div><div>Jan 7 21:37:42 hv-b1dkcy1 corosync[12564]: [QUORUM] Members[1]: 6</div><div>Jan 7 21:37:42 hv-b1dkcy1 corosync[12564]: [CPG ] chosen downlist: sender r(0) ip(10.14.18.77) ; members(old:0 left:0)</div>
<div>Jan 7 21:37:42 hv-b1dkcy1 corosync[12564]: [MAIN ] Completed service synchronization, ready to provide service.</div><div>Jan 7 21:37:46 hv-b1dkcy1 fenced[12620]: fenced 3.0.12.1 started</div><div>Jan 7 21:37:46 hv-b1dkcy1 dlm_controld[12643]: dlm_controld 3.0.12.1 started</div>
<div>Jan 7 21:37:47 hv-b1dkcy1 gfs_controld[12695]: gfs_controld 3.0.12.1 started</div><div>Jan 7 21:37:54 hv-b1dkcy1 fenced[12620]: fencing node hv-b1clcy1</div><div><font face="courier new, monospace"><br></font></div>
<div><font face="courier new, monospace"><div>sudo -i corosync-objctl |grep member</div><div><br></div><div>totem.interface.member.memberaddr=hv-b1clcy1</div><div>totem.interface.member.memberaddr=hv-b1fmcy1</div><div>totem.interface.member.memberaddr=hv-b1dmcy1</div>
<div>totem.interface.member.memberaddr=hv-b1fkcy1</div><div>totem.interface.member.memberaddr=hv-b1flcy1</div><div>totem.interface.member.memberaddr=hv-b1dkcy1</div><div>runtime.totem.pg.mrp.srp.members.6.ip=r(0) ip(10.14.18.77)</div>
<div>runtime.totem.pg.mrp.srp.members.6.join_count=1</div><div>runtime.totem.pg.mrp.srp.members.6.status=joined</div><div><br></div></font></div><div><font face="courier new, monospace"><br></font></div><div><div style="font-family:arial">
<font face="courier new, monospace">Existing Node </font></div><div style="font-family:arial"><font face="courier new, monospace">=============</font></div></div><div style="font-family:arial"><font face="courier new, monospace"><br>
</font></div><div style="font-family:arial"><font face="courier new, monospace">member 6 has not been added to the quorum list :</font></div><div style="font-family:arial"><font face="courier new, monospace"><br></font></div>
</font><div><span style="font-family:'courier new',monospace">Jan 7 21:36:28 hv-b1clcy1 corosync[7769]: [QUORUM] Members[4]: 1 2 3 5</span><br></div><div><font face="courier new, monospace"><div>Jan 7 21:37:54 hv-b1clcy1 corosync[7769]: [TOTEM ] A processor joined or left the membership and a new membership was formed.</div>
<div>Jan 7 21:37:54 hv-b1clcy1 corosync[7769]: [CPG ] chosen downlist: sender r(0) ip(10.14.18.65) ; members(old:4 left:0)</div><div><br></div></font><div><font face="courier new, monospace"><br>Node Sts Inc Joined Name<br>
1 M 4468 2013-12-10 14:33:27 hv-b1clcy1<br> 2 M 4468 2013-12-10 14:33:27 hv-b1flcy1<br> 3 M 5036 2014-01-07 17:51:26 hv-b1fmcy1<br> 4 X 4468 hv-b1dmcy1 (dead at the moment)<br>
5 M 4468 2013-12-10 14:33:27 hv-b1fkcy1<br> 6 X 0 hv-b1dkcy1 <--- added</font><br></div></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace"><div>Jan 7 21:36:28 hv-b1clcy1 corosync[7769]: [QUORUM] Members[4]: 1 2 3 5</div><div>Jan 7 21:37:54 hv-b1clcy1 corosync[7769]: [TOTEM ] A processor joined or left the membership and a new membership was formed.</div>
<div>Jan 7 21:37:54 hv-b1clcy1 corosync[7769]: [CPG ] chosen downlist: sender r(0) ip(10.14.18.65) ; members(old:4 left:0)</div><div>Jan 7 21:37:54 hv-b1clcy1 corosync[7769]: [MAIN ] Completed service synchronization, ready to provide service.</div>
<div><br></div><div><br></div><div><div>totem.interface.member.memberaddr=hv-b1clcy1</div><div>totem.interface.member.memberaddr=hv-b1fmcy1</div><div>totem.interface.member.memberaddr=hv-b1dmcy1</div><div>totem.interface.member.memberaddr=hv-b1fkcy1</div>
<div>totem.interface.member.memberaddr=hv-b1flcy1.</div><div>runtime.totem.pg.mrp.srp.members.1.ip=r(0) ip(10.14.18.65)</div><div>runtime.totem.pg.mrp.srp.members.1.join_count=1</div><div>runtime.totem.pg.mrp.srp.members.1.status=joined</div>
<div>runtime.totem.pg.mrp.srp.members.2.ip=r(0) ip(10.14.18.66)</div><div>runtime.totem.pg.mrp.srp.members.2.join_count=1</div><div>runtime.totem.pg.mrp.srp.members.2.status=joined</div><div>runtime.totem.pg.mrp.srp.members.4.ip=r(0) ip(10.14.18.68)</div>
<div>runtime.totem.pg.mrp.srp.members.4.join_count=1</div><div>runtime.totem.pg.mrp.srp.members.4.status=left</div><div>runtime.totem.pg.mrp.srp.members.5.ip=r(0) ip(10.14.18.70)</div><div>runtime.totem.pg.mrp.srp.members.5.join_count=1</div>
<div>runtime.totem.pg.mrp.srp.members.5.status=joined</div><div>runtime.totem.pg.mrp.srp.members.3.ip=r(0) ip(10.14.18.67)</div><div>runtime.totem.pg.mrp.srp.members.3.join_count=3</div><div>runtime.totem.pg.mrp.srp.members.3.status=joined</div>
</div><div><br></div><div><br></div><div>cluster.conf:</div><div><br></div><div><div><?xml version="1.0"?></div><div><cluster config_version="32" name="hv-1618-110-1"></div><div> <fence_daemon clean_start="0"/></div>
<div> <cman transport="udpu" expected_votes="1"/></div><div> <logging debug="off"/></div><div> <clusternodes></div><div> <clusternode name="hv-b1clcy1" votes="1" nodeid="1"><fence><method name="single"><device name="human"/></method></fence></clusternode></div>
<div> <clusternode name="hv-b1fmcy1" votes="1" nodeid="3"><fence><method name="single"><device name="human"/></method></fence></clusternode></div>
<div> <clusternode name="hv-b1dmcy1" votes="1" nodeid="4"><fence><method name="single"><device name="human"/></method></fence></clusternode></div>
<div> <clusternode name="hv-b1fkcy1" votes="1" nodeid="5"><fence><method name="single"><device name="human"/></method></fence></clusternode></div>
<div> <clusternode name="hv-b1flcy1" votes="1" nodeid="2"><fence><method name="single"><device name="human"/></method></fence></clusternode></div>
<div> <clusternode name="hv-b1dkcy1" votes="1" nodeid="6"><fence><method name="single"><device name="human"/></method></fence></clusternode></div>
<div> </clusternodes></div><div> <fencedevices></div><div> <fencedevice name="human" agent="manual"/></fencedevices></div><div> <rm/></div><div></cluster></div></div>
<div><br></div><div>(manual fencing just for testing)</div><div><br></div><div><br></div><div>corosync.conf:</div><div><br></div><div><div>compatibility: whitetank</div><div>totem {</div><div> version: 2</div><div> secauth: off</div>
<div> threads: 0</div><div> # fail_recv_const: 5000</div><div> interface {</div><div> ringnumber: 0</div><div> bindnetaddr: 10.14.18.0</div><div> mcastaddr: 239.0.0.4</div><div> mcastport: 5405</div><div> }</div>
<div>}</div><div>logging {</div><div> fileline: off</div><div> to_stderr: no</div><div> to_logfile: yes</div><div> to_syslog: yes</div><div> # the pathname of the log file</div><div> logfile: /var/log/cluster/corosync.log</div>
<div> debug: off</div><div> timestamp: on</div><div> logger_subsys {</div><div> subsys: AMF</div><div> debug: off</div><div> }</div><div>}</div><div><br></div><div>amf {</div><div> mode: disabled</div><div>}</div>
</div><div><br></div><div><br></div><div>Many thanks,</div><div>Bjoern</div><div><br></div></font></div></div></div>