<div>Hi Alfredo,</div>
<div> </div>
<div>As you stated, as all the fencing methods will fail, the last resort will be manual fencing, but it is not supported.</div>
<div> </div>
<div>Another way to do so would have been the suicide, but it is what qdisk does when communications with the other nodes and qdisk are lost, but that happens after a short while (a few tens of seconds).</div>
<div> </div>
<div>There is one point I do not completely agree with you in 2-I</div>
<div> </div>
<div>B doesn't realize it is not quorate after the 1 minute timeout, it just gives up after this timeout (tko * interval) . </div>
<div>Depending on the qdisk interval (1s in my case), B already knows it is not quorate anymore after the first qdisk miss occurs (the interval value). </div>
<div> </div>
<div>The concept of having B disk being updated during the short time before tko * interval timeout , then A taking over B services on one mirror leg (not up 2 date) at A and the B modifications are just discarded at resynch kind of disturbs me.</div>
<div> </div>
<div> Let me detail this:</div>
<div> </div>
<div>1) node B hosts a database S made of mirror D for data and mirror L for logs</div>
<div> </div>
<div>2) isolation occurs</div>
<div> </div>
<div>3) B continues (or at least is flushing) to write to D and L which are both half mirrors now (Db and Lb)</div>
<div> </div>
<div>4) regions/sectors Bd and Bi on Db and Lb are upated</div>
<div> </div>
<div>5) qdisk timeout after interval * tko occurs, node is reset</div>
<div> </div>
<div>6) fencing (manual one) succeds (admin ack)</div>
<div> </div>
<div>7) A takes over S with D and L being both half mirrors (Da and La) (Da != Db, La != Lb )</div>
<div> </div>
<div>8) regions/sectors Ad and Ai on Da and La are updated </div>
<div> </div>
<div>9) outage is over, B is back </div>
<div> </div>
<div>10) depending on the mirroring solution, couldn't we end up having a partial resynch from B to A concerning Lb and Db and another partial resynch from A to B concerning La and Da ?</div>
<div> </div>
<div>Array based replication wouldn't be this way as there can be only one replication direction at a time (at leat on the arrays I know), but soft mirroring (mdadm, lvm mirror etc...) may have such multidirectionnal resynch capabilites, no ?</div>
<div> </div>
<div>Brem </div>
<div> </div>
<div> </div>
<div> </div>
<div> </div>
<div> </div>
<div> <br><br> </div>
<div><span class="gmail_quote">2009/9/11, Moralejo, Alfredo <<a href="mailto:alfredo.moralejo@roche.com">alfredo.moralejo@roche.com</a>>:</span>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<div lang="EN-US" vlink="blue" link="blue">
<div>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">Hi,</span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"> </span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">From my point of view the problem is not so related to what does the “bad node” is down but what happen when communications are restored. Let me explain it.</span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"> </span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">1. Let’s start in a 2 nodes clean cluster, each in a different site. Data duplication is done from the host using md or lvm mirror. There is a service running on each node. Qdisk or third node in a third site for quorum.</span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"> </span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">2. Communications are lost in site B (where node B runs.). What happens?, not sure, but as my understanding:</span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"> </span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"> I- Node B will continue working for some time until it realizes it’s not quorated (depending on timeouts, let’s say 1 minute). Data writes on this time are only written to Disks on site B, modifications not written to disks in site A.</span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"> II- Finally, Node B detects it lost qdisk and detects it’s inquorate and rgmanager stops all services running in node B.</span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"> III- In node A, time some time to detect A is dead and never will become inquorate. Services running in node A will continue working, but writes will only be done to disks in disks in site A. Mirror is lost.</span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"> IV- Finally, Node A detects node B is dead and will try to fence it (probably it will need to use manual fence for confirmation).</span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"> V- Until fence is successful, services running originally in node B will not be transferred to node A, so service will be never running simultaneously on both nodes.</span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"> VI- After fence is successful, service starts in node A using disks in site A, without any modification done since the outage until failure is deteced by node B (from I to II). Data modification done from node A are only done to these disks.</span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"> </span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">3. Communications are restored in site B. At this time node B will join the cluster again. Acces to disks in site B is recovered by node A. At this time mirror should be synchronized from disks in site A to site B always, so that, we have a coherent view of the data in both disks, and changes done from node B in the qdisk timeout (from I to II) will be definitively lost.</span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"> </span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">I think this the expected behavior for a multisite cluster in this scenario.</span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"> </span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">Best regards,</span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"> </span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">Alfredo</span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"> </span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"> </span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"> </span></font></p>
<div>
<div style="TEXT-ALIGN: center" align="center"><font face="Times New Roman" size="3"><span lang="ES" style="FONT-SIZE: 12pt">
<hr align="center" width="100%" size="2">
</span></font></div>
<p><b><font face="Tahoma" size="2"><span style="FONT-WEIGHT: bold; FONT-SIZE: 10pt; FONT-FAMILY: Tahoma">From:</span></font></b><font face="Tahoma" size="2"><span style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"> <a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:linux-cluster-bounces@redhat.com" target="_blank">linux-cluster-bounces@redhat.com</a> [mailto:<a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:linux-cluster-bounces@redhat.com" target="_blank">linux-cluster-bounces@redhat.com</a>] <b><span style="FONT-WEIGHT: bold">On Behalf Of </span></b>brem belguebli<br>
<b><span style="FONT-WEIGHT: bold">Sent:</span></b> Thursday, </span></font><font face="Tahoma" size="2"><span lang="ES" style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma">September 10, 2009 11:23 PM<span class="q"><br><b><span style="FONT-WEIGHT: bold">To:</span></b> linux clustering<br>
<b><span style="FONT-WEIGHT: bold">Subject:</span></b> [Linux-cluster] Re: Fencing question in geo cluster (dual sites clustering)</span></span></font><span lang="ES"></span></p></div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt"> </span></font></p>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt">Hi,</span></font></p>
<div><span class="e" id="q_123a869de3a52634_3">
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt"> </span></font></p></div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt">No comments on this RHCS gurus ? Am I trying to setup (multisite cluster) something that 'll never be supported ?</span></font></p></div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt"> </span></font></p></div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt">Or is the qdiskd reboot action considered as sufficient? (Reboot action should be a dirty power reset to prevent data syncing) </span></font></p></div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt"> </span></font></p></div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt">If so, all IO's on the wrong nodes (at the isolated site) should be frozen untill quorum is eventually regained. If not it'll end up with a (dirty) reboot.</span></font></p>
</div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt"> </span></font></p></div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt">Brem </span></font></p></div>
<div>
<p style="MARGIN-BOTTOM: 12pt"><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt"> </span></font></p>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt">2009/8/21 brem belguebli <<a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:brem.belguebli@gmail.com" target="_blank">brem.belguebli@gmail.com</a>></span></font></p>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt">Hi,</span></font></p></div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt"> </span></font></p></div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt">I'm trying to find out what best fencing solution could fit a dual sites cluster.</span></font></p></div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt"> </span></font></p></div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt">Cluster is equally sized on each site (2 nodes/site), each site hosting a SAN array so that each node from any site can see the 2 arrays.</span></font></p>
</div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt"> </span></font></p></div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt">Quorum disk (iscsi LUN) is hosted on a 3rd site.</span></font></p></div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt"> </span></font></p></div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt">SAN and LAN using the same telco infrastructure (2 redundant DWDM loops). </span></font></p></div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt"> </span></font></p></div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt">In case something happens at Telco level (both DWDM loops are broken) that makes 1 of the 2 sites completely isolated from the rest of the world,</span></font></p>
</div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt">the nodes at the good site (the one still operationnal) won't be able to fence any node from the wrong site (the one that is isolated) as there is no way for them to reach their ILO's or do any SAN fencing as the switches at the wrong site are no more reachable.</span></font></p>
</div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt"> </span></font></p></div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt">As qdiskd is not reachable from the wrong nodes, they end up being rebooted by qdisk, but there is a short time (a few seconds) during which the wrong nodes are still seing their local SAN array storage and may potentially have written data on it.</span></font></p>
</div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt"> </span></font></p></div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt">Any ideas or comments on how to ensure data integrity in such setup ?</span></font></p></div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt"> </span></font></p></div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt">Regards</span></font></p></div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt"> </span></font></p></div>
<div>
<p><font face="Times New Roman" color="#888888" size="3"><span style="FONT-SIZE: 12pt; COLOR: #888888">Brem</span></font></p></div></div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt"> </span></font></p></div></span></div></div></div><br>--<br>Linux-cluster mailing list<br><a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:Linux-cluster@redhat.com">Linux-cluster@redhat.com</a><br>
<a onclick="return top.js.OpenExtLink(window,event,this)" href="https://www.redhat.com/mailman/listinfo/linux-cluster" target="_blank">https://www.redhat.com/mailman/listinfo/linux-cluster</a><br></blockquote></div><br>