[Linux-cluster] Heuristics for quorum disk used as a tiebreaker in a two node cluster.

[Lon Hohberger]

On Thu, 2010-12-09 at 03:57 +0000, Jankowski, Chris wrote:
> Lon,
> 
> Thank you for your suggestions. 
> 
> 1.
> I like very much your idea of having additional fencing agent (called as the first one in the chain) with delay dependent on the presence of the service on the node.  I understand the code.  What I do not know is what are the steps in adding my own fencing agents. They all live in /usr/sbin.  
> 
> Is it as simple as placing the new fencing agent in /usr/bin?  Is some kind of registration required e.g. so ccs_config_validate will recognise it?

You can put the absolute path in the fencedevice tag:

  <fencedevice agent="/my/custom/agent" ... />

Your agent should not have extra parameters.

Also, I think my first inclination was wrong; you shouldn't combine it
with other devices in the same level.  My apologies.

Instead:

* Your script should *always* exit 1 (failure).  The only thing we want
this script to do is sleep if the service is running on the other guy;
we do not want it to feed fenced any sort of "success" value - ever.

* If you leave it returning 0 and you delete your "real" fencedevice
later, your data will be at risk.

So, make the script return 1 (always) and the cluster.conf would look
like this:


<clusternode>
  <clusternode name="xxx" >
    <fence>
      <method name="delayservice">
        <device name="delayservice" />
      </method>
      <method name="1">
        <device name="ilo" ... />
      </method>
    </fence>
  </clusternode>
  ...
</clusternode>
...
<fencedevices>
  ...
  <fencedevice name="delayservice" agent="/my/custom/agent.sh" />
</fencedevices>
...


> 2.
> I'd guess that the extra fencing agent can also solve the problem of both nodes being fenced when the inter-node link goes down.  This is a distinct from the scenario where the communication through quorum disk ceases.  This will be a bonus.

That's actually what it does...


> 3.
> I am using quorum disk as a natural way to assure that the cluster of 2 nodes has quorum with just one node. I am aware of the <cman two_node="1"/> option. 

Ok.  With the custom agent, you can use pretty much no heuristics.
Qdiskd will auto-configure everything for you (see below though).

However, when using qdiskd for your configuration (where you are using a
custom, extra fencing agent to delay fencing based on service location),
you should explicitly set master_wins to 0.


> What are the advantages or disadvantages of using quorum disk for two nodes compared with no quorum disk and the two_node="1" attribute set?

If you have a cluster where the fence devices are accessible only over
the same network as the cluster communicates, there is no real advantage
to using qdiskd.

If you have the fencing devices on a separate network than the cluster
uses for communication, then using qdiskd can prevent three fencing
problems: a fence race, fence death, and a fencing loop.

http://people.redhat.com/lhh/ClusterPitfalls.pdf

* The delayservice hack eliminates the fencing race.

* Qdiskd holds off fence-loops, but fence-death can still occur in rare
cases when simultaneously starting both cluster nodes from a total
outage.


-- Lon