[Linux-cluster] 2 node vs 3 node cluster

[Nathan Dietsch]

Hello Adam,

Adam Cassar wrote:

>Hi Guys,
>
>What are the benefits of running a 3 node cluster, as only one node can
>fail before bringing the entire cluster down?
>
>It appears that a single node cannot be a member of a cluster if the
>other hosts are missing. I take it that is to prevent single nodes
>splitting and making themselves independent clusters? 
>  
>
I think something is missing in your understanding of cluster concepts 
in general and someone please correct me if I am wrong in my explanation.

I am sure others can answer the linux-cluster specific attributes , 
however this is a matter of quorum (finding a majority view of the 
cluster) in general.

It is important that in the case of failure, split-brain scenarios (as 
you pointed out) are avoided. If the number of nodes is even and each 
node has one vote, you face a problem. How this is resolved is 
implementation dependent, but  the explanation below might help;

In other clusters this is handled by allocating a device which both 
machines have access to. (Each machine has a vote, plus the device has a 
vote making an odd number of votes).
When the machines lose sight of each other, they race to grab hold of 
the device and whoever gets it (using SCSI-3 reservations usually) gets 
to remain " in the cluster". The other node is "fenced off" from the 
disks containing the data, usually panics and then reboots, only being 
allowed back into the cluster once it can communicate with its peers.

Quorum can also be handled by allocating a higher-number of votes to a 
specific node (I believe linux-cluster handles things this way from what 
I have read).

So to answer your question. Having a three-node  (or any odd number) 
cluster is ideal because it reduces the complexity of quorum issues. 
However, if all you need is the power of two nodes, properly configuring 
quorum (implementation dependent) can alleviate your problems.

FYI, the notion of quorum is used in other scenarios such as the 
meta-databases in Solaris Volume Manager (formerly Sun Disksuite).  I 
never really understood this one completely, but it does provide an 
example.

I hope this helps, I am sure others will have different and better 
explanations for the linux-cluster specifics. For more general cluster 
information, I recommend Gregory Pfister's book "In Search of Clusters".

Regards,

Nathan Dietsch