[Linux-cluster] Network problems

[Patrick Caulfield]

Jon Scottorn wrote:
> Hi,
> 
>    How does a node rejoin the cluster if network connectivity is lost?
> 
> I have 3 nodes that are in the cluster.  I have unplugged the network
> connection from one of the nodes, for testing purposes, the cluster sees
> the node disappear, when I plug the network connection back in it does
> not rejoin the cluster.  I looks as though it forms it's own cluster.  ie...
> 
> cat /proc/cluster/nodes shows two of the nodes with the third missing.
> cat /proc/cluster/nodes on the system that lost network connectivity
> shows itself in the clusterand the other two nodes missing and the
> quorum is locked as well.
> 
> Can this issue be prevented when network connectivity is lost or when
> that happens do I just have to reboot the system?

You'll need to reboot the system - or at least restart all the cluster services
which probably amnounts to the same thing.

When a cluster is partitioned, the partitions (obviously) don't know what has
been going on in the other partition, so it can't just rejoin the cluster
because the whole lockspace/filesystem state may have completely changed.

When network connectivity is re-established the quorate partition (if there is
one) should fence the inquorate one. If it is an even partition where neither
has quorum then they will just stare sullenly at each other for ever, unfortunately.

-- 

patrick