[Linux-cluster] cluster.conf not being propagated to all nodes

[Neale Ferguson]

I created a simple two-node cluster (cnode1, cnode2). I also created a linux instance from where I run luci (cmanager). I was able to create the two nodes successfully. On both nodes /etc/cluster/cluster.conf is created. However, any updates to the configuration only are being reflected on one of the nodes (cnode1). The other node (cnode2) is aware that the configuration has been updated but the file is never updated so corosync complains:

Dec 23 11:29:08 corosync [CMAN  ] Unable to load new config in corosync: New configuration version has to be newer than current running configuration

Dec 23 11:29:08 corosync [CMAN  ] Can't get updated config version 2: New configuration version has to be newer than current running configuration
.
Dec 23 11:29:08 corosync [CMAN  ] Activity suspended on this node
Dec 23 11:29:08 corosync [CMAN  ] Error reloading the configuration, will retry every second

I can't see anything in luci.log on cmanager or any of the other log files on cnode2 that gives me any clue as to what's happening.

I have run tcpdump on cmanager and can see traffic flowing to cnode1 but not to cnode2. 

How are the changes to cluster.conf propagated to the nodes within a cluster? What type of things should I look at to help diagnose this problem?

Neale