[Linux-cluster] qdiskd: change the number of votes while the cluster is up, possible?

[Pena, Francisco Javier]

Hello,

I am doing some tests with a GFS cluster in our lab. As part of the tests, I am trying to upgrade a two-node GFS cluster to three nodes. This usually requires shutting down the cluster, but if we use qdisk I supposed this was not true anymore...

So, I set up a cluster with 2 nodes and a quorum disk. Each node has 1 vote and the quorum disk has 3 votes. Everything runs fine and as expected. Then, I add a third node to the cluster. I assume I have to adjust the votes for the quorum disk accordingly, so I set it to 4. Then, I do the following:

# ccs_tool update /etc/cluster/cluster.conf
# cman_tool version -r <new config version>

And start the cluster services on the third node. So far, so good. Everything looks ok, but then I find that the number of votes for the quorum disk is different, depending on which node I ask. For one of the "old" nodes:

------------------------------------------
# cman_tool status
Protocol version: 5.0.1
Config version: 10
Cluster name: GFS_TEST
Cluster ID: 18942
Cluster Member: Yes
Membership state: Cluster-Member
Nodes: 3
Expected_votes: 3
Total_votes: 6
Quorum: 4
Active subsystems: 6
Node name: gfsnode1
Node ID: 1
Node addresses: X.Y.Z.W

# cman_tool nodes
Node  Votes Exp Sts  Name
   0    3    0   M   /dev/emcpowera1
   1    1    2   M   gfsnode1
   2    1    2   M   gfsnode2
   3    1    3   M   gfsnode3
------------------------------------------

For the "new" node:

------------------------------------------
# cman_tool status
Protocol version: 5.0.1
Config version: 10
Cluster name: GFS_TEST
Cluster ID: 18942
Cluster Member: Yes
Membership state: Cluster-Member
Nodes: 3
Expected_votes: 3
Total_votes: 7
Quorum: 4
Active subsystems: 3
Node name: gfsnode3
Node ID: 3
Node addresses: A.B.C.D

# cman_tool nodes
Node  Votes Exp Sts  Name
   0    4    0   M   /dev/emcpowera1
   1    1    2   M   gfsnode1
   2    1    2   M   gfsnode2
   3    1    3   M   gfsnode3
------------------------------------------

So the new node thinks the quorum disk has 4 votes, while the old nodes still think it has 3. The only way to make the old nodes see that the quorum disk now has 4 votes is to stop and restart their cluster services. Is this the expected behavior? I am not 100% sure if this could be a problem in the long term.

Thanks for any hint, and sorry for the long thread :).

Javier Peña

PS: I have attached the cluster.conf file for the 3-node cluster, just in case anyone wants to see it.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cluster.conf
Type: application/octet-stream
Size: 2008 bytes
Desc: cluster.conf
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20061128/c2786fdc/attachment.obj>