[Linux-cluster] Quorum disk votes not used when starting node?

[Jos Vos]

Hi,

On (RHEL4 U5) 3-node test cluster I have defined a quorum disk with a
test heuristic as follows:

    <cman expected_votes="6"/>
    ....
    <quorumd interval="1" label="/dev/sda1" min_score="1" tko="10" votes="3">
        <heuristic interval="2" program="test -f /tmp/qdisk" score="1"/>
    </quorumd>

The idea is (when replacing the heuristic with one or more "real"
heuristics) that when the heuristic score is not high enough, the
node considers itself not sane enough to join the cluster.

Now, this all works fine, cman_tool shows what I expected and when I
remove the file /tmp/qdisk on a node, that node reboots instantaneously.

However, after the reboot, while the file tested in the heuristic does
still not exist, the node is joining the cluster again and starts some
cluster services!

Even more strange, the "cman_tool status" output on that node includes:

    Cluster Member: Yes
    Membership state: Cluster-Member
    Nodes: 3
    Expected_votes: 6
    Total_votes: 3
    Quorum: 4

Why is the node a cluster member with only 3 votes out of 6?
The output of "cman_tool nodes" looks like:

    Node  Votes Exp Sts  Name
       0    3    0   X   /dev/sda1
       1    1    6   M   node1
       2    1    6   M   node2
       3    1    6   M   node3
    Member Status: Quorate

While booting, the logging includes this:

Oct 10 15:36:59 node1 ccsd[4125]: Initial status:: Inquorate
Oct 10 15:37:01 node1 kernel: CMAN: sending membership request
Oct 10 15:37:02 node1 kernel: CMAN: got node node2
Oct 10 15:37:02 node1 kernel: CMAN: got node node3
Oct 10 15:37:02 node1 kernel: CMAN: quorum regained, resuming activity
Oct 10 15:37:02 node1 ccsd[4125]: Cluster is quorate.  Allowing connections.
Oct 10 15:37:02 node1 cman: startup succeeded
Oct 10 15:37:02 node1 qdiskd[4232]: <info> Quorum Daemon Initializing
Oct 10 15:37:02 node1 qdiskd: Starting the Quorum Disk Daemon: succeeded
Oct 10 15:37:04 node1 fenced: startup succeeded
Oct 10 15:37:06 node1 clvmd: Cluster LVM daemon started - connected to CMAN
Oct 10 15:37:06 node1 clvmd: clvmd startup succeeded
Oct 10 15:37:13 node1 qdiskd[4232]: <info> Initial score 0/1
Oct 10 15:37:13 node1 qdiskd[4232]: <info> Initialization complete

I would expect that the node would not join the cluster, because the
heuristic score is 0 and thus the 3 votes are not given.

Can someone explain what's wrong?  My config, the behavior shown above,
or something else?

Thanks for any suggestions.

--
--    Jos Vos <jos at xos.nl>
--    X/OS Experts in Open Systems BV   |   Phone: +31 20 6938364
--    Amsterdam, The Netherlands        |     Fax: +31 20 6948204