[Linux-cluster] cman_tool nodes shows different Inc numbers; should I be concerned?

[Stewart Walters]

Hello List Members,

I've just joined, so please forgive me in advance if I break some list 
etiquette :-)

I have a two  node cluster (RHEL5) whereby running "cman_tool nodes" on 
each node net's the following results:

[root at node01 ~]# cman_tool nodes
Node  Sts   Inc   Joined               Name
   1   M    512   2009-01-08 10:59:53  node01.example.com
   2   M    516   2009-01-08 10:59:54  node02.example.com

[root at node02 ~]# cman_tool nodes
Node  Sts   Inc   Joined               Name
   1   M    516   2009-01-08 10:59:53  node01.example.com
   2   M    504   2009-01-08 10:35:59  node02.example.com

As you can see the "Inc" numbers are seen as different from both nodes.

First off, should I be concerned that they are different?

And secondly, what is the Inc number signify anyway?  The man page for 
cman_tool doesn't directly describe what an Inc number is for.  I think 
in my travels in trying to answer this question I found a vague 
reference to the fact that it's something to do with openais, but I 
wouldn't mind if someone could confirm this and/or hit me over the head 
with the clue stick.

 From manpage for cman_tools:
<snip>
       Example:

       In this example we have a five node cluster that has experienced 
a net-
       work partition. Here is the output of cman_tool nodes from all 
systems:
       Node  Sts   Inc   Joined               Name
          1   M   2372   2007-11-05 02:58:55  node-01.example.com
          2   d   2376   2007-11-05 02:58:56  node-02.example.com
          3   d   2376   2007-11-05 02:58:56  node-03.example.com
          4   M   2376   2007-11-05 02:58:56  node-04.example.com
          5   M   2376   2007-11-05 02:58:56  node-05.example.com

       Node  Sts   Inc   Joined               Name
          1   d   2372   2007-11-05 02:58:55  node-01.example.com
          2   M   2376   2007-11-05 02:58:56  node-02.example.com
          3   M   2376   2007-11-05 02:58:56  node-03.example.com
          4   d   2376   2007-11-05 02:58:56  node-04.example.com
          5   d   2376   2007-11-05 02:58:56  node-05.example.com

       Node  Sts   Inc   Joined               Name
          1   d   2372   2007-11-05 02:58:55  node-01.example.com
          2   M   2376   2007-11-05 02:58:56  node-02.example.com
          3   M   2376   2007-11-05 02:58:56  node-03.example.com
          4   d   2376   2007-11-05 02:58:56  node-04.example.com
          5   d   2376   2007-11-05 02:58:56  node-05.example.com

       Node  Sts   Inc   Joined               Name
          1   M   2372   2007-11-05 02:58:55  node-01.example.com
          2   d   2376   2007-11-05 02:58:56  node-02.example.com
          3   d   2376   2007-11-05 02:58:56  node-03.example.com
          4   M   2376   2007-11-05 02:58:56  node-04.example.com
          5   M   2376   2007-11-05 02:58:56  node-05.example.com

       Node  Sts   Inc   Joined               Name
          1   M   2372   2007-11-05 02:58:55  node-01.example.com
          2   d   2376   2007-11-05 02:58:56  node-02.example.com
          3   d   2376   2007-11-05 02:58:56  node-03.example.com
          4   M   2376   2007-11-05 02:58:56  node-04.example.com
          5   M   2376   2007-11-05 02:58:56  node-05.example.com
<snip>

At least in the man page example, node-01 consistently has Inc number 
2372, as seen consistently from all nodes.

But as you can see in my cluster, both nodes register a different Inc 
number for themselves and the other.

Thanks in advance for any information you can provide me regarding this.

Kind Regards,

Stewart