Hello list<br><br>I have a problem with a two-node cluster going split-brain. When I first boot the other node, it correctly starts all the services and informs that cluster is quorate. Then when I boot the other node, on the boot phase when it starts the cluster software it does not find the node already running and starts the same services already running on node 1! When the boot is complete I can see that the nodes have found each other for a small period of time but then immediately disconnect from each other. The cluster is created with Conga with shared disk support though no shared disks are created yet. This is on CentOS 5.
<br><br>cluster.conf:<br><br><?xml version="1.0"?><br><cluster alias="testcluster" config_version="11" name="testcluster"><br> <fence_daemon clean_start="0" post_fail_delay="5" post_join_delay="1200"/>
<br> <clusternodes><br> <clusternode name="hume" nodeid="1" votes="1"><br> <fence><br> <method name="2">
<br> <device name="ilohume"/><br> </method><br> </fence><br> </clusternode><br>
<clusternode name="kant" nodeid="2" votes="1"><br> <fence><br> <method name="1"><br> <device name="ilokant"/>
<br> </method><br> </fence><br> </clusternode><br> </clusternodes><br> <cman expected_votes="1" two_node="1"/>
<br> <fencedevices><br> <fencedevice agent="fence_ilo" hostname="x.x.x.x" login="*" name="ilohume" passwd="*"/><br> <fencedevice agent="fence_ilo" hostname="
x.x.x.x" login="*" name="ilokant" passwd="*"/><br> </fencedevices><br> <rm><br> <failoverdomains/><br> <resources/>
<br> <service autostart="1" exclusive="0" name="test" recovery="relocate"><br> <script file="/etc/init.d/pgtest" name="pg"/>
<br> </service><br> <service autostart="1" exclusive="0" name="test2"><br> <script file="/etc/init.d/pgtest2" name="pg2"/>
<br> </service><br> </rm><br></cluster><br><br>clustat & cman_tool status & cman_tool nodes on node already running:<br><br>$ sudo clustat<br>Member Status: Quorate<br><br>Member Name ID Status
<br>------ ---- ---- ------<br>hume 1 Online, Local, rgmanager<br>kant 2 Offline<br><br>Service Name Owner (Last) State<br>
------- ---- ----- ------ -----<br>service:test hume started<br>service:test2 hume started<br><br><br>$ sudo cman_tool status<br>Version: 6.0.1
<br>Config Version: 11<br>Cluster Name: testcluster<br>Cluster Id: 31540<br>Cluster Member: Yes<br>Cluster Generation: 32<br>Membership state: Cluster-Member<br>Nodes: 1<br>Expected votes: 1<br>Total votes: 1<br>Quorum: 1
<br>Active subsystems: 8<br>Flags: 2node<br>Ports Bound: 0 11 177<br>Node name: hume<br>Node ID: 1<br>Multicast addresses: <a href="http://239.192.123.175">239.192.123.175</a><br>Node addresses: <a href="http://193.166.192.100">
193.166.192.100</a><br><br><br>$ sudo cman_tool nodes<br>Node Sts Inc Joined Name<br> 1 M 4 2007-10-10 14:58:53 hume<br> 2 X 28 kant<br><br><br>Here's what gets logged in /var/log/messages
<br><br>Oct 11 07:20:15 hume openais[2410]: [TOTEM] entering GATHER state from 9.<br>Oct 11 07:20:15 hume openais[2410]: [TOTEM] Creating commit token because I am the rep.<br>Oct 11 07:20:15 hume openais[2410]: [TOTEM] Saving state aru 13bd2 high seq received 13bd2
<br>Oct 11 07:20:15 hume openais[2410]: [TOTEM] entering COMMIT state.<br>Oct 11 07:20:15 hume openais[2410]: [TOTEM] entering RECOVERY state.<br>Oct 11 07:20:15 hume openais[2410]: [TOTEM] position [0] member <a href="http://193.166.192.100">
193.166.192.100</a>:<br>Oct 11 07:20:15 hume openais[2410]: [TOTEM] previous ring seq 24 rep <a href="http://193.166.192.100">193.166.192.100</a><br>Oct 11 07:20:15 hume openais[2410]: [TOTEM] aru 13bd2 high delivered 13bd2 received flag 0
<br>Oct 11 07:20:15 hume openais[2410]: [TOTEM] position [1] member <a href="http://193.166.192.101">193.166.192.101</a>:<br>Oct 11 07:20:15 hume openais[2410]: [TOTEM] previous ring seq 4 rep <a href="http://193.166.192.101">
193.166.192.101</a><br>Oct 11 07:20:15 hume openais[2410]: [TOTEM] aru 27 high delivered 27 received flag 0<br>Oct 11 07:20:15 hume openais[2410]: [TOTEM] Did not need to originate any messages in recovery.<br>Oct 11 07:20:15 hume openais[2410]: [TOTEM] Storing new sequence id for ring 1c
<br>Oct 11 07:20:15 hume kernel: dlm: connecting to 2<br>Oct 11 07:20:15 hume openais[2410]: [TOTEM] Sending initial ORF token<br>Oct 11 07:20:15 hume kernel: dlm: got connection from 2<br>Oct 11 07:20:15 hume openais[2410]: [CLM ] CLM CONFIGURATION CHANGE
<br>Oct 11 07:20:15 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:15 hume openais[2410]: [CLM ] New Configuration:<br>Oct 11 07:20:15 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>
Oct 11 07:20:15 hume openais[2410]: [CLM ] r(0) ip(<a href="http://193.166.192.100">193.166.192.100</a>)<br>Oct 11 07:20:15 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:15 hume openais[2410]: [CLM ] Members Left:
<br>Oct 11 07:20:15 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:15 hume openais[2410]: [CLM ] Members Joined:<br>Oct 11 07:20:15 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>
Oct 11 07:20:15 hume openais[2410]: [SYNC ] This node is within the primary component and will provide service.<br>Oct 11 07:20:15 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:15 hume openais[2410]: [CLM ] CLM CONFIGURATION CHANGE
<br>Oct 11 07:20:15 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:15 hume openais[2410]: [CLM ] New Configuration:<br>Oct 11 07:20:15 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>
Oct 11 07:20:15 hume openais[2410]: [CLM ] r(0) ip(<a href="http://193.166.192.100">193.166.192.100</a>)<br>Oct 11 07:20:15 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:15 hume openais[2410]: [CLM ] r(0) ip(
<a href="http://193.166.192.101">193.166.192.101</a>)<br>Oct 11 07:20:15 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:15 hume openais[2410]: [CLM ] Members Left:<br>Oct 11 07:20:15 hume kernel: dlm: lockspace 30002 from 2 type 1 not found
<br>Oct 11 07:20:15 hume openais[2410]: [CLM ] Members Joined:<br>Oct 11 07:20:15 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:15 hume openais[2410]: [CLM ] r(0) ip(<a href="http://193.166.192.101">
193.166.192.101</a>)<br>Oct 11 07:20:15 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:15 hume openais[2410]: [SYNC ] This node is within the primary component and will provide service.<br>Oct 11 07:20:15 hume kernel: dlm: lockspace 20002 from 2 type 1 not found
<br>Oct 11 07:20:15 hume openais[2410]: [TOTEM] entering OPERATIONAL state.<br>Oct 11 07:20:15 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:15 hume openais[2410]: [CLM ] got nodejoin message
<a href="http://193.166.192.100">193.166.192.100</a><br>Oct 11 07:20:15 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:15 hume openais[2410]: [CLM ] got nodejoin message <a href="http://193.166.192.101">
193.166.192.101</a><br>Oct 11 07:20:15 hume openais[2410]: [CPG ] got joinlist message from node 1<br>Oct 11 07:20:15 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:15 hume openais[2410]: [CPG ] got joinlist message from node 2
<br>Oct 11 07:20:15 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:15 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:15 hume kernel: dlm: lockspace 20002 from 2 type 1 not found
<br>Oct 11 07:20:16 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:16 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:16 hume kernel: dlm: lockspace 30002 from 2 type 1 not found
<br>Oct 11 07:20:16 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:16 hume kernel: device eth0 entered promiscuous mode<br>Oct 11 07:20:16 hume kernel: dlm: lockspace 30002 from 2 type 1 not found
<br>Oct 11 07:20:16 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:16 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:16 hume kernel: dlm: lockspace 20002 from 2 type 1 not found
<br>Oct 11 07:20:16 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:16 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:17 hume kernel: dlm: lockspace 30002 from 2 type 1 not found
<br>Oct 11 07:20:17 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:17 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:17 hume kernel: dlm: lockspace 20002 from 2 type 1 not found
<br>Oct 11 07:20:17 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:17 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:18 hume kernel: dlm: lockspace 30002 from 2 type 1 not found
<br>Oct 11 07:20:18 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:18 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:18 hume kernel: dlm: lockspace 20002 from 2 type 1 not found
<br>Oct 11 07:20:18 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:18 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:19 hume kernel: dlm: lockspace 30002 from 2 type 1 not found
<br>Oct 11 07:20:19 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:19 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:19 hume kernel: dlm: lockspace 20002 from 2 type 1 not found
<br>Oct 11 07:20:20 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:20 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:20 hume kernel: dlm: lockspace 30002 from 2 type 1 not found
<br>Oct 11 07:20:20 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:21 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:21 hume kernel: dlm: lockspace 20002 from 2 type 1 not found
<br>Oct 11 07:20:21 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:21 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:22 hume kernel: dlm: lockspace 30002 from 2 type 1 not found
<br>Oct 11 07:20:22 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:22 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:22 hume kernel: dlm: lockspace 20002 from 2 type 1 not found
<br>Oct 11 07:20:23 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:23 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:23 hume kernel: dlm: lockspace 30002 from 2 type 1 not found
<br>Oct 11 07:20:23 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:24 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:24 hume kernel: dlm: lockspace 20002 from 2 type 1 not found
<br>Oct 11 07:20:25 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:25 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:25 hume kernel: dlm: lockspace 30002 from 2 type 1 not found
<br>Oct 11 07:20:25 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:25 hume kernel: device eth0 left promiscuous mode<br>Oct 11 07:20:26 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>
Oct 11 07:20:26 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:27 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:27 hume kernel: dlm: lockspace 20002 from 2 type 1 not found
<br>Oct 11 07:20:27 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:27 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:28 hume kernel: device eth0 entered promiscuous mode
<br>Oct 11 07:20:28 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:28 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:29 hume kernel: dlm: lockspace 30002 from 2 type 1 not found
<br>Oct 11 07:20:29 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:29 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:29 hume kernel: dlm: lockspace 20002 from 2 type 1 not found
<br>Oct 11 07:20:30 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:30 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:31 hume kernel: dlm: lockspace 30002 from 2 type 1 not found
<br>Oct 11 07:20:31 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:32 hume openais[2410]: [TOTEM] The token was lost in the OPERATIONAL state.<br>Oct 11 07:20:32 hume openais[2410]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes).
<br>Oct 11 07:20:32 hume openais[2410]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).<br>Oct 11 07:20:32 hume openais[2410]: [TOTEM] entering GATHER state from 2.<br>Oct 11 07:20:32 hume kernel: dlm: lockspace 30002 from 2 type 1 not found
<br>Oct 11 07:20:32 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:32 hume kernel: device eth0 left promiscuous mode<br>Oct 11 07:20:33 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>
Oct 11 07:20:33 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:34 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:34 hume kernel: dlm: lockspace 20002 from 2 type 1 not found
<br>Oct 11 07:20:34 hume kernel: dlm: lockspace 30002 from 2 type 1 not found<br>Oct 11 07:20:34 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:35 hume kernel: dlm: lockspace 30002 from 2 type 1 not found
<br>Oct 11 07:20:35 hume kernel: dlm: lockspace 20002 from 2 type 1 not found<br>Oct 11 07:20:36 hume kernel: dlm: connecting to 2<br>Oct 11 07:20:36 hume openais[2410]: [TOTEM] entering GATHER state from 0.<br>Oct 11 07:20:36 hume openais[2410]: [TOTEM] Creating commit token because I am the rep.
<br>Oct 11 07:20:36 hume openais[2410]: [TOTEM] Saving state aru 1d high seq received 20<br>Oct 11 07:20:36 hume openais[2410]: [TOTEM] entering COMMIT state.<br>Oct 11 07:20:36 hume openais[2410]: [TOTEM] entering RECOVERY state.
<br>Oct 11 07:20:36 hume openais[2410]: [TOTEM] position [0] member <a href="http://193.166.192.100">193.166.192.100</a>:<br>Oct 11 07:20:36 hume openais[2410]: [TOTEM] previous ring seq 28 rep <a href="http://193.166.192.100">
193.166.192.100</a><br>Oct 11 07:20:36 hume openais[2410]: [TOTEM] aru 1d high delivered 1d received flag 0<br>Oct 11 07:20:36 hume openais[2410]: [TOTEM] copying all old ring messages from 1e-20.<br>Oct 11 07:20:36 hume openais[2410]: [TOTEM] Originated 0 messages in RECOVERY.
<br>Oct 11 07:20:36 hume openais[2410]: [TOTEM] Originated for recovery:<br>Oct 11 07:20:36 hume openais[2410]: [TOTEM] Not Originated for recovery: 1e 1f 20<br>Oct 11 07:20:36 hume openais[2410]: [TOTEM] Storing new sequence id for ring 20
<br>Oct 11 07:20:36 hume kernel: dlm: closing connection to node 2<br>Oct 11 07:20:36 hume openais[2410]: [TOTEM] Sending initial ORF token<br>Oct 11 07:20:37 hume openais[2410]: [CLM ] CLM CONFIGURATION CHANGE<br>Oct 11 07:20:37 hume openais[2410]: [CLM ] New Configuration:
<br>Oct 11 07:20:37 hume openais[2410]: [CLM ] r(0) ip(<a href="http://193.166.192.100">193.166.192.100</a>)<br>Oct 11 07:20:37 hume openais[2410]: [CLM ] Members Left:<br>Oct 11 07:20:37 hume openais[2410]: [CLM ] r(0) ip(
<a href="http://193.166.192.101">193.166.192.101</a>)<br>Oct 11 07:20:37 hume openais[2410]: [CLM ] Members Joined:<br>Oct 11 07:20:37 hume openais[2410]: [SYNC ] This node is within the primary component and will provide service.
<br>Oct 11 07:20:37 hume openais[2410]: [CLM ] CLM CONFIGURATION CHANGE<br>Oct 11 07:20:37 hume openais[2410]: [CLM ] New Configuration:<br>Oct 11 07:20:37 hume openais[2410]: [CLM ] r(0) ip(<a href="http://193.166.192.100">
193.166.192.100</a>)<br>Oct 11 07:20:37 hume openais[2410]: [CLM ] Members Left:<br>Oct 11 07:20:37 hume openais[2410]: [CLM ] Members Joined:<br>Oct 11 07:20:37 hume openais[2410]: [SYNC ] This node is within the primary component and will provide service.
<br>Oct 11 07:20:37 hume openais[2410]: [TOTEM] entering OPERATIONAL state.<br>Oct 11 07:20:37 hume openais[2410]: [CLM ] got nodejoin message <a href="http://193.166.192.100">193.166.192.100</a><br>Oct 11 07:20:37 hume openais[2410]: [CPG ] got joinlist message from node 1
<br><br><br>and on the other node:<br><br><br>Oct 11 07:20:16 kant openais[2411]: [TOTEM] entering GATHER state from 11.<br>Oct 11 07:20:16 kant openais[2411]: [TOTEM] Saving state aru 27 high seq received 27<br>Oct 11 07:20:16 kant openais[2411]: [TOTEM] entering COMMIT state.
<br>Oct 11 07:20:16 kant openais[2411]: [TOTEM] entering RECOVERY state.<br>Oct 11 07:20:16 kant openais[2411]: [TOTEM] position [0] member <a href="http://193.166.192.100">193.166.192.100</a>:<br>Oct 11 07:20:16 kant openais[2411]: [TOTEM] previous ring seq 24 rep
<a href="http://193.166.192.100">193.166.192.100</a><br>Oct 11 07:20:16 kant openais[2411]: [TOTEM] aru 13bd2 high delivered 13bd2 received flag 0<br>Oct 11 07:20:16 kant openais[2411]: [TOTEM] position [1] member <a href="http://193.166.192.101">
193.166.192.101</a>:<br>Oct 11 07:20:16 kant openais[2411]: [TOTEM] previous ring seq 4 rep <a href="http://193.166.192.101">193.166.192.101</a><br>Oct 11 07:20:16 kant openais[2411]: [TOTEM] aru 27 high delivered 27 received flag 0
<br>Oct 11 07:20:16 kant openais[2411]: [TOTEM] Did not need to originate any messages in recovery.<br>Oct 11 07:20:16 kant openais[2411]: [TOTEM] Storing new sequence id for ring 1c<br>Oct 11 07:20:16 kant openais[2411]: [CLM ] CLM CONFIGURATION CHANGE
<br>Oct 11 07:20:16 kant kernel: dlm: connecting to 1<br>Oct 11 07:20:16 kant openais[2411]: [CLM ] New Configuration:<br>Oct 11 07:20:16 kant kernel: dlm: got connection from 1<br>Oct 11 07:20:16 kant openais[2411]: [CLM ] r(0) ip(
<a href="http://193.166.192.101">193.166.192.101</a>)<br>Oct 11 07:20:16 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:16 kant openais[2411]: [CLM ] Members Left:<br>Oct 11 07:20:16 kant kernel: dlm: lockspace 30001 from 1 type 1 not found
<br>Oct 11 07:20:16 kant openais[2411]: [CLM ] Members Joined:<br>Oct 11 07:20:16 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:16 kant openais[2411]: [SYNC ] This node is within the primary component and will provide service.
<br>Oct 11 07:20:16 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:16 kant openais[2411]: [CLM ] CLM CONFIGURATION CHANGE<br>Oct 11 07:20:16 kant kernel: dlm: lockspace 20001 from 1 type 1 not found
<br>Oct 11 07:20:16 kant openais[2411]: [CLM ] New Configuration:<br>Oct 11 07:20:16 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:16 kant openais[2411]: [CLM ] r(0) ip(<a href="http://193.166.192.100">
193.166.192.100</a>)<br>Oct 11 07:20:16 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:16 kant openais[2411]: [CLM ] r(0) ip(<a href="http://193.166.192.101">193.166.192.101</a>)<br>Oct 11 07:20:16 kant kernel: dlm: lockspace 30001 from 1 type 1 not found
<br>Oct 11 07:20:16 kant openais[2411]: [CLM ] Members Left:<br>Oct 11 07:20:16 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:16 kant openais[2411]: [CLM ] Members Joined:<br>Oct 11 07:20:16 kant kernel: dlm: lockspace 30001 from 1 type 1 not found
<br>Oct 11 07:20:16 kant openais[2411]: [CLM ] r(0) ip(<a href="http://193.166.192.100">193.166.192.100</a>)<br>Oct 11 07:20:16 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:16 kant openais[2411]: [SYNC ] This node is within the primary component and will provide service.
<br>Oct 11 07:20:16 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:16 kant openais[2411]: [TOTEM] entering OPERATIONAL state.<br>Oct 11 07:20:16 kant kernel: dlm: lockspace 20001 from 1 type 1 not found
<br>Oct 11 07:20:16 kant openais[2411]: [CLM ] got nodejoin message <a href="http://193.166.192.100">193.166.192.100</a><br>Oct 11 07:20:16 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:16 kant openais[2411]: [CLM ] got nodejoin message
<a href="http://193.166.192.101">193.166.192.101</a><br>Oct 11 07:20:16 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:16 kant openais[2411]: [CPG ] got joinlist message from node 1<br>Oct 11 07:20:16 kant kernel: dlm: lockspace 30001 from 1 type 1 not found
<br>Oct 11 07:20:16 kant openais[2411]: [CPG ] got joinlist message from node 2<br>Oct 11 07:20:16 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:16 kant kernel: dlm: lockspace 30001 from 1 type 1 not found
<br>Oct 11 07:20:16 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:16 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:17 kant kernel: dlm: lockspace 20001 from 1 type 1 not found
<br>Oct 11 07:20:17 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:17 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:17 kant kernel: dlm: lockspace 30001 from 1 type 1 not found
<br>Oct 11 07:20:17 kant openais[2411]: [TOTEM] Retransmit List: 1e<br>Oct 11 07:20:17 kant openais[2411]: [TOTEM] Retransmit List: 1e<br>Oct 11 07:20:17 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f<br>Oct 11 07:20:17 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f
<br>Oct 11 07:20:17 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20<br>Oct 11 07:20:17 kant last message repeated 29 times<br>Oct 11 07:20:17 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:17 kant kernel: dlm: lockspace 20001 from 1 type 1 not found
<br>Oct 11 07:20:17 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20<br>Oct 11 07:20:17 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:17 kant kernel: dlm: lockspace 30001 from 1 type 1 not found
<br>Oct 11 07:20:18 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:18 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:18 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20
<br>Oct 11 07:20:18 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:18 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:18 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20
<br>Oct 11 07:20:18 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:18 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:18 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20
<br>Oct 11 07:20:18 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:18 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:19 kant kernel: dlm: lockspace 20001 from 1 type 1 not found
<br>Oct 11 07:20:19 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:19 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20<br>Oct 11 07:20:19 kant kernel: dlm: lockspace 20001 from 1 type 1 not found
<br>Oct 11 07:20:19 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:19 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20<br>Oct 11 07:20:20 kant kernel: dlm: lockspace 20001 from 1 type 1 not found
<br>Oct 11 07:20:20 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:20 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20<br>Oct 11 07:20:20 kant kernel: dlm: lockspace 20001 from 1 type 1 not found
<br>Oct 11 07:20:20 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:20 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20<br>Oct 11 07:20:20 kant kernel: dlm: lockspace 20001 from 1 type 1 not found
<br>Oct 11 07:20:20 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:20 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20<br>Oct 11 07:20:21 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20
<br>Oct 11 07:20:21 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:21 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:21 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20
<br>Oct 11 07:20:21 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:21 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:22 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20
<br>Oct 11 07:20:22 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:22 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:22 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20
<br>Oct 11 07:20:22 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:22 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:22 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20
<br>Oct 11 07:20:23 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20<br>Oct 11 07:20:23 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:23 kant kernel: dlm: lockspace 30001 from 1 type 1 not found
<br>Oct 11 07:20:23 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20<br>Oct 11 07:20:23 kant openais[2411]: [TOTEM] FAILED TO RECEIVE<br>Oct 11 07:20:23 kant openais[2411]: [TOTEM] entering GATHER state from 6.<br>Oct 11 07:20:23 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20
<br>Oct 11 07:20:23 kant openais[2411]: [TOTEM] FAILED TO RECEIVE<br>Oct 11 07:20:23 kant openais[2411]: [TOTEM] entering GATHER state from 6.<br>Oct 11 07:20:23 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>
Oct 11 07:20:23 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:24 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20<br>Oct 11 07:20:24 kant openais[2411]: [TOTEM] FAILED TO RECEIVE<br>Oct 11 07:20:24 kant openais[2411]: [TOTEM] entering GATHER state from 6.
<br>Oct 11 07:20:24 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:24 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:24 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20
<br>Oct 11 07:20:24 kant openais[2411]: [TOTEM] FAILED TO RECEIVE<br>Oct 11 07:20:24 kant openais[2411]: [TOTEM] entering GATHER state from 6.<br>Oct 11 07:20:24 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>
Oct 11 07:20:24 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:25 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20<br>Oct 11 07:20:25 kant openais[2411]: [TOTEM] FAILED TO RECEIVE<br>Oct 11 07:20:25 kant openais[2411]: [TOTEM] entering GATHER state from 6.
<br>Oct 11 07:20:25 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:25 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:25 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20
<br>Oct 11 07:20:25 kant openais[2411]: [TOTEM] FAILED TO RECEIVE<br>Oct 11 07:20:25 kant openais[2411]: [TOTEM] entering GATHER state from 6.<br>Oct 11 07:20:26 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20<br>Oct 11 07:20:26 kant openais[2411]: [TOTEM] FAILED TO RECEIVE
<br>Oct 11 07:20:26 kant openais[2411]: [TOTEM] entering GATHER state from 6.<br>Oct 11 07:20:26 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:26 kant kernel: dlm: lockspace 30001 from 1 type 1 not found
<br>Oct 11 07:20:26 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20<br>Oct 11 07:20:26 kant openais[2411]: [TOTEM] FAILED TO RECEIVE<br>Oct 11 07:20:26 kant openais[2411]: [TOTEM] entering GATHER state from 6.<br>Oct 11 07:20:26 kant kernel: dlm: lockspace 20001 from 1 type 1 not found
<br>Oct 11 07:20:26 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:27 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20<br>Oct 11 07:20:27 kant openais[2411]: [TOTEM] FAILED TO RECEIVE<br>Oct 11 07:20:27 kant openais[2411]: [TOTEM] entering GATHER state from 6.
<br>Oct 11 07:20:27 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:27 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:27 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20
<br>Oct 11 07:20:27 kant openais[2411]: [TOTEM] FAILED TO RECEIVE<br>Oct 11 07:20:27 kant openais[2411]: [TOTEM] entering GATHER state from 6.<br>Oct 11 07:20:28 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20<br>Oct 11 07:20:28 kant openais[2411]: [TOTEM] FAILED TO RECEIVE
<br>Oct 11 07:20:28 kant openais[2411]: [TOTEM] entering GATHER state from 6.<br>Oct 11 07:20:28 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:28 kant kernel: dlm: lockspace 30001 from 1 type 1 not found
<br>Oct 11 07:20:28 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20<br>Oct 11 07:20:28 kant openais[2411]: [TOTEM] FAILED TO RECEIVE<br>Oct 11 07:20:28 kant openais[2411]: [TOTEM] entering GATHER state from 6.<br>Oct 11 07:20:28 kant kernel: dlm: lockspace 20001 from 1 type 1 not found
<br>Oct 11 07:20:28 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:29 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20<br>Oct 11 07:20:29 kant openais[2411]: [TOTEM] FAILED TO RECEIVE<br>Oct 11 07:20:29 kant openais[2411]: [TOTEM] entering GATHER state from 6.
<br>Oct 11 07:20:29 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:29 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:29 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20
<br>Oct 11 07:20:29 kant openais[2411]: [TOTEM] FAILED TO RECEIVE<br>Oct 11 07:20:29 kant openais[2411]: [TOTEM] entering GATHER state from 6.<br>Oct 11 07:20:30 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20<br>Oct 11 07:20:30 kant openais[2411]: [TOTEM] FAILED TO RECEIVE
<br>Oct 11 07:20:30 kant openais[2411]: [TOTEM] entering GATHER state from 6.<br>Oct 11 07:20:30 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:30 kant kernel: dlm: lockspace 30001 from 1 type 1 not found
<br>Oct 11 07:20:30 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20<br>Oct 11 07:20:30 kant openais[2411]: [TOTEM] FAILED TO RECEIVE<br>Oct 11 07:20:30 kant openais[2411]: [TOTEM] entering GATHER state from 6.<br>Oct 11 07:20:31 kant kernel: dlm: lockspace 20001 from 1 type 1 not found
<br>Oct 11 07:20:31 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:31 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20<br>Oct 11 07:20:31 kant openais[2411]: [TOTEM] FAILED TO RECEIVE<br>Oct 11 07:20:31 kant openais[2411]: [TOTEM] entering GATHER state from 6.
<br>Oct 11 07:20:31 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20<br>Oct 11 07:20:31 kant openais[2411]: [TOTEM] FAILED TO RECEIVE<br>Oct 11 07:20:31 kant openais[2411]: [TOTEM] entering GATHER state from 6.<br>Oct 11 07:20:31 kant kernel: dlm: lockspace 20001 from 1 type 1 not found
<br>Oct 11 07:20:31 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:32 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20<br>Oct 11 07:20:32 kant openais[2411]: [TOTEM] FAILED TO RECEIVE<br>Oct 11 07:20:32 kant openais[2411]: [TOTEM] entering GATHER state from 6.
<br>Oct 11 07:20:32 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20<br>Oct 11 07:20:32 kant openais[2411]: [TOTEM] FAILED TO RECEIVE<br>Oct 11 07:20:32 kant openais[2411]: [TOTEM] entering GATHER state from 6.<br>Oct 11 07:20:32 kant kernel: dlm: lockspace 20001 from 1 type 1 not found
<br>Oct 11 07:20:32 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:33 kant openais[2411]: [TOTEM] Retransmit List: 1e 1f 20<br>Oct 11 07:20:33 kant openais[2411]: [TOTEM] FAILED TO RECEIVE<br>Oct 11 07:20:33 kant openais[2411]: [TOTEM] entering GATHER state from 6.
<br>Oct 11 07:20:33 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:33 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:34 kant kernel: dlm: lockspace 20001 from 1 type 1 not found
<br>Oct 11 07:20:34 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:35 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:35 kant kernel: dlm: lockspace 30001 from 1 type 1 not found
<br>Oct 11 07:20:36 kant kernel: dlm: lockspace 20001 from 1 type 1 not found<br>Oct 11 07:20:36 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:37 kant kernel: dlm: lockspace 20001 from 1 type 1 not found
<br>Oct 11 07:20:37 kant kernel: dlm: lockspace 30001 from 1 type 1 not found<br>Oct 11 07:20:37 kant openais[2411]: [TOTEM] entering GATHER state from 0.<br>Oct 11 07:20:37 kant openais[2411]: [TOTEM] Creating commit token because I am the rep.
<br>Oct 11 07:20:37 kant openais[2411]: [TOTEM] Saving state aru 20 high seq received 20<br>Oct 11 07:20:37 kant openais[2411]: [TOTEM] entering COMMIT state.<br>Oct 11 07:20:37 kant openais[2411]: [TOTEM] entering RECOVERY state.
<br>Oct 11 07:20:37 kant openais[2411]: [TOTEM] position [0] member <a href="http://193.166.192.101">193.166.192.101</a>:<br>Oct 11 07:20:37 kant openais[2411]: [TOTEM] previous ring seq 28 rep <a href="http://193.166.192.100">
193.166.192.100</a><br>Oct 11 07:20:37 kant openais[2411]: [TOTEM] aru 20 high delivered 20 received flag 0<br>Oct 11 07:20:37 kant openais[2411]: [TOTEM] Did not need to originate any messages in recovery.<br>Oct 11 07:20:37 kant openais[2411]: [TOTEM] Storing new sequence id for ring 20
<br>Oct 11 07:20:37 kant openais[2411]: [TOTEM] Sending initial ORF token<br>Oct 11 07:20:37 kant openais[2411]: [CLM ] CLM CONFIGURATION CHANGE<br>Oct 11 07:20:37 kant openais[2411]: [CLM ] New Configuration:<br>Oct 11 07:20:37 kant kernel: dlm: closing connection to node 1
<br>Oct 11 07:20:37 kant openais[2411]: [CLM ] r(0) ip(<a href="http://193.166.192.101">193.166.192.101</a>)<br>Oct 11 07:20:37 kant kernel: dlm: connect from non cluster node<br>Oct 11 07:20:37 kant openais[2411]: [CLM ] Members Left:
<br>Oct 11 07:20:37 kant openais[2411]: [CLM ] r(0) ip(<a href="http://193.166.192.100">193.166.192.100</a>)<br>Oct 11 07:20:38 kant openais[2411]: [CLM ] Members Joined:<br>Oct 11 07:20:38 kant openais[2411]: [SYNC ] This node is within the primary component and will provide service.
<br>Oct 11 07:20:38 kant openais[2411]: [CLM ] CLM CONFIGURATION CHANGE<br>Oct 11 07:20:38 kant openais[2411]: [CLM ] New Configuration:<br>Oct 11 07:20:38 kant openais[2411]: [CLM ] r(0) ip(<a href="http://193.166.192.101">
193.166.192.101</a>)<br>Oct 11 07:20:38 kant openais[2411]: [CLM ] Members Left:<br>Oct 11 07:20:38 kant openais[2411]: [CLM ] Members Joined:<br>Oct 11 07:20:38 kant openais[2411]: [SYNC ] This node is within the primary component and will provide service.
<br>Oct 11 07:20:38 kant openais[2411]: [TOTEM] entering OPERATIONAL state.<br>Oct 11 07:20:38 kant openais[2411]: [CLM ] got nodejoin message <a href="http://193.166.192.101">193.166.192.101</a><br>Oct 11 07:20:38 kant openais[2411]: [CPG ] got joinlist message from node 2
<br>Oct 11 07:21:31 kant snmpd[2664]: Connection from UDP: [<a href="http://193.166.218.61">193.166.218.61</a>]:55646<br>Oct 11 07:21:31 kant snmpd[2664]: Received SNMP packet(s) from UDP: [<a href="http://193.166.218.61">
193.166.218.61</a>]:55646<br>Oct 11 07:21:31 kant snmpd[2664]: Connection from UDP: [<a href="http://193.166.218.61">193.166.218.61</a>]:55646<br>Oct 11 07:21:31 kant snmpd[2664]: Connection from UDP: [<a href="http://193.166.218.61">
193.166.218.61</a>]:55647<br>Oct 11 07:21:31 kant snmpd[2664]: Received SNMP packet(s) from UDP: [<a href="http://193.166.218.61">193.166.218.61</a>]:55647<br>Oct 11 07:21:31 kant snmpd[2664]: Connection from UDP: [<a href="http://193.166.218.61">
193.166.218.61</a>]:55647<br>Oct 11 07:21:31 kant last message repeated 2 times<br>Oct 11 07:21:31 kant snmpd[2664]: Connection from UDP: [<a href="http://193.166.218.61">193.166.218.61</a>]:55646<br>Oct 11 07:21:41 kant ntpd[2696]: synchronized to LOCAL(0), stratum 10
<br>Oct 11 07:21:41 kant ntpd[2696]: kernel time sync enabled 0001<br>Oct 11 07:22:45 kant ntpd[2696]: synchronized to <a href="http://193.166.211.70">193.166.211.70</a>, stratum 2<br>Oct 11 07:26:35 kant snmpd[2664]: Connection from UDP: [
<a href="http://193.166.218.61">193.166.218.61</a>]:56021<br>Oct 11 07:26:35 kant snmpd[2664]: Received SNMP packet(s) from UDP: [<a href="http://193.166.218.61">193.166.218.61</a>]:56021<br>Oct 11 07:26:35 kant snmpd[2664]: Connection from UDP: [
<a href="http://193.166.218.61">193.166.218.61</a>]:56021<br>Oct 11 07:26:35 kant snmpd[2664]: Connection from UDP: [<a href="http://193.166.218.61">193.166.218.61</a>]:56022<br>Oct 11 07:26:35 kant snmpd[2664]: Received SNMP packet(s) from UDP: [
<a href="http://193.166.218.61">193.166.218.61</a>]:56022<br>Oct 11 07:26:35 kant snmpd[2664]: Connection from UDP: [<a href="http://193.166.218.61">193.166.218.61</a>]:56022<br>Oct 11 07:26:35 kant last message repeated 2 times
<br>Oct 11 07:26:35 kant snmpd[2664]: Connection from UDP: [<a href="http://193.166.218.61">193.166.218.61</a>]:56021<br>Oct 11 07:30:20 kant openais[2411]: [TOTEM] entering GATHER state from 11.<br>Oct 11 07:30:20 kant openais[2411]: [TOTEM] Saving state aru 14 high seq received 14
<br>Oct 11 07:30:20 kant openais[2411]: [TOTEM] entering COMMIT state.<br>Oct 11 07:30:20 kant openais[2411]: [TOTEM] entering RECOVERY state.<br>Oct 11 07:30:20 kant openais[2411]: [TOTEM] position [0] member <a href="http://193.166.192.100">
193.166.192.100</a>:<br>Oct 11 07:30:20 kant openais[2411]: [TOTEM] previous ring seq 32 rep <a href="http://193.166.192.100">193.166.192.100</a><br>Oct 11 07:30:20 kant openais[2411]: [TOTEM] aru 15 high delivered 15 received flag 0
<br>Oct 11 07:30:20 kant openais[2411]: [TOTEM] position [1] member <a href="http://193.166.192.101">193.166.192.101</a>:<br>Oct 11 07:30:20 kant openais[2411]: [TOTEM] previous ring seq 32 rep <a href="http://193.166.192.101">
193.166.192.101</a><br>Oct 11 07:30:20 kant openais[2411]: [TOTEM] aru 14 high delivered 14 received flag 0<br>Oct 11 07:30:20 kant openais[2411]: [TOTEM] Did not need to originate any messages in recovery.<br>Oct 11 07:30:20 kant openais[2411]: [TOTEM] Storing new sequence id for ring 24
<br>Oct 11 07:30:20 kant openais[2411]: [CLM ] CLM CONFIGURATION CHANGE<br>Oct 11 07:30:20 kant openais[2411]: [CLM ] New Configuration:<br>Oct 11 07:30:20 kant openais[2411]: [CLM ] r(0) ip(<a href="http://193.166.192.101">
193.166.192.101</a>)<br>Oct 11 07:30:20 kant openais[2411]: [CLM ] Members Left:<br>Oct 11 07:30:20 kant openais[2411]: [CLM ] Members Joined:<br>Oct 11 07:30:20 kant openais[2411]: [SYNC ] This node is within the primary component and will provide service.
<br>Oct 11 07:30:20 kant openais[2411]: [CLM ] CLM CONFIGURATION CHANGE<br>Oct 11 07:30:20 kant openais[2411]: [CLM ] New Configuration:<br>Oct 11 07:30:20 kant openais[2411]: [CLM ] r(0) ip(<a href="http://193.166.192.100">
193.166.192.100</a>)<br>Oct 11 07:30:20 kant openais[2411]: [CLM ] r(0) ip(<a href="http://193.166.192.101">193.166.192.101</a>)<br>Oct 11 07:30:20 kant openais[2411]: [CLM ] Members Left:<br>Oct 11 07:30:20 kant openais[2411]: [CLM ] Members Joined:
<br>Oct 11 07:30:20 kant openais[2411]: [CLM ] r(0) ip(<a href="http://193.166.192.100">193.166.192.100</a>)<br>Oct 11 07:30:20 kant openais[2411]: [SYNC ] This node is within the primary component and will provide service.
<br>Oct 11 07:30:20 kant openais[2411]: [TOTEM] entering OPERATIONAL state.<br>Oct 11 07:30:20 kant openais[2411]: [MAIN ] Killing node hume because it has rejoined the cluster without cman_tool join<br>Oct 11 07:30:20 kant openais[2411]: [CMAN ] cman killed by node 1 for reason 3
<br>Oct 11 07:30:20 kant dlm_controld[2433]: cluster is down, exiting<br>Oct 11 07:30:20 kant kernel: dlm: closing connection to node 2<br>Oct 11 07:30:20 kant gfs_controld[2439]: groupd_dispatch error -1 errno 11<br>Oct 11 07:30:20 kant fenced[2427]: cluster is down, exiting
<br>Oct 11 07:30:20 kant gfs_controld[2439]: groupd connection died<br>Oct 11 07:30:20 kant gfs_controld[2439]: cluster is down, exiting<br>Oct 11 07:30:47 kant ccsd[2403]: Unable to connect to cluster infrastructure after 30 seconds.
<br><br>