[Linux-cluster] Set time on cluster

[Dryden, Tom]

Hello All

I had a problem when time adjusted that caused node evictions.   The
sequence of events are in the log clip below. 

 The events were, 

1. ntpd restarted

2. ntpd adjusted time  +155 sec

3. openais thinks it has lost communication and begins to evict nodes.

 

My question is how do I adjust the time without causing the cluster to
get confused?

 

Thanks for your assistance. 

Tom

 

 

LOG CLIP:

Oct 22 16:05:02 bpas-j5 ntpd[27399]: Listening on interface bond0:9,
172.26.5.21#123 Enabled

Oct 22 16:05:02 bpas-j5 ntpd[27399]: kernel time sync status 0040

Oct 22 16:05:03 bpas-j5 ntpd[27399]: frequency initialized 40.201 PPM
from /var/lib/ntp/drift

Oct 22 16:08:15 bpas-j5 ntpd[27399]: synchronized to 169.229.70.201,
stratum 3

Oct 22 16:10:50 bpas-j5 ntpd[27399]: time reset +155.075282 s

Oct 22 16:10:50 bpas-j5 ntpd[27399]: kernel time sync enabled 0001

Oct 22 16:10:50 bpas-j5 openais[7238]: [TOTEM] The token was lost in the
OPERATIONAL state.

Oct 22 16:10:50 bpas-j5 openais[7238]: [TOTEM] Receive multicast socket
recv buffer size (288000 bytes).

Oct 22 16:10:50 bpas-j5 openais[7238]: [TOTEM] Transmit multicast socket
send buffer size (262142 bytes).

Oct 22 16:10:50 bpas-j5 openais[7238]: [TOTEM] entering GATHER state
from 2.

Oct 22 16:10:55 bpas-j5 openais[7238]: [TOTEM] entering GATHER state
from 0.

Oct 22 16:10:55 bpas-j5 openais[7238]: [TOTEM] Creating commit token
because I am the rep.

Oct 22 16:10:55 bpas-j5 openais[7238]: [TOTEM] Saving state aru 33 high
seq received 33

Oct 22 16:10:55 bpas-j5 openais[7238]: [TOTEM] Storing new sequence id
for ring 1f48

Oct 22 16:10:55 bpas-j5 openais[7238]: [TOTEM] entering COMMIT state.

Oct 22 16:10:55 bpas-j5 openais[7238]: [TOTEM] entering RECOVERY state.

Oct 22 16:10:55 bpas-j5 openais[7238]: [TOTEM] position [0] member
172.26.5.11:

Oct 22 16:10:55 bpas-j5 openais[7238]: [TOTEM] previous ring seq 8004
rep 172.26.5.11

Oct 22 16:10:55 bpas-j5 openais[7238]: [TOTEM] aru 33 high delivered 33
received flag 1

Oct 22 16:10:55 bpas-j5 openais[7238]: [TOTEM] Did not need to originate
any messages in recovery.

Oct 22 16:10:55 bpas-j5 openais[7238]: [TOTEM] Sending initial ORF token

Oct 22 16:10:55 bpas-j5 openais[7238]: [CLM  ] CLM CONFIGURATION CHANGE

Oct 22 16:10:55 bpas-j5 openais[7238]: [CLM  ] New Configuration:

Oct 22 16:10:55 bpas-j5 kernel: dlm: closing connection to node 2

Oct 22 16:10:55 bpas-j5 openais[7238]: [CLM  ]  r(0) ip(172.26.5.11)

Oct 22 16:10:55 bpas-j5 openais[7238]: [CLM  ] Members Left:

Oct 22 16:10:55 bpas-j5 openais[7238]: [CLM  ]  r(0) ip(172.26.6.11)

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20121023/706de981/attachment.htm>