[Linux-cluster] NTP time steps causes cluster reconfiguration

[Martin Waite]

Hi,

NTP has a step-threshold - if the time difference is greater than the
threshold, it will step the time rather than speeding it up or down.  So
even using ntpd can cause clock steps (especially in our test
environment where our crappy overloaded NTP servers sometimes lose 30
seconds).

On some VMware test hosts, I did manage to make the cluster fence some
nodes through changing the time backwards and forwards, but I could not
reproduce the effect on physical hosts.     I was hoping that the
fencing was caused by a combination of clock changes and VM guest timing
flakiness, but from your description, it sounds like this might be a
real risk on physical servers too.

I had better do some more testing.

Thanks for the input.

regards,
Martin

> -----Original Message-----
> From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com]
> On Behalf Of Kaloyan Kovachev
> Sent: 16 July 2010 15:36
> To: linux clustering
> Subject: Re: [Linux-cluster] NTP time steps causes cluster
reconfiguration
> 
> Hi,
>  i can confirm, that time steps do cause reconfiguration. Not sure if
this
> was the reason, but one of my nodes was fenced from time to time
> (previously) after several reconfigurations and also it caused some
> problems with gfs being withdrawn.
>  ntpdate running as cron job does step changes, but ntpd should not
cause
> step changes. It should instead speed-up or slow-down the clock until
it is
> synchronized. However using the -g option you may ask that the clock
jumps
> once at the start of ntpd.
>  I have configured all cluster nodes to synchronize from each other
via
> ntpd (configured as peers) and each from one (different) additional
> (startum 1 or 2) source as server. Since then i don't see
reconfiguration
> in the logs.
> 
> On Fri, 16 Jul 2010 14:18:22 +0100, "Martin Waite"
> <Martin.Waite at datacash.com> wrote:
> > Hi,
> >
> >
> >
> > During testing, I noticed that a time step caused by ntpd caused the
> > cluster to drop into GATHER state:
> >
> >
> >
> > Jun 16 12:13:16 cp1edidbm001 ntpd[30917]: time reset -16.332117 s
> >
> > Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] entering GATHER
> > state from 12.
> >
> > Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] Creating commit
> > token because I am the rep.
> >
> > Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] Saving state
aru 9e
> > high seq received 9e
> >
> > Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] Storing new
> > sequence id for ring 328
> >
> > Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] entering COMMIT
> > state.
> >
> > Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] entering
RECOVERY
> > state.
> >
> > ...
> >
> >
> >
> > This is easily repeatable through setting the clock forwards by 20
> > seconds using /bin/date.  This probably causes comms timeouts to
expire
> > prematurely, and almost every time causes the cluster to reconfigure
-
> > luckily without affecting running services.
> >
> >
> >
> > Stepping the clock backwards also causes a similar disruption, but
there
> > is a long lag between changing the time and the cluster
reconfiguring:
> > perhaps this extends a timeout or sleep on the affected node,
causing
> > genuine timeouts on the other nodes.
> >
> >
> >
> > All I am looking for is some reassurance that clock changes are not
> > going to crash the cluster.  Is anyone able to confirm this please ?
> >
> >
> >
> > regards,
> >
> > Martin
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster