[Freeipa-users] Windows sync agreement becomes uninitialized and crashes directory server

Tue Jul 14 01:07:11 UTC 2015

2 FreeIPA 4.1.4 servers running on CentOS 7.
dc1 has a sync agreement to a windows server.

It has been running fine since June 5 when I re-initialized a sync
agreement that had somehow uninitialized itself.  Original issue report
here :
https://www.redhat.com/archives/freeipa-users/2015-June/msg00147.html
Bug report here : https://fedorahosted.org/freeipa/ticket/5054

It appears the same thing may have happened again, one month later, but
this time randomly, as we have not made any changes to our sync agreement
since the initial change in June.  it appears to have unitialized itself
without us changing it and managed to crash the directory server in doing
so.  Note that during the last week I could still login to the web ui, but
around the time the log entries change, I became unable to.

I tried to login to the web server today and it would not let me login, so
I went to the shell on the server and noticed that ipactl command would
freeze up again.  I looked at the logs (which I will paste below) and
restarted the directory server service.

I then found that my sync agreement had become uninitialized.

--- output ---
[root at dc1 slapd-IPADOMAIN-NET]# ldapsearch -xLLL -D "cn=directory manager"
-W -b cn=config objectclass=nsDSWindowsReplicationAgreement
Enter LDAP Password:
dn: cn=meToofficedc2.office.addomain.net,cn=replica,cn=dc\3Dipadomain
 \2Cdc\3Dnet,cn=mapping tree,cn=config
nsds7WindowsReplicaSubtree: OU=Staff,DC=office,DC=addomain,DC=net
nsds7DirectoryReplicaSubtree: cn=users,cn=accounts,dc=ipadomain,dc=net
cn: meToofficedc2.office.addomain.net
nsds7NewWinGroupSyncEnabled: false
objectClass: nsDSWindowsReplicationAgreement
objectClass: top
nsDS5ReplicaTransportInfo: TLS
description: me to officedc2.office.addomain.net
nsDS5ReplicaRoot: dc=ipadomain,dc=net
nsDS5ReplicaHost: officedc2.office.addomain.net
nsds5replicaTimeout: 120
nsDS5ReplicaBindDN: cn=freeipa syncuser,ou=Service
Account,dc=office,dc=addomain,dc=net
nsds7NewWinUserSyncEnabled: true
nsDS5ReplicaPort: 389
nsds7WindowsDomain: ipadomain.net
nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof
idnssoaserial
  entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount
nsDS5ReplicaBindMethod: simple
nsDS5ReplicaCredentials:
{AES-TUhNR0NTcUdTSWIzRFFFRkRUQm1NRVVHQ1NxR1NJYjNEUUVG
 RERBNEJDUmtOelUzTTJJNVlpMDBaV1EyTTJRMQ0KWXkwNU0yTm1aV05sTVMxbU5qRXpaak5oTlFBQ
 0FRSUNBU0F3Q2dZSUtvWklodmNOQWdjd0hRWUpZSVpJQVdVRA0KQkFFcUJCQ2k0N0NxRGZFd2JIdm
 I0MFVFZVI3MA==}gWI9NIB8lbt9tmNszzbBFCAe4Vs/e0sMyn5+NZPJg9E=
nsds7DirsyncCookie::
TVNEUwMAAABoJPGME7jQAQAAAAAAAAAAYAEAAPc1qQAAAAAAAAAAAAAAA
 AD3NakAAAAAAMUjuImqVZhBkOkdt24C0IsBAAAAAAAAAA4AAAAAAAAAY4GwFkVcvEmMMExrVon4d6
 13PwAAAAAADGzFNzznrESIxHzA74fbsz4HUgAAAAAAOnFoO5OE2E27lR/g4EcjQTLbIwAAAAAAuEm
 PWjYok0qGS0HM/+TDmK7FgAMAAAAA6PTFXvAdnkaJSIkZT1lS+/FcJAAAAAAA4qTQaC46/Ua4KXgP
 /ixNcbK4dgAAAAAAWowbgYD1akibZ+sCul5C4VNsMQAAAAAAxSO4iapVmEGQ6R23bgLQi/c1qQAAA
 AAAogC6jFcyFUmhBp4B7FkaBcRHwwEAAAAAyhKMxsP0uUKGEnG2lsyA8eTUwgYAAAAA4n8Xx1bAlU
 mBUl3zhlZ9WBngDAAAAAAA71vM2ebFEkCJkBaLjB4CGU+4CQMAAAAAGfO+4ndZCkaVKnwZNlNsf90
 NDAAAAAAAgD6n+M2bcUGkOwo5gPLx7IOjAwAAAAAA
nsds50ruv: {replicageneration} 553fe9bb000000040000
nsds50ruv: {replica 4 ldap://dc1.ipadomain.net:389} 553fe9c9
 000000040000 557f49fb000200040000
nsds50ruv: {replica 3 ldap://dc2.ipadomain.net:389} 553fe9c
 4000000030000 557f3e4a000200030000
nsruvReplicaLastModified: {replica 4 ldap://dc1.ipadomain.ne
 t:389} 557f494a
nsruvReplicaLastModified: {replica 3 ldap://dc2.ipadomain.n
 et:389} 557f3d95
oneWaySync: fromWindows
nsds5ReplicaEnabled: on
nsds5replicareapactive: 0
nsds5replicaLastUpdateStart: 0
nsds5replicaLastUpdateEnd: 0
nsds5replicaChangesSentSinceStartup:
nsds5replicaLastUpdateStatus: -1  - LDAP error: Can't contact LDAP server
nsds5replicaUpdateInProgress: FALSE
nsds5replicaLastInitStart: 0
nsds5replicaLastInitEnd: 0
--- output ---

Here are the error logs for the last month for the directory server.  They
are totally empty until July 2.

---output---
        389-Directory/1.3.3.8 B2015.040.128
        dc1.ipadomain.net:636 (/etc/dirsrv/slapd-IPADOMAIN-NET)

[02/Jul/2015:03:19:02 +0000] NSMMReplicationPlugin - windows sync - failed
to send dirsync search request: 2
[02/Jul/2015:06:10:29 +0000] - Entry
"uid=jenkinsdev,cn=users,cn=accounts,dc=ipadomain,dc=net" missing
attribute "sn" required by object class "person"
[03/Jul/2015:02:04:02 +0000] NSMMReplicationPlugin - windows sync - failed
to send dirsync search request: 2
[03/Jul/2015:05:39:01 +0000] NSMMReplicationPlugin - windows sync - failed
to send dirsync search request: 2
[03/Jul/2015:17:09:00 +0000] NSMMReplicationPlugin - windows sync - failed
to send dirsync search request: 2
[03/Jul/2015:22:41:32 +0000] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2
(Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
failure.  Minor code may provide more information (Cannot contact any KDC
for realm 'IPADOMAIN.NET')) errno 115 (Operation now in progress)
[03/Jul/2015:22:41:32 +0000] slapi_ldap_bind - Error: could not perform
interactive bind for id [] authentication mechanism [GSSAPI]: error -2
(Local error)
[03/Jul/2015:22:41:32 +0000] NSMMReplicationPlugin -
agmt="cn=meTodc2.ipadomain.net" (dc2:389): Replication bind with GSSAPI
auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure:
GSSAPI Error: Unspecified GSS failure.  Minor code may provide more
information (Cannot contact any KDC for realm 'IPADOMAIN.NET'))
[03/Jul/2015:22:41:36 +0000] NSMMReplicationPlugin -
agmt="cn=meTodc2.ipadomain.net" (dc2:389): Replication bind with GSSAPI
auth resumed
[05/Jul/2015:19:24:00 +0000] NSMMReplicationPlugin - windows sync - failed
to send dirsync search request: 2
[06/Jul/2015:02:46:50 +0000] - Entry
"uid=accounting,cn=users,cn=accounts,dc=ipadomain,dc=net" missing
attribute "sn" required by object class "person"
[06/Jul/2015:17:47:04 +0000] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[06/Jul/2015:17:47:04 +0000] NSMMReplicationPlugin - windows sync -
agmt="cn=meToofficedc2.office.addomain.net" (officedc2:389): Replication
bind with SIMPLE auth failed: LDAP error -1 (Can't contact LDAP server)
((null))
[06/Jul/2015:17:47:07 +0000] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[06/Jul/2015:17:47:13 +0000] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[06/Jul/2015:17:47:25 +0000] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)

... repeats for 7 days ...

[13/Jul/2015:21:49:21 +0000] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 2 (No such
file or directory)
[13/Jul/2015:21:49:45 +0000] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 2 (No such
file or directory)
[13/Jul/2015:21:50:33 +0000] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 2 (No such
file or directory)
[13/Jul/2015:21:52:09 +0000] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 2 (No such
file or directory)
[13/Jul/2015:21:54:00 +0000] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 2 (No such
file or directory)
[13/Jul/2015:23:04:05 +0000] set_krb5_creds - Could not get initial
credentials for principal [ldap/dc1.ipadomain.net at IPADOMAIN.NET] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328324 (Generic error (see e-text))
[13/Jul/2015:23:04:05 +0000] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2
(Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
failure.  Minor code may provide more information (Ticket expired)) errno
2 (No such file or directory)
[13/Jul/2015:23:04:10 +0000] set_krb5_creds - Could not get initial
credentials for principal [ldap/dc1.ipadomain.net at IPADOMAIN.NET] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328324 (Generic error (see e-text))
[13/Jul/2015:23:04:10 +0000] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2
(Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
failure.  Minor code may provide more information (Ticket expired)) errno
2 (No such file or directory)
[13/Jul/2015:23:04:10 +0000] slapi_ldap_bind - Error: could not perform
interactive bind for id [] authentication mechanism [GSSAPI]: error -2
(Local error)
[13/Jul/2015:23:04:10 +0000] NSMMReplicationPlugin -
agmt="cn=meTodc2.ipadomain.net" (dc2:389): Replication bind with GSSAPI
auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure:
GSSAPI Error: Unspecified GSS failure.  Minor code may provide more
information (Ticket expired))

---output---