[Freeipa-users] Windows sync agreement becomes uninitialized and crashes directory server

Tue Jul 14 02:10:50 UTC 2015

On 07/13/2015 07:07 PM, nathan at nathanpeters.com wrote:
> 2 FreeIPA 4.1.4 servers running on CentOS 7.
> dc1 has a sync agreement to a windows server.
>
>
> It has been running fine since June 5 when I re-initialized a sync
> agreement that had somehow uninitialized itself.  Original issue report
> here :
> https://www.redhat.com/archives/freeipa-users/2015-June/msg00147.html
> Bug report here : https://fedorahosted.org/freeipa/ticket/5054
>
> It appears the same thing may have happened again, one month later, but
> this time randomly, as we have not made any changes to our sync agreement
> since the initial change in June.  it appears to have unitialized itself
> without us changing it and managed to crash the directory server in doing
> so.

Crash? http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes
# debuginfo-install 389-ds-base ipa-server slapi-nis

>   Note that during the last week I could still login to the web ui, but
> around the time the log entries change, I became unable to.
>
> I tried to login to the web server today and it would not let me login, so
> I went to the shell on the server and noticed that ipactl command would
> freeze up again.  I looked at the logs (which I will paste below) and
> restarted the directory server service.
>
> I then found that my sync agreement had become uninitialized.
>
> --- output ---
> [root at dc1 slapd-IPADOMAIN-NET]# ldapsearch -xLLL -D "cn=directory manager"
> -W -b cn=config objectclass=nsDSWindowsReplicationAgreement
> Enter LDAP Password:
> dn: cn=meToofficedc2.office.addomain.net,cn=replica,cn=dc\3Dipadomain
>   \2Cdc\3Dnet,cn=mapping tree,cn=config
> nsds7WindowsReplicaSubtree: OU=Staff,DC=office,DC=addomain,DC=net
> nsds7DirectoryReplicaSubtree: cn=users,cn=accounts,dc=ipadomain,dc=net
> cn: meToofficedc2.office.addomain.net
> nsds7NewWinGroupSyncEnabled: false
> objectClass: nsDSWindowsReplicationAgreement
> objectClass: top
> nsDS5ReplicaTransportInfo: TLS
> description: me to officedc2.office.addomain.net
> nsDS5ReplicaRoot: dc=ipadomain,dc=net
> nsDS5ReplicaHost: officedc2.office.addomain.net
> nsds5replicaTimeout: 120
> nsDS5ReplicaBindDN: cn=freeipa syncuser,ou=Service
> Account,dc=office,dc=addomain,dc=net
> nsds7NewWinUserSyncEnabled: true
> nsDS5ReplicaPort: 389
> nsds7WindowsDomain: ipadomain.net
> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof
> idnssoaserial
>    entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount
> nsDS5ReplicaBindMethod: simple
> nsDS5ReplicaCredentials:
> {AES-TUhNR0NTcUdTSWIzRFFFRkRUQm1NRVVHQ1NxR1NJYjNEUUVG
>   RERBNEJDUmtOelUzTTJJNVlpMDBaV1EyTTJRMQ0KWXkwNU0yTm1aV05sTVMxbU5qRXpaak5oTlFBQ
>   0FRSUNBU0F3Q2dZSUtvWklodmNOQWdjd0hRWUpZSVpJQVdVRA0KQkFFcUJCQ2k0N0NxRGZFd2JIdm
>   I0MFVFZVI3MA==}gWI9NIB8lbt9tmNszzbBFCAe4Vs/e0sMyn5+NZPJg9E=
> nsds7DirsyncCookie::
> TVNEUwMAAABoJPGME7jQAQAAAAAAAAAAYAEAAPc1qQAAAAAAAAAAAAAAA
>   AD3NakAAAAAAMUjuImqVZhBkOkdt24C0IsBAAAAAAAAAA4AAAAAAAAAY4GwFkVcvEmMMExrVon4d6
>   13PwAAAAAADGzFNzznrESIxHzA74fbsz4HUgAAAAAAOnFoO5OE2E27lR/g4EcjQTLbIwAAAAAAuEm
>   PWjYok0qGS0HM/+TDmK7FgAMAAAAA6PTFXvAdnkaJSIkZT1lS+/FcJAAAAAAA4qTQaC46/Ua4KXgP
>   /ixNcbK4dgAAAAAAWowbgYD1akibZ+sCul5C4VNsMQAAAAAAxSO4iapVmEGQ6R23bgLQi/c1qQAAA
>   AAAogC6jFcyFUmhBp4B7FkaBcRHwwEAAAAAyhKMxsP0uUKGEnG2lsyA8eTUwgYAAAAA4n8Xx1bAlU
>   mBUl3zhlZ9WBngDAAAAAAA71vM2ebFEkCJkBaLjB4CGU+4CQMAAAAAGfO+4ndZCkaVKnwZNlNsf90
>   NDAAAAAAAgD6n+M2bcUGkOwo5gPLx7IOjAwAAAAAA
> nsds50ruv: {replicageneration} 553fe9bb000000040000
> nsds50ruv: {replica 4 ldap://dc1.ipadomain.net:389} 553fe9c9
>   000000040000 557f49fb000200040000
> nsds50ruv: {replica 3 ldap://dc2.ipadomain.net:389} 553fe9c
>   4000000030000 557f3e4a000200030000
> nsruvReplicaLastModified: {replica 4 ldap://dc1.ipadomain.ne
>   t:389} 557f494a
> nsruvReplicaLastModified: {replica 3 ldap://dc2.ipadomain.n
>   et:389} 557f3d95
> oneWaySync: fromWindows
> nsds5ReplicaEnabled: on
> nsds5replicareapactive: 0
> nsds5replicaLastUpdateStart: 0
> nsds5replicaLastUpdateEnd: 0
> nsds5replicaChangesSentSinceStartup:
> nsds5replicaLastUpdateStatus: -1  - LDAP error: Can't contact LDAP server
> nsds5replicaUpdateInProgress: FALSE
> nsds5replicaLastInitStart: 0
> nsds5replicaLastInitEnd: 0
> --- output ---
>
> Here are the error logs for the last month for the directory server.  They
> are totally empty until July 2.
>
> ---output---
>          389-Directory/1.3.3.8 B2015.040.128
>          dc1.ipadomain.net:636 (/etc/dirsrv/slapd-IPADOMAIN-NET)
>
> [02/Jul/2015:03:19:02 +0000] NSMMReplicationPlugin - windows sync - failed
> to send dirsync search request: 2
> [02/Jul/2015:06:10:29 +0000] - Entry
> "uid=jenkinsdev,cn=users,cn=accounts,dc=ipadomain,dc=net" missing
> attribute "sn" required by object class "person"
> [03/Jul/2015:02:04:02 +0000] NSMMReplicationPlugin - windows sync - failed
> to send dirsync search request: 2
> [03/Jul/2015:05:39:01 +0000] NSMMReplicationPlugin - windows sync - failed
> to send dirsync search request: 2
> [03/Jul/2015:17:09:00 +0000] NSMMReplicationPlugin - windows sync - failed
> to send dirsync search request: 2
> [03/Jul/2015:22:41:32 +0000] slapd_ldap_sasl_interactive_bind - Error:
> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2
> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
> failure.  Minor code may provide more information (Cannot contact any KDC
> for realm 'IPADOMAIN.NET')) errno 115 (Operation now in progress)
> [03/Jul/2015:22:41:32 +0000] slapi_ldap_bind - Error: could not perform
> interactive bind for id [] authentication mechanism [GSSAPI]: error -2
> (Local error)
> [03/Jul/2015:22:41:32 +0000] NSMMReplicationPlugin -
> agmt="cn=meTodc2.ipadomain.net" (dc2:389): Replication bind with GSSAPI
> auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure:
> GSSAPI Error: Unspecified GSS failure.  Minor code may provide more
> information (Cannot contact any KDC for realm 'IPADOMAIN.NET'))
> [03/Jul/2015:22:41:36 +0000] NSMMReplicationPlugin -
> agmt="cn=meTodc2.ipadomain.net" (dc2:389): Replication bind with GSSAPI
> auth resumed
> [05/Jul/2015:19:24:00 +0000] NSMMReplicationPlugin - windows sync - failed
> to send dirsync search request: 2
> [06/Jul/2015:02:46:50 +0000] - Entry
> "uid=accounting,cn=users,cn=accounts,dc=ipadomain,dc=net" missing
> attribute "sn" required by object class "person"
> [06/Jul/2015:17:47:04 +0000] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [06/Jul/2015:17:47:04 +0000] NSMMReplicationPlugin - windows sync -
> agmt="cn=meToofficedc2.office.addomain.net" (officedc2:389): Replication
> bind with SIMPLE auth failed: LDAP error -1 (Can't contact LDAP server)
> ((null))
> [06/Jul/2015:17:47:07 +0000] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [06/Jul/2015:17:47:13 +0000] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [06/Jul/2015:17:47:25 +0000] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
>
> ... repeats for 7 days ...
>
> [13/Jul/2015:21:49:21 +0000] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 2 (No such
> file or directory)
> [13/Jul/2015:21:49:45 +0000] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 2 (No such
> file or directory)
> [13/Jul/2015:21:50:33 +0000] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 2 (No such
> file or directory)
> [13/Jul/2015:21:52:09 +0000] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 2 (No such
> file or directory)
> [13/Jul/2015:21:54:00 +0000] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 2 (No such
> file or directory)
> [13/Jul/2015:23:04:05 +0000] set_krb5_creds - Could not get initial
> credentials for principal [ldap/dc1.ipadomain.net at IPADOMAIN.NET] in keytab
> [FILE:/etc/dirsrv/ds.keytab]: -1765328324 (Generic error (see e-text))
> [13/Jul/2015:23:04:05 +0000] slapd_ldap_sasl_interactive_bind - Error:
> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2
> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
> failure.  Minor code may provide more information (Ticket expired)) errno
> 2 (No such file or directory)
> [13/Jul/2015:23:04:10 +0000] set_krb5_creds - Could not get initial
> credentials for principal [ldap/dc1.ipadomain.net at IPADOMAIN.NET] in keytab
> [FILE:/etc/dirsrv/ds.keytab]: -1765328324 (Generic error (see e-text))
> [13/Jul/2015:23:04:10 +0000] slapd_ldap_sasl_interactive_bind - Error:
> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2
> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
> failure.  Minor code may provide more information (Ticket expired)) errno
> 2 (No such file or directory)
> [13/Jul/2015:23:04:10 +0000] slapi_ldap_bind - Error: could not perform
> interactive bind for id [] authentication mechanism [GSSAPI]: error -2
> (Local error)
> [13/Jul/2015:23:04:10 +0000] NSMMReplicationPlugin -
> agmt="cn=meTodc2.ipadomain.net" (dc2:389): Replication bind with GSSAPI
> auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure:
> GSSAPI Error: Unspecified GSS failure.  Minor code may provide more
> information (Ticket expired))
>
> ---output---
>
>