<html> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> Please keep freeipa-users in loop Well indeed something bad is happening with replication, did you tried reinitialize replica? Maybe guys from DS will know what is happening. Martin <div class="moz-cite-prefix">On 06.05.2016 21:51, Devin Acosta wrote: </div> <blockquote cite="mid:572CF5B9.70104@pabstatencio.com" type="cite"> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> <div style="font-size: 11pt;font-family: tt;">Martin, Well it initially started when I noticed errors in the logs about having a conflict on a record. So i was trying to get that record cleaned up. I then though oh maybe I should just have it reload everything from another server, and i wonder if now that's why the box is just giving strange results. i had ipa1-i2x.rsinc.local reload from ipa01-aws.rsinc.local, you can see the output of the commands below about replication status. I can still log into ipa1-i2x.rsinc.local, [dacosta@ipa1-i2x ~]$ ipa-replica-manage -v list ipa02-aws.rsinc.local ipa: WARNING: session memcached servers not running ipa01-aws.rsinc.local: replica last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: 0 Replica acquired successfully: Incremental update started last update ended: 1970-01-01 00:00:00+00:00 [dacosta@ipa1-i2x ~]$ ipa-replica-manage -v list ipa01-aws.rsinc.local ipa: WARNING: session memcached servers not running ipa02-aws.rsinc.local: replica last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: 0 Replica acquired successfully: Incremental update succeeded last update ended: 2016-05-06 19:47:26+00:00 ipa1-i2x.rsinc.local: replica last init status: 0 Total update succeeded last init ended: 2016-05-06 18:46:29+00:00 last update status: 0 Replica acquired successfully: Incremental update succeeded last update ended: 2016-05-06 19:46:59+00:00 [dacosta@ipa1-i2x ~]$ ipa-replica-manage -v list ipa1-i2x.rsinc.local ipa: WARNING: session memcached servers not running ipa01-aws.rsinc.local: replica last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: 1 Can't acquire busy replica last update ended: 1970-01-01 00:00:00+00:00 I do have these errors on (idm1-i2x) in the errors: [06/May/2016:18:48:46 +0000] NSMMReplicationPlugin - ruv_compare_ruv: RUV [changelog max RUV] does not contain element [{replica 4 <a class="moz-txt-link-freetext" href="ldap://ipa01-aws.rsinc.local:389">ldap://ipa01-aws.rsinc.local:389</a>} 56e2f9e7000000040000 572ce681000200040000] which is present in RUV [database RUV] [06/May/2016:18:48:46 +0000] NSMMReplicationPlugin - replica_check_for_data_reload: Warning: for replica dc=rsinc,dc=local there were some differences between the changelog max RUV and the database RUV. If there are obsolete elements in the database RUV, you should remove them using the CLEANALLRUV task. If they are not obsolete, you should check their status to see why there are no changes from those servers in the changelog. [06/May/2016:18:48:46 +0000] NSMMReplicationPlugin - ruv_compare_ruv: RUV [changelog max RUV] does not contain element [{replica 91 <a class="moz-txt-link-freetext" href="ldap://ipa1-i2x.rsinc.local:389">ldap://ipa1-i2x.rsinc.local:389</a>} 56f02d3b0000005b0000 56f02d600007005b0000] which is present in RUV [database RUV] [06/May/2016:18:48:46 +0000] NSMMReplicationPlugin - replica_check_for_data_reload: Warning: for replica o=ipaca there were some differences between the changelog max RUV and the database RUV. If there are obsolete elements in the database RUV, you should remove them using the CLEANALLRUV task. If they are not obsolete, you should check their status to see why there are no changes from those servers in the changelog. [06/May/2016:18:48:46 +0000] set_krb5_creds - Could not get initial credentials for principal [<a class="moz-txt-link-abbreviated" href="mailto:ldap/ipa1-i2x.rsinc.local@RSINC.LOCAL">ldap/ipa1-i2x.rsinc.local@RSINC.LOCAL</a>] in keytab [<a class="moz-txt-link-freetext" href="FILE:/etc/dirsrv/ds.keytab">FILE:/etc/dirsrv/ds.keytab</a>]: -1765328324 (Generic error (see e-text)) [06/May/2016:18:48:46 +0000] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available)) errno 0 (Success) [06/May/2016:18:48:46 +0000] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -2 (Local error) [06/May/2016:18:48:46 +0000] NSMMReplicationPlugin - agmt="cn=meToipa01-aws.rsinc.local" (ipa01-aws:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available)) [06/May/2016:18:48:46 +0000] - slapd started. Listening on All Interfaces port 389 for LDAP requests [06/May/2016:18:48:46 +0000] - Listening on All Interfaces port 636 for LDAPS requests [06/May/2016:18:48:46 +0000] - Listening on /var/run/slapd-RSINC-LOCAL.socket for LDAPI requests [06/May/2016:18:48:50 +0000] NSMMReplicationPlugin - agmt="cn=meToipa01-aws.rsinc.local" (ipa01-aws:389): Replication bind with GSSAPI auth resumed [06/May/2016:18:49:18 +0000] - Retry count exceeded in delete [06/May/2016:18:49:18 +0000] DSRetroclPlugin - delete_changerecord: could not delete change record 436145 (rc: 51) Thanks for your help. Martin Basti wrote: <blockquote type="cite"> On 06.05.2016 21:29, Devin Acosta wrote: <blockquote type="cite"> <blockquote type="cite"> I am running the latest FreeIPA on CentOS 7.2. I noticed I had a “nsds5ReplConflict” with an item, i tried to follow the webpage to rename and delete but that failed. I then tried to have ipa1-i2x reload from ipa01-aws instance, now now it seems to have gone maybe worse? can you please advise how to get back to a healthy system. I initially added a system account as recommended so i could have say like Jira/Confluence do User searches against IDM. [dacosta@ipa1-i2x ~]$ ldapsearch -x -D "cn=directory manager" -w ‘password' -b "dc=rsinc,dc=local" "nsds5ReplConflict=*" \* nsds5ReplConflict # extended LDIF # # LDAPv3 # base <dc=rsinc,dc=local> with scope subtree # filter: nsds5ReplConflict=* # requesting: * nsds5ReplConflict # # 7ad08581-059911e6-b55c83a4-93228cdf + ldapsearch, sysaccounts, etc, rsinc.loc al dn: nsuniqueid=7ad08581-059911e6-b55c83a4-93228cdf+uid=ldapsearch,cn=sysaccoun ts,cn=etc,dc=rsinc,dc=local userPassword:: e1NTSEF9M3krdTh5TkdYV= = uid: ldapsearch objectClass: account objectClass: simplesecurityobject objectClass: top nsds5ReplConflict: namingConflict uid=ldapsearch,cn=sysaccounts,cn=etc,dc=rsin c,dc=local # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 [dacosta@ipa1-i2x ~]$ ./ipa_check_consistency -H "ipa1-i2x.local ipa01-aws.rsinc.local" -d RSINC.LOCAL Directory Manager password: FreeIPA servers: ipa1-i2x ipa01-aws STATE =================================================== Active Users ERROR 33 FAIL Stage Users ERROR 0 FAIL Preserved Users ERROR 0 FAIL User Groups ERROR 7 FAIL Hosts ERROR 82 FAIL Host Groups ERROR 1 FAIL HBAC Rules ERROR 2 FAIL SUDO Rules ERROR 4 FAIL DNS Zones ERROR 14 FAIL LDAP Conflicts ERROR YES FAIL Anonymous BIND ERROR on FAIL Replication Status ipa02-aws 0 ipa1-i2x 0 =================================================== [dacosta@ipa1-i2x ~]$ ipa-replica-manage list ipa: WARNING: session memcached servers not running ipa02-aws.rsinc.local: master ipa01-aws.rsinc.local: master ipa1-i2x.rsinc.local: master Devin Acosta Linux Certified Engineer e: <a class="moz-txt-link-abbreviated" href="mailto:devin@linuxguru.co">devin@linuxguru.co</a> </blockquote> </blockquote> hello, it is not clear to me what is wrong, do you have there conflicts? The output of command is not tool supported by freeIPA, I have no idea what is wrong. to check replication status for each IPA server run ipa-replica-manage -v list <hostname> can you kinit on all replicas? can you do ldapsearch as directory manager on each server? Martin</blockquote> </div> </blockquote> </body> </html>