[Freeipa-users] Me Again

[Ian Harding]

I used to have a lot of replicas, but like a house of cards, it all came
crashing down.

I was down to two, that seemed to be replicating, but last few days I've
noticed that they haven't always been.

freeipa-sea.bpt.rocks is where we do all our admin.
seattlenfs.bpt.rocks is also up and running and can be used for
authentication.

When I noticed that logins were failing after password changes I did

ipa-replica-manage re-initialize --from=freeipa-sea.bpt.rocks

on seattlenfs.bpt.rocks and replication appeared to be working again.

Well it happened again, and this time I peeked at the dirsrv errors log
and saw some scary things having to do with the CA.

[19/Sep/2016:02:55:50 -0700] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-1 (Can't contact LDAP server) ((null)) errno 0 (Success)
[19/Sep/2016:02:55:50 -0700] slapi_ldap_bind - Error: could not perform
interactive bind for id [] authentication mechanism [GSSAPI]: error -1
(Can't contact LDAP server)
[19/Sep/2016:02:55:50 -0700] NSMMReplicationPlugin -
agmt="cn=meTofreeipa-sea.bpt.rocks" (freeipa-sea:389): Replication bind
with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) ()
[19/Sep/2016:02:56:04 -0700] NSMMReplicationPlugin -
agmt="cn=meTofreeipa-sea.bpt.rocks" (freeipa-sea:389): Replication bind
with GSSAPI auth resumed
[20/Sep/2016:10:18:25 -0700] NSMMReplicationPlugin -
multimaster_be_state_change: replica dc=bpt,dc=rocks is going offline;
disabling replication
[20/Sep/2016:10:18:26 -0700] - WARNING: Import is running with
nsslapd-db-private-import-mem on; No other process is allowed to access
the database
[20/Sep/2016:10:18:29 -0700] - import userRoot: Workers finished;
cleaning up...
[20/Sep/2016:10:18:29 -0700] - import userRoot: Workers cleaned up.
[20/Sep/2016:10:18:29 -0700] - import userRoot: Indexing complete.
Post-processing...
[20/Sep/2016:10:18:29 -0700] - import userRoot: Generating
numsubordinates (this may take several minutes to complete)...
[20/Sep/2016:10:18:29 -0700] - import userRoot: Generating
numSubordinates complete.
[20/Sep/2016:10:18:29 -0700] - import userRoot: Gathering ancestorid
non-leaf IDs...
[20/Sep/2016:10:18:29 -0700] - import userRoot: Finished gathering
ancestorid non-leaf IDs.
[20/Sep/2016:10:18:29 -0700] - import userRoot: Creating ancestorid
index (new idl)...
[20/Sep/2016:10:18:29 -0700] - import userRoot: Created ancestorid index
(new idl).
[20/Sep/2016:10:18:29 -0700] - import userRoot: Flushing caches...
[20/Sep/2016:10:18:29 -0700] - import userRoot: Closing files...
[20/Sep/2016:10:18:29 -0700] - import userRoot: Import complete.
Processed 1324 entries in 3 seconds. (441.33 entries/sec)
[20/Sep/2016:10:18:29 -0700] NSMMReplicationPlugin -
multimaster_be_state_change: replica dc=bpt,dc=rocks is coming online;
enabling replication
[20/Sep/2016:10:18:29 -0700] NSMMReplicationPlugin - replica_reload_ruv:
Warning: new data for replica dc=bpt,dc=rocks does not match the data in
the changelog.
 Recreating the changelog file. This could affect replication with
replica's  consumers in which case the consumers should be reinitialized.
[20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target
cn=groups,cn=compat,dc=bpt,dc=rocks does not exist
[20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target
cn=computers,cn=compat,dc=bpt,dc=rocks does not exist
[20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target
cn=ng,cn=compat,dc=bpt,dc=rocks does not exist
[20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target
ou=sudoers,dc=bpt,dc=rocks does not exist
[20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target
cn=users,cn=compat,dc=bpt,dc=rocks does not exist
[20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target
cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist
[20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target
cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist
[20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target
cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist
[20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target
cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist
[20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target
cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist
[20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target
cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist
[20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target
cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist
[20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target
cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist
[20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target
cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist
[20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target
cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist
[20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target
cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist
[20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target
cn=ad,cn=etc,dc=bpt,dc=rocks does not exist
[20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target
cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=bpt,dc=rocks
does not exist
[20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target
cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=bpt,dc=rocks
does not exist

Any clues what that's about?  The high numbered RUV are for CA RUV I
assume.  Those other machines are still installed but IPA services
turned off so people can't authenticate to them because replication was
not working.

If there was some way I could go down to one machine (freeipa-sea) and
get it all cleaned up, no ghost RUV, everything quiet in the logs, and
start over creating replicas, I would love to do that.  Seems like
someone smarter than me could stop the server, back up the ldif files
and edit them to make all the cruft go away.

Is that possible?  I've started a conversation with RedHat about getting
on board with the official bits and support but I want to know if it's
possible/cost effective to do what I describe, along with, I assume,
migrating to the official versions of Spacewalk and FreeIPA.

Thanks!

Ian
-- 
Ian Harding
IT Director
Brown Paper Tickets
1-800-838-3006 ext 7186
http://www.brownpapertickets.com