[Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

Tue Feb 9 19:32:19 UTC 2016

> On Feb 9, 2016, at 2:58 AM, Rob Crittenden <rcritten at redhat.com> wrote:
> 
> Timothy Geier wrote:
>> 
>> 
>> The debug log has a lot of instances of:
>> 
>> Could not connect to LDAP server host xxx.xxxx port 636 Error
>> netscape.ldap.LDAPException: IO Error creating JSS SSL Socket (-1)
>> Internal Database Error encountered: Could not connect to LDAP server
>> host xxx.xxxx port 636 Error netscape.ldap.LDAPException: IO Error
>> creating JSS SSL Socket (-1)
> 
> The 389-ds access log may show connection attempts and you might be able to correlate from there.
> 

[09/Feb/2016:12:55:41 -0600] conn=109598 fd=287 slot=287 SSL connection from master_ip to master_ip
[09/Feb/2016:12:55:41 -0600] conn=109597 op=0 EXT oid="1.3.6.1.4.1.1466.20037" name="startTLS"
[09/Feb/2016:12:55:41 -0600] conn=109597 op=0 RESULT err=0 tag=120 nentries=0 etime=0
[09/Feb/2016:12:55:41 -0600] conn=109598 Netscape Portable Runtime error -8181 (Peer's Certificate has expired.); unauthenticated client CN=CA Subsystem,O=XXXXXXX.NET; issuer CN=Certificate Authority,O=XXXXXXX.NET
[09/Feb/2016:12:55:41 -0600] conn=109598 op=-1 fd=287 closed - Peer's Certificate has expired.

<snip>

> 
> Some data is also stored in CS.cfg so you might ensure that the certificates stored there are correct as well.
> 

CS.cfg is correct/restored from backup.

> There are a few entries in ou=People,o=ipaca that need to reflect the current state of certificates as well.
> 

While looking in cn=config for any o=ipaca entries, the following popped up..it looks like most of the old replicas and the old master hostnames weren’t properly cleaned up and there’s still RUVs for them..nsDS5ReplicaHost and Port also seem to be improperly set:

cn=cloneAgreement1-current-master.XXXXXXXXX.net-pki-tomcat,cn=replica,cn=o\\3Dipaca,cn=mapping tree,cn=config
cn: cloneAgreement1-current-master.XXXXXXXXX.net-pki-tomcat
description: cloneAgreement1-current-master.XXXXXXXXX.net-pki-tomcat
nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-current-master.XXXXXXXXX.net-pki-tomcat,ou=csusers,cn=config
nsDS5ReplicaBindMethod: Simple
nsDS5ReplicaHost: old-master.XXXXXXXXX.net
nsDS5ReplicaPort: 7389
nsDS5ReplicaRoot: o=ipaca
nsDS5ReplicaTransportInfo: TLS
nsds50ruv: {replicageneration} 5304e078000000600000
nsds50ruv: {replica 96 ldap://old-master.XXXXXXXXX.net:7389} 5304e084000000600000 561f3e8f000500600000
nsds50ruv: {replica 81 ldap://current-master.XXXXXXXXX.net:389} 561f361b000000510000 561f369c000000510000
nsds50ruv: {replica 86 ldap://old-replica3.XXXXXXXXX.net:7389} 53f7b70d000000560000 5616ad64000900560000
nsds50ruv: {replica 91 ldap://old-replica2.XXXXXXXXX.net:7389} 53a841e50000005b0000 5616ad5f0005005b0000
nsds50ruv: {replica 97 ldap://old-replica2.XXXXXXXXX.net:7389} 5304e080000000610000 539b4f96000100610000
nsruvReplicaLastModified: {replica 96 ldap://old-master.XXXXXXXXX.net:7389} 00000000
nsruvReplicaLastModified: {replica 81 ldap://current-master.XXXXXXXXX.net:389} 00000000
nsruvReplicaLastModified: {replica 86 ldap://old-replica3.XXXXXXXXX.net:7389} 00000000
nsruvReplicaLastModified: {replica 91 ldap://old-replica2.XXXXXXXXX.net:7389} 00000000
nsruvReplicaLastModified: {replica 97 ldap://old-replica2.XXXXXXXXX.net:7389} 00000000
objectClass: top
objectClass: nsds5replicationagreement
nsds5replicareapactive: 0
nsds5replicaLastUpdateStart: 19700101000000Z
nsds5replicaLastUpdateEnd: 19700101000000Z
nsds5replicaChangesSentSinceStartup:
nsds5replicaLastUpdateStatus: -1 Unable to acquire replicaLDAP error: Can't contact LDAP server
nsds5replicaUpdateInProgress: FALSE
nsds5replicaLastInitStart: 19700101000000Z
nsds5replicaLastInitEnd: 19700101000000Z

cn=replica,cn=o\\3Dipaca,cn=mapping tree,cn=config
cn: replica
nsDS5Flags: 1
nsDS5ReplicaBindDN: cn=Replication Manager cloneAgreement1-current-master.XXXXXXXXX.net-pki-tomcat,ou=csusers,cn=config
nsDS5ReplicaId: 81
nsDS5ReplicaName: 56db714e-72e411e5-87dbe691-e12b51f2
nsDS5ReplicaRoot: o=ipaca
nsDS5ReplicaType: 3
nsState:: UQAAAAAAAADkWbdWAAAAAAAAAAAAAAAAwCYAAAAAAAABAAAAAAAAAA==
objectClass: top
objectClass: nsDS5Replica
objectClass: extensibleobject
nsds5ReplicaChangeCount: 11463
nsds5replicareapactive: 0

ou=People doesn’t seem to exist in either the main tree or cn=config.

>> 
>>> You mentioned privately that you renamed the IPA host. This is
>>> probably what broke half of the renewals. The hosts and keytabs and
>>> many entries in IPA have the hostname baked in so you can't simply
>>> rename the host.
>>> 
>> 
>> Technically, the host wasn’t renamed; a new CentOS 7 host was added to
>> the existing IPA cluster that had 4 hosts (1 master CA) at CentOS 6
>> (using the documentation at
>> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/upgrading.html),
>> it was promoted to the master CA, all of the C6 hosts were
>> decommissioned/removed from replication, and then a new set of C7 hosts
>> were created and added as replicas.
> 
> Ok, you'll need to look at the host keytabs because something seems wrong with them. certmonger can't get a ticket using them. This could be because IPA isn't fully running but it is worth a look.

All of the host keytabs on all of the IPA servers are correct..are there any other keytabs to check?

> 
>> Is this the correct procedure to follow when time shifted?
>> 
>> - Stop IPA
>> - Change the system clock (and the hardware clock) to a point before the
>> expiration
>> - Start IPA
>> - Run getcert resubmit on the appropriate request IDs
>> - Stop IPA
>> - Return to real time
>> - Start IPA
>> 
>> We haven’t tried it this week yet but all attempts at it last week
>> failed without any indication as to why the certs weren’t renewing; are
>> there any other logs to check/other steps in the procedure?
> 
> Yes, that is correct.
> 

Thanks for the clarification.  The IPA cluster itself is still doing ok after about a week of being in this state but something else we’re wondering is if we’ll be able to add replica servers while this is going on..it’s not critical to do so, but it is going to come up fairly soon.

> rob

"This message and any attachments may contain confidential information. If you
have received this  message in error, any use or distribution is prohibited. 
Please notify us by reply e-mail if you have mistakenly received this message,
and immediately and permanently delete it and any attachments. Thank you."