[Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

[Rob Crittenden]

Timothy Geier wrote:
>
>
> The debug log has a lot of instances of:
>
> Could not connect to LDAP server host xxx.xxxx port 636 Error
> netscape.ldap.LDAPException: IO Error creating JSS SSL Socket (-1)
> Internal Database Error encountered: Could not connect to LDAP server
> host xxx.xxxx port 636 Error netscape.ldap.LDAPException: IO Error
> creating JSS SSL Socket (-1)

The 389-ds access log may show connection attempts and you might be able 
to correlate from there.

> but nothing else of note other than those errors.
>
> We’ve also noticed lots of 500 errors in
> /var/log/pki/pki-tomcat/localhost_access.log
> [08/Feb/2016:10:34:29 -0600] "GET
> /ca/ee/ca/profileSubmit?profileId=caServerCert&serial_num=5&renewal=true&xml=true
> HTTP/1.1" 500 2134
> [08/Feb/2016:10:34:32 -0600] "GET
> /ca/ee/ca/profileSubmit?profileId=caServerCert&serial_num=2&renewal=true&xml=true
> HTTP/1.1" 500 2134
> [08/Feb/2016:10:34:50 -0600] "GET
> /ca/ee/ca/profileSubmit?profileId=caServerCert&serial_num=4&renewal=true&xml=true
> HTTP/1.1" 500 2134
>
> which looks like certmonger is continuously trying to renew the 3 certs.
>
> These dates and times from selftests.log are not accurate and are from
> an earlier attempt to renew the certs while time shifted:
>
> 0.localhost-startStop-1 - [15/Jan/2016:17:39:34 CST] [20] [1]
> SelfTestSubsystem: Initializing self test plugins:
> 0.localhost-startStop-1 - [15/Jan/2016:17:39:34 CST] [20] [1]
> SelfTestSubsystem:  loading all self test plugin logger parameters
> 0.localhost-startStop-1 - [15/Jan/2016:17:39:34 CST] [20] [1]
> SelfTestSubsystem:  loading all self test plugin instances
> 0.localhost-startStop-1 - [15/Jan/2016:17:39:34 CST] [20] [1]
> SelfTestSubsystem:  loading all self test plugin instance parameters
> 0.localhost-startStop-1 - [15/Jan/2016:17:39:34 CST] [20] [1]
> SelfTestSubsystem:  loading self test plugins in on-demand order
> 0.localhost-startStop-1 - [15/Jan/2016:17:39:34 CST] [20] [1]
> SelfTestSubsystem:  loading self test plugins in startup order
> 0.localhost-startStop-1 - [15/Jan/2016:17:39:34 CST] [20] [1]
> SelfTestSubsystem: Self test plugins have been successfully loaded!
> 0.localhost-startStop-1 - [15/Jan/2016:17:39:38 CST] [20] [1]
> SelfTestSubsystem: Running self test plugins specified to be executed at
> startup:
> 0.localhost-startStop-1 - [15/Jan/2016:17:39:38 CST] [20] [1]
> CAPresence:  CA is present
> 0.localhost-startStop-1 - [15/Jan/2016:17:39:38 CST] [20] [1]
> SystemCertsVerification: system certs verification success
> 0.localhost-startStop-1 - [15/Jan/2016:17:39:38 CST] [20] [1]
> SelfTestSubsystem: All CRITICAL self test plugins ran SUCCESSFULLY at
> startup!
>
> There’s nothing in that log with any February dates, so when we try to
> start pki-tomcatd in real time, it's likely not even getting this far.

Some data is also stored in CS.cfg so you might ensure that the 
certificates stored there are correct as well.

There are a few entries in ou=People,o=ipaca that need to reflect the 
current state of certificates as well.

>
>> You mentioned privately that you renamed the IPA host. This is
>> probably what broke half of the renewals. The hosts and keytabs and
>> many entries in IPA have the hostname baked in so you can't simply
>> rename the host.
>>
>
> Technically, the host wasn’t renamed; a new CentOS 7 host was added to
> the existing IPA cluster that had 4 hosts (1 master CA) at CentOS 6
> (using the documentation at
> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/upgrading.html),
> it was promoted to the master CA, all of the C6 hosts were
> decommissioned/removed from replication, and then a new set of C7 hosts
> were created and added as replicas.

Ok, you'll need to look at the host keytabs because something seems 
wrong with them. certmonger can't get a ticket using them. This could be 
because IPA isn't fully running but it is worth a look.

> Is this the correct procedure to follow when time shifted?
>
> - Stop IPA
> - Change the system clock (and the hardware clock) to a point before the
> expiration
> - Start IPA
> - Run getcert resubmit on the appropriate request IDs
> - Stop IPA
> - Return to real time
> - Start IPA
>
> We haven’t tried it this week yet but all attempts at it last week
> failed without any indication as to why the certs weren’t renewing; are
> there any other logs to check/other steps in the procedure?

Yes, that is correct.

rob