[Freeipa-users] Certificate system unavailable

Fri Feb 14 13:32:08 UTC 2014

On Fri, January 31, 2014 20:32, Rob Crittenden wrote:
> Sigbjorn Lie wrote:
>
>>
>>
>>
>> On Fri, January 17, 2014 16:37, Rob Crittenden wrote:
>>
>>> Sigbjorn Lie wrote:
>>>
>>>
>>>>
>>>> This worked better than expected. Thank you! :)
>>>>
>>>>
>>>>
>>>> ipa01 and ipa02 seem to be happy again, "getcert list" no longer displays any certificates
>>>> out of date, and all certificates in need of renewal within 28 days has been renewed. The
>>>> webui also started working again and things seem to be back to normal.
>>>>
>>>> ipa03 however is still having issues. I could not renew any certificates on this server to
>>>> begin with, but I managed to renew the certificates for the directory servers by changing
>>>> the xmlrpc url to another ipa server in /etc/ipa/default.conf and resubmitting these
>>>> requests.
>>>>
>>>> "getcert resubmit -i <request-id" says SUBMITTING and the fails with
>>>> NEED_GUIDANCE after a short while for the certificates for the PKI service.
>>>>
>>>>
>>>>
>>>> /var/log/messages says: "certmonger: #033[?1034h28800" and "python:
>>>> Updated certificate for ipaCert not available".
>>>>
>>>>
>>>>
>>>> There is a lot of information in the /var/log/pki-ca/debug, but nothing
>>>> that I can easily distinguish as an error from all the other output. Anything in particular
>>>> I
>>>> should look for?
>>>
>>> Ok, so this is a bug in IPA related to python readline. Garbage is
>>> getting inserted and causing bad things to happen,
>>> https://fedorahosted.org/freeipa/ticket/4064
>>>
>>>
>>>
>>> So the question is, are the certs available or not.
>>>
>>>
>>>
>>> A number of the same certificates are shared amongst all the CAs. One
>>> does the renewal and stuffs the result into cn=ca_renewal,cn=ipa,cn=etc,$SUFFIX. The other CAs
>>>  refer to that location for an updated cert and will load them if they are updated.
>>>
>>> Look to see if the certs are updated there. Given that you have 2
>>> working masters I'm assuming that is the case, so it may just be a matter of fixing the
>>> python.
>>>
>>
>> I could not get anywhere even after manually patching the python script as mentioned in the
>> ticket you provided.
>>
>>
>> I ended up removing and re-adding the replica during a maintenance window. For future
>> reference, what I did was to remove the replica as per the Identity Management Guide on
>> docs.redhat.com. I then re-created the replica installation file and installed the replica.
>>
>> At this point Certmonger managed to retrieve new certificates for the expired certificates, but
>> it kept segfaulting when it attempted to save the certificate to disk. I restarted certmonger a
>> few times, but certmonger just ended up segfaulting over and over. I decided to block the ipa
>> server off the network and change the date back to before the certs expired. After the date was
>> changed I restarted certmonger. Certmonger managed to save the certs successfully this time and
>> a "getcert list" now displays only certificates with an expire date of 2015 or 2016 and a status
>> of MONTORING.
>>
>>
>> I changed the date back to correct date and time and removed the iptables rules. The replica
>> now works just fine.
>>
>> Thank you for your assistance.
>>
>
> Sounds like https://bugzilla.redhat.com/show_bug.cgi?id=1032760
>

It would seem like we're still encountering some issues. The date has now passed for when the old
certificate expired, and the "ipa" cli command no longer works. The webui is still working just
fine.

These are the errors I receive.

$ ipa user-find
ipa: ERROR: cert validation failed for "CN=serveripa03.example.com,O=EXAMPLE.COM"
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked as not trusted by the
user.)
ipa: ERROR: cert validation failed for "CN=serveripa01.example.com,O=EXAMPLE.COM"
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked as not trusted by the
user.)
ipa: ERROR: cert validation failed for "CN=serveripa02.example.com,O=EXAMPLE.COM"
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked as not trusted by the
user.)
ipa: ERROR: cannot connect to Gettext('any of the configured servers', domain='ipa',
localedir=None): https://serveripa03.example.com/ipa/xml, https://serveripa01.example.com/ipa/xml,
https://serveripa02.example.com/ipa/xml

I've been looking through the old threads on the list for similar issues, but these all seem to be
related to the date expiring and the issue is fixed by changing the date back to before the
certificate expired and request a new certificate - which is what we've done as well.

A "getcert list" shows valid certificates on all 3 ipa servers.

$ sudo getcert list|grep expires
	expires: 2016-01-24 20:15:34 UTC
	expires: 2015-12-28 14:25:19 UTC
	expires: 2015-12-28 14:25:56 UTC
	expires: 2015-12-28 14:25:56 UTC
	expires: 2016-01-13 20:21:26 UTC
	expires: 2016-01-24 20:15:32 UTC
	expires: 2016-01-24 20:15:35 UTC
	expires: 2015-12-28 14:25:56 UTC

Somehow I seem to have  2 "Server-Cert" on ipa01. But would that affect all the servers?

$ sudo certutil -L -d /etc/httpd/alias

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

NO.EP.CORP.LOCAL IPA CA                                      CT,C,C
Signing-Cert                                                 u,u,u
Server-Cert                                                  u,u,u
Server-Cert                                                  u,u,u
ipaCert                                                      u,u,u

$ sudo certutil -L -d /etc/httpd/alias -n 'Server-Cert'
----snip----
            Not Before: Thu Jan 19 19:46:07 2012
            Not After : Sun Jan 19 19:46:07 2014
----snip----
and
----snip----
            Not Before: Tue Jan 07 14:28:46 2014
            Not After : Fri Jan 08 14:28:46 2016
----snip----

Any suggestions to where I should continue troubleshooting?

Regards,
Siggi