[Freeipa-users] Certificate system unavailable

[Sigbjorn Lie]

On 13/01/14 19:37, Rob Crittenden wrote:
> Sigbjorn Lie wrote:
>>
>>
>>
>> On Mon, January 13, 2014 16:34, Rob Crittenden wrote:
>>> Sigbjorn Lie wrote:
>>>
>>>>
>>>>
>>>>
>>>> On Mon, January 13, 2014 15:58, Rob Crittenden wrote:
>>>>
>>>>> Sigbjorn Lie wrote:
>>>>>
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> I seem to have issues with the certificate system on my IPA 
>>>>>> installation. Looking up hosts
>>>>>> in the IPA WEBUI on any of the IPA servers says "Certificate 
>>>>>> format error: [Errno -8015]
>>>>>> error (-8015)
>>>>>> unknown".
>>>>>>
>>>>>> I also notice that hosts says the certificate system is unavailable.
>>>>>>
>>>>>>
>>>>>>
>>>>>> certmonger: Server failed request, will retry: 4301 (RPC failed 
>>>>>> at server.  Certificate
>>>>>> operation cannot be completed: Failure decoding Certificate 
>>>>>> Signing Request).
>>>>>>
>>>>>>
>>>>>> Looking at the pki-ca logs on the ipa servers I see that some 
>>>>>> selftest failed:
>>>>>>
>>>>>>
>>>>>>
>>>>>> # tail -100 selftests.log
>>>>>> 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] 
>>>>>> SelfTestSubsystem: Initializing self test
>>>>>> plugins:
>>>>>> 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] 
>>>>>> SelfTestSubsystem:  loading all self test
>>>>>> plugin logger parameters 28697.main - [13/Jan/2014:15:06:33 CET] 
>>>>>> [20] [1] SelfTestSubsystem:
>>>>>>   loading all self test plugin instances 28697.main - 
>>>>>> [13/Jan/2014:15:06:33 CET] [20] [1]
>>>>>> SelfTestSubsystem:  loading all self test plugin
>>>>>> instance parameters 28697.main - [13/Jan/2014:15:06:33 CET] [20] 
>>>>>> [1] SelfTestSubsystem:
>>>>>> loading self test plugins in on-demand order 28697.main - 
>>>>>> [13/Jan/2014:15:06:33 CET] [20]
>>>>>> [1]
>>>>>> SelfTestSubsystem:  loading self test plugins in
>>>>>> startup order 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] 
>>>>>> SelfTestSubsystem: Self test
>>>>>> plugins have been successfully loaded! 28697.main - 
>>>>>> [13/Jan/2014:15:06:34 CET] [20] [1]
>>>>>> SelfTestSubsystem: Running self test plugins
>>>>>> specified to be executed at startup: 28697.main - 
>>>>>> [13/Jan/2014:15:06:34 CET] [20] [1]
>>>>>> CAPresence:
>>>>>> CA is present
>>>>>> 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] 
>>>>>> SystemCertsVerification: system certs
>>>>>> verification failure 28697.main - [13/Jan/2014:15:06:34 CET] [20] 
>>>>>> [1] SelfTestSubsystem: The
>>>>>>   CRITICAL self test plugin
>>>>>> called selftests.container.instance.SystemCertsVerification 
>>>>>> running at startup FAILED!
>>>>>>
>>>>>> the pki-cad service is running and "pki-cad status" displays the 
>>>>>> ports available.
>>>>>> /etc/init.d/pki-cad status
>>>>>> pki-ca (pid 28697) is running...                           [  OK  ]
>>>>>>
>>>>>>
>>>>>> My main consern is that the certmonger requests for renew of 
>>>>>> certificates for LDAP on 2 out
>>>>>> of 3
>>>>>> of the IPA servers has failed, and the current certificate is 
>>>>>> expiring the 19th of January,
>>>>>> under a week from now.
>>>>>>
>>>>>> Do you have any suggestions to where I can start troubleshootng 
>>>>>> this issue?
>>>>>>
>>>>>>
>>>>>
>>>>> Check the trust on the audit certificate:
>>>>>
>>>>>
>>>>>
>>>>> # certutil -L -d /var/lib/pki-ca/alias/
>>>>> ...
>>>>> auditSigningCert cert-pki-ca                                 u,u,Pu
>>>>>
>>>>> If the trust is not u,u,Pu then you can fix it with:
>>>>>
>>>>>
>>>>>
>>>>> # certutil -M -d /var/lib/pki-ca/alias -n 'auditSigningCert 
>>>>> cert-pki-ca'
>>>>> -t u,u,Pu
>>>>>
>>>>>
>>>>>
>>>>> Then restart the CA and it should be ok.
>>>>>
>>>>>
>>>>
>>>> Looks like this certificate is expired. This is the same output on 
>>>> all 3 of the ipa servers.
>>>>
>>>>
>>>> How can this be fixed?
>>>>
>>>>
>>>>
>>>> # certutil -L -d /var/lib/pki-ca/alias/ -n "auditSigningCert 
>>>> cert-pki-ca"
>>>> Certificate:
>>>> Data:
>>>> Version: 3 (0x2)
>>>> Serial Number: 5 (0x5)
>>>> Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
>>>> Issuer: "CN=Certificate Authority,O=DNS.DOMAIN"
>>>> Validity:
>>>> Not Before: Thu Jan 19 19:44:24 2012
>>>> Not After : Wed Jan 08 19:44:24 2014
>>>>
>>>>
>>>>
>>>
>>> Go back in time to the 7th or 8th and run:
>>>
>>>
>>> # getcert resubmit -d /var/lib/pki-ca/alias -n "auditSigningCert
>>> cert-pki-ca"
>>>
>>> There may be other certs in a similar situation. getcert list will 
>>> show you.
>>>
>>>
>>
>> Ouch. That would be rather disruptive I suppose. There is quite a lot 
>> of activity going to this
>> server, not to mention it's the primary ntp and dns server for the 
>> network.
>>
>> Do you suppose this todo list will work ?
>>
>> Firewall off the rest of the network, leaving the ipa server alone
>> Stop ntpd
>> Set date to 8th of January
>> Run the getcert resubmit command.
>> Change date back to correct date
>> Start ntpd
>> Remove the firewall rules
>
> Looks good. I'd restart the certmonger service rather than 
> resubmitting each individually. Be prepared for renewal to not 
> succeed. For some reason it didn't on and before expiration time so 
> whatever problem existed then likely still remains.
>
> So the question to ask is "what will I do if renewal fails again?"
>
> Nothing catastrophic will happen, but it will likely mean having to 
> roll forward again, debug, roll back, try again, and perhaps more than 
> once. It's hard to say w/o knowing why it failed in the first place.
>
>> How many of the services is required to be restarted for the renewal 
>> to work after the date is
>> changed to the 7th?
>
> The renewal itself should restart the required services.
>

This worked better than expected. Thank you! :)

ipa01 and ipa02 seem to be happy again, "getcert list" no longer 
displays any certificates out of date, and all certificates in need of 
renewal within 28 days has been renewed. The webui also started working 
again and things seem to be back to normal.

ipa03 however is still having issues. I could not renew any certificates 
on this server to begin with, but I managed to renew the certificates 
for the directory servers by changing the xmlrpc url to another ipa 
server in /etc/ipa/default.conf and resubmitting these requests.

"getcert resubmit -i <request-id" says SUBMITTING and the fails with 
NEED_GUIDANCE after a short while for the certificates for the PKI service.

/var/log/messages says: "certmonger: #033[?1034h28800" and "python: 
Updated certificate for ipaCert not available".

There is a lot of information in the /var/log/pki-ca/debug, but nothing 
that I can easily distinguish as an error from all the other output. 
Anything in particular I should look for?


Regards,
Siggi