[Freeipa-users] IPA server having cert issues
Rob Crittenden
rcritten at redhat.com
Wed Apr 27 17:11:07 UTC 2016
Bret Wortman wrote:
> So in lieu of fixing these certs, is there an acceptable way to dump
> them all and start over /without losing the contents of the IPA
> database/? Or otherwise really screwing ourselves?
I don't believe there is a way.
> We have a replica that's still up and running and we've switched
> everyone over to talking to it, but we're at risk with just the one.
I'd ignore the two unknown certs for now. They look like someone was
experimenting with issuing a cert and didn't quite get things working.
The CA seems to be throwing an error. I'd check the syslog for messages
from certmonger and look at the CA debug log and selftest log.
rob
>
> Thanks!
>
>
> On 04/27/2016 06:05 AM, Bret Wortman wrote:
>> Was this at all informative?
>>
>> On 04/26/2016 02:06 PM, Bret Wortman wrote:
>>>
>>>
>>> On 04/26/2016 01:45 PM, Rob Crittenden wrote:
>>>> Bret Wortman wrote:
>>>>> I think I've found a deeper problem, in that I can't update these
>>>>> because IPA simply won't start at all now.
>>>>>
>>>>> I mistyped one of these -- the 2016-03-11 is actually 2018-03-11, and
>>>>> 2016-04-01 is actually 2036-04-01.
>>>>>
>>>>> As for the unknowns, the first says status: CA_REJECTED and the error
>>>>> says "hostname in subject of request 'zw198.private.net' does not
>>>>> match
>>>>> principal hostname 'private.net'", with stuck: yes.
>>>>>
>>>>> The second is similar, but for a different host.
>>>>
>>>> Is it really a different host and why? I think we'd need to see the
>>>> full output to know what's going on.
>>>>
>>>
>>> Full output:
>>>
>>> Number of certificates and requests being tracked: 10.
>>> Request ID '20140428181940':
>>> status: MONITORING
>>> stuck: no
>>> key pair storage:
>>> type=NSSDB,location='/etc/dirsrv/slapd-PRIVATE-NET',nickname='Server-Cert',token='NSS
>>> Certificate DB',pinfile='/etc/dirsrv/slapd-PRIVATE-NET/pwdfile.txt'
>>> certificate:
>>> type=NSSDB,location='/etc/dirsrv/slapd-PRIVATE-NET',nickname='Server-Cert',token='NSS
>>> Certificate DB'
>>> CA: IPA
>>> issuer: CN=Certificate Authority,O=PRIVATE.NET
>>> subject: CN=zsipa.private.net,O=PRIVATE.NET
>>> expires: 2018-04-02 13:04:51 UTC
>>> principal name: ldap/zsipa.private.net at PRIVATE.NET
>>> key usage:
>>> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>>> eku: id-kp-serverAuth,id-kp-clientAuth
>>> pre-save command:
>>> post-save command:
>>> track: yes
>>> auto-renew: yes
>>> Request ID '20140428182016':
>>> status: MONITORING
>>> stuck: no
>>> key pair storage:
>>> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
>>> Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
>>> certificate:
>>> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
>>> Certificate DB'
>>> CA: IPA
>>> issuer: CN=Certificate Authority,O=PRIVATE.NET
>>> subject: CN=zsipa.private.net,O=PRIVATE.NET
>>> expires: 2018-04-02 13:04:31 UTC
>>> principal name: HTTP/zsipa.private.net at PRIVATE.NET
>>> key usage:
>>> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>>> eku: id-kp-serverAuth,id-kp-clientAuth
>>> pre-save command:
>>> post-save command:
>>> track: yes
>>> auto-renew: yes
>>> Request ID '20150211141945':
>>> status: CA_REJECTED
>>> ca-error: Server at https://zsipa.private.net/ipa/xml denied our
>>> request, giving up: 2100 (RPC failed at server. Insufficient access:
>>> hostname in subject of request 'zw198.private.net' does not match
>>> principal hostname 'private.net').
>>> stuck: yes
>>> key pair storage:
>>> type=NSSDB,location='/etc/pki/nssdb',nickname='Server-Cert',token='NSS Certificate
>>> DB'
>>> certificate:
>>> type=NSSDB,location='/etc/pki/nssdb',nickname='Server-Cert'
>>> CA: IPA
>>> issuer:
>>> subject:
>>> expires: unknown
>>> pre-save command:
>>> post-save command:
>>> track: yes
>>> auto-renew: yes
>>> Request ID '20150816194107':
>>> status: CA_UNREACHABLE
>>> ca-error: Internal error
>>> stuck: no
>>> key pair storage:
>>> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
>>> cert-pki-ca',token='NSS Certificate DB',pin='424151811070'
>>> certificate:
>>> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
>>> cert-pki-ca',token='NSS Certificate DB'
>>> CA: dogtag-ipa-ca-renew-agent
>>> issuer: CN=Certificate Authority,O=PRIVATE.NET
>>> subject: CN=CA Audit,O=PRIVATE.NET
>>> expires: 2016-04-17 18:19:19 UTC
>>> key usage: digitalSignature,nonRepudiation
>>> pre-save command:
>>> post-save command:
>>> track: yes
>>> auto-renew: yes
>>> Request ID '20150816194108':
>>> status: CA_UNREACHABLE
>>> ca-error: Internal error
>>> stuck: no
>>> key pair storage:
>>> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
>>> cert-pki-ca',token='NSS Certificate DB',pin='424151811070'
>>> certificate:
>>> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
>>> cert-pki-ca',token='NSS Certificate DB'
>>> CA: dogtag-ipa-ca-renew-agent
>>> issuer: CN=Certificate Authority,O=PRIVATE.NET
>>> subject: CN=OCSP Subsystem,O=PRIVATE.NET
>>> expires: 2016-04-17 18:19:18 UTC
>>> key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
>>> eku: id-kp-OCSPSigning
>>> pre-save command:
>>> post-save command:
>>> track: yes
>>> auto-renew: yes
>>> Request ID '20150816194109':
>>> status: CA_UNREACHABLE
>>> ca-error: Internal error
>>> stuck: no
>>> key pair storage:
>>> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
>>> cert-pki-ca',token='NSS Certificate DB',pin='424151811070'
>>> certificate:
>>> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
>>> cert-pki-ca',token='NSS Certificate DB'
>>> CA: dogtag-ipa-ca-renew-agent
>>> issuer: CN=Certificate Authority,O=PRIVATE.NET
>>> subject: CN=CA Subsystem,O=PRIVATE.NET
>>> expires: 2016-04-17 18:19:19 UTC
>>> key usage:
>>> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>>> eku: id-kp-serverAuth,id-kp-clientAuth
>>> pre-save command:
>>> post-save command:
>>> track: yes
>>> auto-renew: yes
>>> Request ID '20150816194110':
>>> status: MONITORING
>>> stuck: no
>>> key pair storage:
>>> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
>>> cert-pki-ca',token='NSS Certificate DB',pin='424151811070'
>>> certificate:
>>> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
>>> cert-pki-ca',token='NSS Certificate DB'
>>> CA: dogtag-ipa-ca-renew-agent
>>> issuer: CN=Certificate Authority,O=PRIVATE.NET
>>> subject: CN=Certificate Authority,O=PRIVATE.NET
>>> expires: 2036-04-01 20:16:39 UTC
>>> key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
>>> pre-save command:
>>> post-save command:
>>> track: yes
>>> auto-renew: yes
>>> Request ID '20150816194111':
>>> status: CA_UNREACHABLE
>>> ca-error: Internal error
>>> stuck: no
>>> key pair storage:
>>> type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
>>> Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
>>> certificate:
>>> type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
>>> Certificate DB'
>>> CA: dogtag-ipa-ca-renew-agent
>>> issuer: CN=Certificate Authority,O=PRIVATE.NET
>>> subject: CN=IPA RA,O=PRIVATE.NET
>>> expires: 2016-04-17 18:19:35 UTC
>>> key usage:
>>> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>>> eku: id-kp-serverAuth,id-kp-clientAuth
>>> pre-save command:
>>> post-save command:
>>> track: yes
>>> auto-renew: yes
>>> Request ID '20150816194112':
>>> status: MONITORING
>>> stuck: no
>>> key pair storage:
>>> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
>>> cert-pki-ca',token='NSS Certificate DB',pin='424151811070'
>>> certificate:
>>> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
>>> cert-pki-ca',token='NSS Certificate DB'
>>> CA: dogtag-ipa-renew-agent
>>> issuer: CN=Certificate Authority,O=PRIVATE.NET
>>> subject: CN=zsipa.private.net,O=PRIVATE.NET
>>> expires: 2018-03-11 13:04:29 UTC
>>> key usage:
>>> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>>> eku: id-kp-serverAuth,id-kp-clientAuth
>>> pre-save command:
>>> post-save command:
>>> track: yes
>>> auto-renew: yes
>>> Request ID '20151214165433':
>>> status: CA_REJECTED
>>> ca-error: Server at https://zsipa.private.net/ipa/xml denied our
>>> request, giving up: 2100 (RPC failed at server. Insufficient access:
>>> hostname in subject of request 'zsipa.private.net' does not match
>>> principal hostname 'www.private.net').
>>> stuck: yes
>>> key pair storage:
>>> type=FILE,location='/etc/pki/tls/private/www.private.net.key'
>>> certificate:
>>> type=FILE,location='/etc/pki/tls/certs/www.private.net.crt'
>>> CA: IPA
>>> issuer:
>>> subject:
>>> expires: unknown
>>> pre-save command:
>>> post-save command:
>>> track: yes
>>> auto-renew: yes
>>>
>>>
>>>> A given host can only get certificates for itself or those delegated
>>>> to it. Hostnames are used for this enforcement so if they don't line
>>>> up you'll see this type of rejection.
>>>>
>>>>>
>>>>> No idea what's wrong with the rest, or why nothing will start. Near
>>>>> as I
>>>>> can tell, Kerberos is failing to start, which is causing everything
>>>>> else
>>>>> to go toes up.
>>>>>
>>>>> Early in the startup, in /var/log/messages, there's:
>>>>>
>>>>> ns-slapd: GSSAPI Error: Unspecified GSS failure. Minor code may
>>>>> provide
>>>>> more information (No Kerberos credentials available)
>>>>
>>>> Without more context it's hard to say. 389 is rather chatty about
>>>> things and of course when it starts it has no ticket so it logs a
>>>> bunch of stuff, eventually (hopefully) gets one, and then shuts up.
>>>>
>>>>>
>>>>> After that, I get a jar file read pboelm on log4j.jar, then a
>>>>> series of
>>>>> property setting attempts that don't find matching properties. Then
>>>>> some
>>>>> cipher errors, then it looks like named starts up okay, and everything
>>>>> pauses for about 5 minutes before it all comes crashing back down.
>>>>>
>>>>
>>>> I wouldn't get too hung up on particular services just yet. Without
>>>> valid certs things will fail and those problems will cascade. I
>>>> think we just need more details at this point.
>>>>
>>>> rob
>>>>
>>>>>
>>>>> Bret
>>>>>
>>>>> On 04/26/2016 12:40 PM, Petr Vobornik wrote:
>>>>>> On 04/26/2016 06:00 PM, Bret Wortman wrote:
>>>>>>> # getcert list | grep expires
>>>>>>> expires: 2018-04-02 13:04:51 UTC
>>>>>>> expires: 2018-04-02 13:04:31 UTC
>>>>>>> expires: unknown
>>>>>>> expires: 2016-04-17 18:19:19 UTC
>>>>>>> expires: 2016-04-17 18:19:18 UTC
>>>>>>> expires: 2016-04-17 18:19:19 UTC
>>>>>>> expires: 2016-04-01 20:16:39 UTC
>>>>>>> expires: 2016-04-17 18:19:35 UTC
>>>>>>> expires: 2016-03-11 13:04:29 UTC
>>>>>>> expires: unknown
>>>>>>> #
>>>>>>>
>>>>>>> So some got updated and most didn't. Is there a recommended way
>>>>>>> to update these
>>>>>>> all? The system is still backdated to 3 April (ntpd disabled) at
>>>>>>> this point.
>>>>>> It's usually good to start renewing(when it doesn't happen
>>>>>> automatically
>>>>>> from some reason) with the cert which is about to expired first, i.e.
>>>>>> the one with "2016-03-11 13:04:29"
>>>>>>
>>>>>> The process is:
>>>>>> - move date before the cert is about to expired
>>>>>> - leave it up to certmonger or manually force resubmit by `getcert
>>>>>> resubmit -i $REQUEST_ID`, where request ID is in `getcert list`
>>>>>> output.
>>>>>>
>>>>>> I'm little worried about the fact that CA cert was renewed at date
>>>>>> which
>>>>>> is after expiration of the other certs.
>>>>>>
>>>>>> Also the `expires: unknown` doesn't look good. Check `getcert list`
>>>>>> output for errors related to the cert.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Bret
>>>>>>>
>>>>>>>
>>>>>>> On 04/26/2016 11:46 AM, Petr Vobornik wrote:
>>>>>>>> On 04/26/2016 03:26 PM, Bret Wortman wrote:
>>>>>>>>> On our non-CA IPA server, this is happening, in case it's
>>>>>>>>> related and illustrative:
>>>>>>>>>
>>>>>>>>> # ipa host-del zw113.private.net
>>>>>>>>> ipa: ERROR: Certificate format error:
>>>>>>>>> (SEC_ERROR_LEGACY_DATABASE) The
>>>>>>>>> certificate/key database is in an old, unsupported format.
>>>>>>>>> #
>>>>>>>> I would start with checking on all IPA servers if and what
>>>>>>>> certificates
>>>>>>>> are expired:
>>>>>>>> # getcert list
>>>>>>>> or short version to check if there are any:
>>>>>>>> # getcert list | grep expires
>>>>>>>>
>>>>>>>> When CA cert is renewed, it is not automatically transfered to
>>>>>>>> clients.
>>>>>>>> There one must run:
>>>>>>>> # ipa-certupdate
>>>>>>>>
>>>>>>>>> On 04/26/2016 09:24 AM, Bret Wortman wrote:
>>>>>>>>>> I rolled the date on the IPA server in question back to April
>>>>>>>>>> 1 and ran
>>>>>>>>>> "ipa-cacert-manage renew", which said it completed
>>>>>>>>>> successfully. I rolled the
>>>>>>>>>> date back to current and tried restarting ipa using ipactl
>>>>>>>>>> stop && ipactl
>>>>>>>>>> start, but no joy. No more ca renewal errors, but right after
>>>>>>>>>> the pause I see
>>>>>>>>>> this in /var/log/messages:
>>>>>>>>>>
>>>>>>>>>> systemd: kadmin.service: main process exited, code=exited,
>>>>>>>>>> status=2/INVALIDARGUMENT
>>>>>>>>>> systemd: Unit kadmin.service entered failed state.
>>>>>>>>>> systemd: kadmin.service failed.
>>>>>>>>>>
>>>>>>>>>> I rebooted the server just in case, and it's still getting
>>>>>>>>>> stuck at the same
>>>>>>>>>> place. ipa-otpd doesn't get around to starting.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Bret
>>>>>>>>>>
>>>>>>>>>> After the several-minutes-long pause after ipactl start
>>>>>>>>>> outputs "Starting
>>>>>>>>>> pki-tomcatd Service", I get the
>>>>>>>>>>
>>>>>>>>>> On 04/26/2016 08:14 AM, Bret Wortman wrote:
>>>>>>>>>>> I have an IPA server on a private network which has
>>>>>>>>>>> apparently run into
>>>>>>>>>>> certificate issues this morning. It's been running without
>>>>>>>>>>> issue for quite a
>>>>>>>>>>> while, and is on 4.1.4-1 on fedora 21.
>>>>>>>>>>>
>>>>>>>>>>> This morning, the gui started giving:
>>>>>>>>>>>
>>>>>>>>>>> IPA Error 907: NetworkError with description "cannot connect to
>>>>>>>>>>> 'https://zsipa.private.net:443/ca/agent/ca/displayBySerial':
>>>>>>>>>>> (SSL_ERROR_EXPIRED_CERRT_ALERT) SSL peer rejected your
>>>>>>>>>>> certificate as expired."
>>>>>>>>>>>
>>>>>>>>>>> I dug into the logs and after trying to restart ipa using
>>>>>>>>>>> ipactl, there was a
>>>>>>>>>>> length pause, then:
>>>>>>>>>>>
>>>>>>>>>>> dogtag-ipa-ca-renew-agent-submit: Updated certificate not
>>>>>>>>>>> available
>>>>>>>>>>> certmonger: Certificate named "ipaCert" in token "NSS
>>>>>>>>>>> Certificate DB" in
>>>>>>>>>>> database "/etc/httpd/alias" is no longer valid.
>>>>>>>>>>> dogtag-ipa-ca-renew-agent-submit: Updated certificate not
>>>>>>>>>>> available
>>>>>>>>>>> certmonger: Certificate named "ocspSigningCert cert-pki-ca"
>>>>>>>>>>> in token "NSS
>>>>>>>>>>> Certificate DB" in database "/etc/pki/pki-tomcat/alias" is no
>>>>>>>>>>> longer valid.
>>>>>>>>>>> dogtag-ipa-ca-renew-agent-submit: Updated certificate not
>>>>>>>>>>> available.
>>>>>>>>>>> named-pkcs11[3437]: client 192.168.208.205#57832: update
>>>>>>>>>>> '208.168.192.in-addr.arpa/IN' denied
>>>>>>>>>>>
>>>>>>>>>>> and then things start shutting down. I can't start ipa at all
>>>>>>>>>>> using ipactl.
>>>>>>>>>>>
>>>>>>>>>>> So at present, our DNS is down. Authentication should work
>>>>>>>>>>> for a while, but
>>>>>>>>>>> I'd like to get this working again as quickly as possible.
>>>>>>>>>>> Any ideas? I deal
>>>>>>>>>>> with certificates so infrequently (like only when something
>>>>>>>>>>> like this
>>>>>>>>>>> happens) that I'm not sure where to start.
>>>>>>>>>>>
>>>>>>>>>>> Thanks!
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> *Bret Wortman*
>>>>>>>>>>> /Coming soon to Kickstarter.../
>>>>>>>>>>> <http://wrapbuddies.co/>
>>>>>>>>>>> http://wrapbuddies.co/
>>>>>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
>
>
More information about the Freeipa-users
mailing list