[Freeipa-users] ipa-getcert list fails to report correctly - RESOLVED
Les Stott
Less at imagine-sw.com
Thu Feb 26 01:02:46 UTC 2015
> -----Original Message-----
> From: Martin Kosek [mailto:mkosek at redhat.com]
> Sent: Wednesday, 25 February 2015 10:35 PM
> To: Les Stott; Rob Crittenden; freeipa-users at redhat.com; Endi Dewata; Jan
> Cholasta
> Subject: Re: [Freeipa-users] ipa-getcert list fails to report correctly -
> RESOLVED
>
> On 02/25/2015 03:11 AM, Les Stott wrote:
> >
> >
> >> -----Original Message-----
> >> From: freeipa-users-bounces at redhat.com [mailto:freeipa-users-
> >> bounces at redhat.com] On Behalf Of Les Stott
> >> Sent: Monday, 23 February 2015 8:01 PM
> >> To: Rob Crittenden; Martin Kosek; freeipa-users at redhat.com; Endi
> >> Dewata; Jan Cholasta
> >> Subject: Re: [Freeipa-users] ipa-getcert list fails to report
> >> correctly
> >>
> >>
> >>
> >>> -----Original Message-----
> >>> From: freeipa-users-bounces at redhat.com [mailto:freeipa-users-
> >>> bounces at redhat.com] On Behalf Of Les Stott
> >>> Sent: Monday, 23 February 2015 12:18 PM
> >>> To: Rob Crittenden; Martin Kosek; freeipa-users at redhat.com; Endi
> >>> Dewata; Jan Cholasta
> >>> Subject: Re: [Freeipa-users] ipa-getcert list fails to report
> >>> correctly
> >>>
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: Rob Crittenden [mailto:rcritten at redhat.com]
> >>>> Sent: Saturday, 21 February 2015 1:39 AM
> >>>> To: Martin Kosek; Les Stott; freeipa-users at redhat.com; Endi Dewata;
> >>>> Jan Cholasta
> >>>> Subject: Re: [Freeipa-users] ipa-getcert list fails to report
> >>>> correctly
> >>>>
> >>>> Martin Kosek wrote:
> >>>>> On 02/20/2015 06:56 AM, Les Stott wrote:
> >>>>>> Hi all,
> >>>>>>
> >>>>>> The following is blocking the ability for me to install a CA replica.
> >>>>>>
> >>>>>> Environment:
> >>>>>>
> >>>>>> RHEL 6.6
> >>>>>>
> >>>>>> IPA 3.0.0-42
> >>>>>>
> >>>>>> PKI 9.0.3-38
> >>>>>>
> >>>>>> On the master the following is happening:
> >>>>>>
> >>>>>> ipa-getcert list
> >>>>>>
> >>>>>> Number of certificates and requests being tracked: 5.
> >>>>>>
> >>>>>> (but it shows no certificate details in the output)
> >>>>>>
> >>>>>> Running "getcert list" shows complete output.
> >>>>>>
> >>>>>> Also, when trying to browse
> >>>>>> https://master.mydomain.com/ca/ee/ca/getCertChain i get a failed
> >>>>>> response. The apache error logs on the master show....
> >>>>>>
> >>>>>> [Thu Feb 19 23:23:23 2015] [error] SSL Library Error: -12271 SSL
> >>>>>> client cannot verify your certificate
> >>>>>>
> >>>>>> The reason I am trying to browse that address is because that's
> >>>>>> what the ipa-ca-install setup is failing at (it complains that
> >>>>>> the CA certificate is not in proper format, in fact it's not able
> >>>>>> to get it at all).
> >>>>>>
> >>>>>> I know from another working ipa setup that ....
> >>>>>>
> >>>>>> Browsing to the above address provides valid xml content and
> >>>>>> ipa-getcert list shows certificate details and not just the
> >>>>>> number of tracked certificates.
> >>>>>>
> >>>>>> Been trying for a long time to figure out the issues without luck.
> >>>>>>
> >>>>>> I would greatly appreciate any help to troubleshoot and resolve
> >>>>>> the above issues.
> >>>>>>
> >>>>>> Regards,
> >>>>>>
> >>>>>> Les
> >>>>>
> >>>>> Endi or JanC, would you have any advise for Les? To me, it looks
> >>>>> like the Apache does not have proper certificate installed.
> >>>>>
> >>>>> My ipa-getcert on RHEL-6.6 shows 3 Server-Certs tracked, making it
> >>>>> in total of 8 certs tracked:
> >>>>>
> >>>>> # ipa-getcert list
> >>>>> Number of certificates and requests being tracked: 8.
> >>>>> Request ID '20141111000002':
> >>>>> status: MONITORING
> >>>>> stuck: no
> >>>>> key pair storage:
> >>>>> type=NSSDB,location='/etc/dirsrv/slapd-IDM-LAB-BOS-REDHAT-
> >>>> COM',nicknam
> >>>>> e='Server-Cert',token='NSS
> >>>>> Certificate
> >>>>> DB',pinfile='/etc/dirsrv/slapd-IDM-LAB-BOS-REDHAT-
> COM/pwdfile.txt'
> >>>>> certificate:
> >>>>> type=NSSDB,location='/etc/dirsrv/slapd-IDM-LAB-BOS-REDHAT-
> >>>> COM',nicknam
> >>>>> e='Server-Cert',token='NSS
> >>>>> Certificate DB'
> >>>>> CA: IPA
> >>>>> issuer: CN=Certificate Authority,O=EXAMPLE.COM
> >>>>> subject: CN=vm-086.example.com,O=EXAMPLE.COM
> >>>>> expires: 2016-11-11 00:00:01 UTC
> >>>>> key usage:
> >>>>> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> >>>>> eku: id-kp-serverAuth,id-kp-clientAuth
> >>>>> pre-save command:
> >>>>> post-save command:
> >>>>> track: yes
> >>>>> auto-renew: yes
> >>>>> Request ID '20141111000047':
> >>>>> status: MONITORING
> >>>>> stuck: no
> >>>>> key pair storage:
> >>>>> type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-
> >> Cert'
> >>>>> ,token='NSS Certificate
> >>>>> DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile.txt'
> >>>>> certificate:
> >>>>> type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-
> >> Cert'
> >>>>> ,token='NSS
> >>>>> Certificate DB'
> >>>>> CA: IPA
> >>>>> issuer: CN=Certificate Authority,O=EXAMPLE.COM
> >>>>> subject: CN=vm-086.example.com,O=EXAMPLE.COM
> >>>>> expires: 2016-11-11 00:00:46 UTC
> >>>>> key usage:
> >>>>> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> >>>>> eku: id-kp-serverAuth,id-kp-clientAuth
> >>>>> pre-save command:
> >>>>> post-save command:
> >>>>> track: yes
> >>>>> auto-renew: yes
> >>>>> Request ID '20141111000302':
> >>>>> status: MONITORING
> >>>>> stuck: no
> >>>>> key pair storage:
> >>>>> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',toke
> >>>>> n= 'N SS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
> >>>>> certificate:
> >>>>> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',toke
> >>>>> n=
> >>>>> 'N
> >>>>> SS
> >>>>> Certificate DB'
> >>>>> CA: IPA
> >>>>> issuer: CN=Certificate Authority,O=EXAMPLE.COM
> >>>>> subject: CN=vm-086.example.com,O=EXAMPLE.COM
> >>>>> expires: 2016-11-11 00:03:02 UTC
> >>>>> key usage:
> >>>>> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> >>>>> eku: id-kp-serverAuth,id-kp-clientAuth
> >>>>> pre-save command:
> >>>>> post-save command:
> >>>>> track: yes
> >>>>> auto-renew: yes
> >>>>>
> >>>>>
> >>>>> What is actually in your Apache NSS database?
> >>>>>
> >>>>> # certutil -L -d /etc/httpd/alias/
> >>>>>
> >>>>> Martin
> >>>>>
> >>>>
> >>>> Remember ipa-getcert is just a shortcut for certificates using the
> >>>> certmonger CA named IPA, so it's more a filter than anything else.
> >>>> I don't know why it wouldn't display any output but I'd file a bug.
> >>>>
> >>>> I think we'd need to see the getcert list output to try to figure
> >>>> out what is going on.
> >>>>
> >>>> As for the SSL error fetching the cert chain I think Martin may be
> >>>> onto something. The request is proxied through Apache. I think the
> >>>> client here might be the Apache proxy client.
> >>>>
> >>>> I believe this command replicates what Apache is doing, you might
> >>>> give it a try on the master. This will get the chain directly from
> >>>> dogtag, bypassing
> >>>> Apache:
> >>>>
> >>>> $ curl -v --cacert /etc/ipa/ca.crt
> >>>> https://`hostname`:9444/ca/ee/ca/getCertChain
> >>>>
> >>>> rob
> >>>
> >>> Certutil shows....
> >>>
> >>> certutil -L -d /etc/httpd/alias/
> >>>
> >>> Certificate Nickname Trust Attributes
> >>>
> >>> SSL,S/MIME,JAR/XPI
> >>>
> >>> MYDOMAIN.COM IPA CA CT,C,C
> >>> ipaCert u,u,u
> >>> Signing-Cert u,u,u
> >>> Server-Cert u,u,u
> >>>
> >>> curl -v --cacert /etc/ipa/ca.crt
> >>> https://`hostname`:9444/ca/ee/ca/getCertChain
> >>> * About to connect() to `hostname` port 9444 (#0)
> >>> * Trying 192.168.1.1... connected
> >>> * Connected to `hostname` (192.168.1.1) port 9444 (#0)
> >>> * Initializing NSS with certpath: sql:/etc/pki/nssdb
> >>> * CAfile: /etc/ipa/ca.crt
> >>> CApath: none
> >>> * SSL connection using TLS_RSA_WITH_AES_128_CBC_SHA
> >>> * Server certificate:
> >>> * subject: CN=`hostname`,O=MYDOMAIN.COM
> >>> * start date: Dec 13 01:21:30 2013 GMT
> >>> * expire date: Dec 03 01:21:30 2015 GMT
> >>> * common name: `hostname`
> >>> * issuer: CN=Certificate Authority,O=MYDOMAIN.COM
> >>>> GET /ca/ee/ca/getCertChain HTTP/1.1
> >>>> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7
> >>>> NSS/3.16.2.3 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> >>>> Host: `hostname`:9444
> >>>> Accept: */*
> >>>>
> >>> < HTTP/1.1 200 OK
> >>> < Server: Apache-Coyote/1.1
> >>> < Content-Type: application/xml
> >>> < Content-Length: 1434
> >>> < Date: Mon, 23 Feb 2015 01:04:29 GMT < <?xml version="1.0"
> >>> encoding="UTF-8"
> >>>
> >>
> standalone="no"?><XMLResponse><Status>0</Status><ChainBase64>MIID
> >>>
> >>
> zwYJKoZIhvcNAQcCoIIDwDCCA7wCAQExADAPBgkqhkiG9w0BBwGgAgQAoII
> >>>
> >>
> DoDCCA5wwggKEoAMCAQICAQEwDQYJKoZIhvcNAQELBQAwOjEYMBYGA1U
> >>>
> >>
> EChMPREVSSVZBVElWRVMuQ09NMR4wHAYDVQQDExVDZXJ0aWZpY2F0ZSB
> >>>
> >>
> BdXRob3JpdHkwHhcNMTMxMjEzMDEyMTI5WhcNMzMxMjEzMDEyMTI5Wj
> >>>
> >>
> A6MRgwFgYDVQQKEw9ERVJJVkFUSVZFUy5DT00xHjAcBgNVBAMTFUNlcnRp
> >>>
> >>
> ZmljYXRlIEF1dGhvcml0eTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCg
> >>>
> >>
> gEBAMAA8EaYhmpjSA8o3/1kB/W1+0K6+FrwCS+njOgRtXhiTdmtSddXSDVxH
> >>>
> >>
> OafFwqN26BR+QRPZbbpJY70gP3SG8W+J6+c37PMVNshWz6UfChGt6ubgFxlS
> >>>
> >>
> TGUUre2Osr9I4C836MXpGJvRx2VDEuMUxv8j7B9iDRnTDglseqPqrMct2No4w
> >>>
> >>
> k4cLtA9puBJb0Es76SOHP9edXlf6GBnuYwR8YMc1yJLqpP8IGpHhEkVxMsRpqk
> >>>
> >>
> EpuuRwEFa7uBcTDhqVV24BpFlseZVubpiOdEgfb3IRBTjvI1Mum9OCJbuj9P/W
> >>>
> >>
> mqMnrA0sQsmF/R3WBwFdMAsN3+bQCRw73+rwoeDNcCAwEAAaOBrDCBq
> >>>
> >>
> TAfBgNVHSMEGDAWgBSO8J+j2jAuyg3a0yE+3oVCQJCWUTAPBgNVHRMBAf8
> >>>
> >>
> EBTADAQH/MA4GA1UdDwEB/wQEAwIBxjAdBgNVHQ4EFgQUjvCfo9owLsoN
> >>>
> >>
> 2tMhPt6FQkCQllEwRgYIKwYBBQUHAQEEOjA4MDYGCCsGAQUFBzABhipodHR
> >>> wOi8vc2I!
> >>> ybW9uMDEuZGVyaXZhdGl2ZXMuY29tOjgwL2* Connection #0 to host
> >> `hostname`
> >>> left intact
> >>> * Closing connection #0
> >>>
> >>
> NhL29jc3AwDQYJKoZIhvcNAQELBQADggEBAKH8YkoTAzX2xNYMkZSDK84EK3
> >>>
> >>
> e4FUixdXxc/EC5ehjrtaqXT1KT9Fl9DAF5/jYNKqgmEmtHnPGlfQ7/Y1ESdhEGcB
> >>>
> >>
> ZjU4qLe4HaFXuw5c9odDYxhtjQUd1g7ifY8SKOcHDCY+6Xx6F/rhFgzrXXMndn8
> >>>
> >>
> ZaYryctPoOAj/5INnLrJq8S4XyLmb2BHM4e1ORQbOhDi8xjhfK2veYXvIu55Brhp
> >>>
> >>
> RSS/goz5oSE8e+QE/H9afRmeV2+WkS/YDhSyoUDb7CYjklRuONzX3GopKtp1y
> >>>
> >>
> yLXQZnBFjCvIJvja0mo3ik3AXxSZuOwUIlV23U8CyPU/rDeiV00iUyA/fLvdkEtZkx
> >>> AA==</ChainBase64></XMLResponse>
> >>>
> >>>
> >>> In any event, I've decided to rebuilt my DR IPA environment. Late
> >>> last year the master in DR had to be rebuilt due to a disk issue.
> >>> While IPA was restored manually and appeared to be working fine, CA
> >>> replication hasn't worked. I finally got CA replication working in
> >>> Prod after enabling needed apache modules and performing a yum
> >>> update to update related packages, but these things didn't help in
> >>> DR. It's my strong suspicion that something got missed when
> >>> restoring the DR master IPA server and this is what is causing all
> >>> my grief. Therefore, I'm going to wipe it
> >> out and start from scratch in DR.
> >>> There are other benefits for me to do this anyway.
> >>>
> >>
> >> Well things have gone from bad to worse.
> >>
> >> I removed IPA in DR. uninstalled all ipa clients, uninstalled
> >> replicas, removed replication agreements and removed the master. Ran
> >> pki-remove to clear any leftover pki instances and used certutil -D
> >> to remove left behind ipa entries in /etc/httpd/alias.
> >>
> >> So, clean slate to start again.
> >>
> >> This time, in order to mirror config with prod, I began an
> >> installation for the master on a different server, let's call it
> >> serverb. It was previously a replica (in my prod environment, serverb
> >> is the true master, servera, serverc, and serverd are replicas).
> >>
> >> So, trying to install a new fresh instance of IPA and it still fails
> >> to configure a CA.
> >>
> >> Attached is the relevant portion of the server install log file
> >> (ipa-server- install.txt). I have removed certificate and copyright info to
> reduce its size.
> >> Also my server to install is serverb.mydomain.com
> >>
> >> Apache logs at the time of the error show:
> >> [Mon Feb 23 03:05:31 2015] [error] SSL Library Error: -12195 Peer
> >> does not recognize and trust the CA that issued your certificate
> >>
> >> Certificate databases only show the following (note that "Server-Cert
> >> cert- pki-ca" got installed before the installer crashed). Prior to
> >> trying installation I had to manually remove server certs left behind
> >> from the previous installation via ...
> >> certutil -d /etc/httpd/alias -D -n "Server-Cert"
> >> certutil -d /etc/httpd/alias -D -n "MYDOMAIN.COM IPA CA"
> >> certutil -d /etc/httpd/alias -D -n ipaCert
> >>
> >> certutil -L -d /var/lib/pki-ca/alias
> >> Certificate Nickname Trust Attributes
> >> SSL,S/MIME,JAR/XPI
> >> Server-Cert cert-pki-ca CTu,Cu,Cu
> >>
> >> certutil -L -d /etc/pki/nssdb
> >> Certificate Nickname Trust Attributes
> >>
> >> SSL,S/MIME,JAR/XPI
> >>
> >>
> >> Selinux is in permissive mode.
> >> Ausearch -m avc does show some selinux issues, but its permissive
> >> mode so it should be ok right? In any event I have previously tried
> >> installing a CA replica with selinux disabled and it didn't help.
> >>
> >> I have tried removing ipa and pki rpms and reinstalling. Then
> >> rerunning the ipa server install script but the same error occurs.
> >>
> >> I noticed that /etc/ipa/ca.crt was still old, and referencing the original
> master.
> >> I removed that and again reran the installer but the same error occurred.
> >>
> >> Note also that /etc/ipa/cr.crt was not recreated when ipa-python was
> >> reinstalled.
> >>
> >> Other logs:
> >>
> >> /var/log/pki-ca/system shows
> >> 5042.main - [23/Feb/2015:03:05:12 EST] [3] [3] Cannot build CA chain.
> >> Error
> >> java.security.cert.CertificateException: Certificate is not a PKCS
> >> #11 certificate 5042.main - [23/Feb/2015:03:05:12 EST] [13] [3] authz
> >> instance DirAclAuthz initialization failed and skipped,
> >> error=Property internaldb.ldapconn.port missing value
> >> 5042.http-9445-1 - [23/Feb/2015:03:05:26 EST] [3] [3] Cannot build CA
> chain.
> >> Error java.security.cert.CertificateException: Certificate is not a
> >> PKCS #11 certificate
> >> 5042.http-9445-1 - [23/Feb/2015:03:05:35 EST] [3] [3] CASigningUnit:
> >> Object certificate not found. Error
> >> org.mozilla.jss.crypto.ObjectNotFoundException
> >>
> >> /var/log/pki-ca/catalina.out
> >> Feb 23, 2015 3:05:11 AM org.apache.catalina.startup.HostConfig
> >> deployDirectory
> >> INFO: Deploying web application directory ca 64-bit osutil library
> >> loaded 64-bit osutil library loaded CMS Warning: FAILURE: Cannot
> >> build CA chain. Error
> >> java.security.cert.CertificateException: Certificate is not a PKCS
> >> #11
> >> certificate|FAILURE: authz instance DirAclAuthz initialization failed
> >> certificate|and
> >> skipped, error=Property internaldb.ldapconn.port missing value|
> >> Server is started.
> >> Feb 23, 2015 3:05:12 AM org.apache.coyote.http11.Http11Protocol start
> >> INFO: Starting Coyote HTTP/1.1 on http-9180 Feb 23, 2015 3:05:12 AM
> >> org.apache.coyote.http11.Http11Protocol start
> >> INFO: Starting Coyote HTTP/1.1 on http-9443 Feb 23, 2015 3:05:12 AM
> >> org.apache.coyote.http11.Http11Protocol start
> >> INFO: Starting Coyote HTTP/1.1 on http-9445 Feb 23, 2015 3:05:12 AM
> >> org.apache.coyote.http11.Http11Protocol start
> >> INFO: Starting Coyote HTTP/1.1 on http-9444 Feb 23, 2015 3:05:12 AM
> >> org.apache.coyote.http11.Http11Protocol start
> >> INFO: Starting Coyote HTTP/1.1 on http-9446 Feb 23, 2015 3:05:12 AM
> >> org.apache.jk.common.ChannelSocket init
> >> INFO: JK: ajp13 listening on /0.0.0.0:9447 Feb 23, 2015 3:05:12 AM
> >> org.apache.jk.server.JkMain start
> >> INFO: Jk running ID=0 time=0/25 config=null Feb 23, 2015 3:05:12 AM
> >> org.apache.catalina.startup.Catalina start
> >> INFO: Server startup in 1655 ms
> >>
> >> I have no idea where to look next. There must be some remnant of the
> >> old system hanging around screwing things up but I cannot figure it
> >> out. This will drive me insane!
> >>
> >> I can provide more logs if needed.
> >>
> >> Thanks in advance for any help.
> >>
> >
> > Have resolved this.
>
> Great! Thanks for reaching back to us.
>
> > Here is the procedure to completely remove FreeIPA so you can start
> again.
>
> To me, that sounds like the FreeIPA uninstaller is missing some clean up
> steps.
> I would personally rather resolve it in the the actual code than just having this
> information in the list archives.
>
> >
> > ipa-server-install --uninstall
> > certutil -d /etc/httpd/alias -D -n "Server-Cert"
> > certutil -d /etc/httpd/alias -D -n "DERIVATIVES.COM IPA CA"
> > certutil -d /etc/httpd/alias -D -n ipaCert certutil -d
> > /etc/httpd/alias -D -n Signing-Cert
>
> This sounds like https://fedorahosted.org/freeipa/ticket/4639. We should
> bump the priority if it is really causing issues.
>
Yes, definitely experienced this behaviour.
> > yum -y remove pki-selinux pki-ca pki-common pki-setup pki-silent
> > pki-java-tools pki-symkey pki-util pki-native-tools ipa-server-selinux
> > ipa-server ipa-client ipa-admintools ipa-python ipa-pki-ca-theme
> > ipa-pki-common-theme 389-ds-base 389-ds-base-libs userdel pkisrv
> > userdel pkiuser
>
> This should not be needed at all, AFAIK.
>
Possibly not, but wanted to start with a clean system without having to reinstall the OS from scratch.
> > rm -rf /etc/pki-ca /var/lib/pki-ca /var/log/pki-ca /etc/certmonger
> > /etc/sysconfig/pki-ca /etc/sysconfig/pki /var/run/pki-ca.pid
> > /usr/share/pki /etc/ipa /var/log/ipa* reboot
> >
> > Now you have a clean slate.
>
> Do you know which step of the steps above actually helped you resolve the
> reinstall issue?
>
The reboot I think was key to the whole process, but pki remnants seemed left behind too which caused grief. Previously I had never rebooted the system in between uninstall/reinstall.
/etc/ipa/ca.crt was also left behind. It caused an issue during one reinstall as it never got updated and the install bombed out because it found a mismatched cert. This led me to deleting all possible ipa/pki directories and then removing/reinstalling rpms to restore to default state.
I noticed that in some cases (I went through this same process on 6 servers to reinstall and setup CA replicas) I could still see a left over process running as the pkiuser (tomcat/java) which stopped the "userdel pkiuser" command from completing. I had to kill that process and then userdel pkiuser worked.
Regards,
Les
> >
> > Then install works as normal for IPA Server, Replica and CA Replica
> installations.
> >
> > Hope this saves someone else time in the future.
> >
> > Regards,
> >
> > Les
>
>
More information about the Freeipa-users
mailing list