[Freeipa-users] RHEL 7 Upgrade experience so far

Mon Aug 4 13:36:37 UTC 2014

On Thu, 2014-07-31 at 06:27 -0700, Erinn Looney-Triggs wrote:
> On 07/30/2014 02:31 PM, Ade Lee wrote:
> > On Tue, 2014-07-29 at 17:49 -0700, Erinn Looney-Triggs wrote:
> >>>> 
> >> 
> >>>> Ok, well I tried deleting it using certutil it deletes both,
> >>>> I tried using keytool to see if it would work any better, no
> >>>> dice there. I'll try the rename, but at this point I am not
> >>>> holding my breath on that, it seems all operation are a bit
> >>>> too coarse. It seems the assumption was being made that there
> >>>> would only be one of each nickname. Which frankly makes me
> >>>> wonder how any of this kept running after the renewal.
> >>>> 
> >>>> For now I'll see what I can do on a copy of the db using
> >>>> python.
> >>> 
> >>> It is a little strange that there are multiple 'caSigningCert 
> >>> cert-pki-ca' as this is the CA itself. It should be good for
> >>> 20 years and isn't something that the current renewal code
> >>> handles yet.
> >>> 
> >>> You probably won't have much luck with python-nss. It can
> >>> handle reading PKCS#12 files but I don't believe it can write
> >>> them (access to key material).
> >>> 
> >>> I'm not sure why certutil didn't do the trick. This should
> >>> work, if you want to give it another try. I'm assuming that
> >>> /root/cacert.p12 has the latest exported certs, adjust as
> >>> necessary:
> >>> 
> >>> # certutil -N -d /tmp/test # pk12util -i /root/cacert.p12 -d 
> >>> /tmp/test # certutil -D -d /tmp/test -n '<nickname>'
> >>> 
> >>> certutil should delete the oldest cert first, it always has
> >>> for me.
> >>> 
> >>> rob
> >>> 
> >> 
> >> Ok folks I managed to clean up the certificate DB so there is
> >> only one valid certificate for each service. Installation
> >> continued pass that step and then failed shortly thereafter on
> >> configuring the ca. So here is my new error:
> >> 
> >> 
> >> pkispawn    : ERROR    ....... Exception from Java Configuration 
> >> Servlet: Error while updating security domain:
> >> java.io.IOException: 2 pkispawn    : DEBUG    ....... Error Type:
> >> HTTPError pkispawn    : DEBUG    ....... Error Message: 500
> >> Server Error: Internal Server Error pkispawn    : DEBUG
> >> .......   File "/usr/sbin/pkispawn", line 374, in main rv =
> >> instance.spawn() File 
> >> "/usr/lib/python2.7/site-packages/pki/deployment/configuration.py",
> >>
> >> 
> line 128, in spawn
> >> json.dumps(data, cls=pki.encoder.CustomTypeEncoder)) File
> >> "/usr/lib/python2.7/site-packages/pki/deployment/pkihelper.py", 
> >> line 2998, in configure_pki_data response =
> >> client.configure(data) File
> >> "/usr/lib/python2.7/site-packages/pki/system.py", line 80, in 
> >> configure r = self.connection.post('/rest/installer/configure',
> >> data, headers) File
> >> "/usr/lib/python2.7/site-packages/pki/client.py", line 64, in
> >> post r.raise_for_status() File
> >> "/usr/lib/python2.7/site-packages/requests/models.py", line 638,
> >> in raise_for_status raise http_error
> >> 
> >> 
> >> 2014-07-30T00:27:48Z CRITICAL failed to configure ca instance
> >> Command '/usr/sbin/pkispawn -vv -s CA -f /tmp/tmpqX9SGx' returned
> >> non-zero exit status 1 2014-07-30T00:27:48Z DEBUG   File 
> >> "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py",
> >>
> >> 
> line 638, in run_script
> >> return_value = main_function()
> >> 
> >> File "/usr/sbin/ipa-replica-install", line 667, in main CA =
> >> cainstance.install_replica_ca(config)
> >> 
> >> File 
> >> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
> >>
> >> 
> line 1678, in install_replica_ca
> >> subject_base=config.subject_base)
> >> 
> >> File 
> >> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
> >>
> >> 
> line 478, in configure_instance
> >> self.start_creation(runtime=210)
> >> 
> >> File 
> >> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
> >> line 364, in start_creation method()
> >> 
> >> File 
> >> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
> >>
> >> 
> line 604, in __spawn_instance
> >> raise RuntimeError('Configuration of CA failed')
> >> 
> >> 2014-07-30T00:27:48Z DEBUG The ipa-replica-install command
> >> failed, exception: RuntimeError: Configuration of CA failed
> >> 
> >> And from the pki-tomcat/ca debug log: isSDHostDomainMaster():
> >> Getting domain.xml from CA... 
> >> [30/Jul/2014:00:27:48][http-bio-8443-exec-3]: getDomainXML start 
> >> [30/Jul/2014:00:27:48][http-bio-8443-exec-3]: getDomainXML:
> >> status=0 [30/Jul/2014:00:27:48][http-bio-8443-exec-3]:
> >> getDomainXML: domainInfo=<?xml version="1.0" encoding="UTF-8" 
> >> standalone="no"?><DomainInfo><Name>IPA</Name><CAList><CA><Host>ipa.example.com</Host><SecurePort>443</SecurePort><SecureAgentPort>443</SecureAgentPort><SecureAdminPort>443</SecureAdminPort><SecureEEClientAuthPort>443</SecureEEClientAuthPort><UnSecurePort>80</UnSecurePort><Clone>FALSE</Clone><SubsystemName>pki-cad</SubsystemName><DomainManager>TRUE</DomainManager></CA><SubsystemCount>1</SubsystemCount></CAList><OCSPList><SubsystemCount>0</SubsystemCount></OCSPList><KRAList><SubsystemCount>0</SubsystemCount></KRAList><RAList><SubsystemCount>0</SubsystemCount></RAList><TKSList><SubsystemCount>0</SubsystemCount></TKSList><TPSList><SubsystemCount>0</SubsystemCount></TPSList></DomainInfo>
> >>
> >> 
> [30/Jul/2014:00:27:48][http-bio-8443-exec-3]: Cloning a domain master
> >> [30/Jul/2014:00:27:48][http-bio-8443-exec-3]: WizardPanelBase 
> >> updateDomainXML start hostname=ipa.example.com port=443 
> >> [30/Jul/2014:00:27:48][http-bio-8443-exec-3]:
> >> updateSecurityDomain: failed to update security domain using
> >> admin port 443: org.xml.sax.SAXParseException; lineNumber: 1;
> >> columnNumber: 50; White spaces are required between publicId and
> >> systemId. [30/Jul/2014:00:27:48][http-bio-8443-exec-3]:
> >> updateSecurityDomain: now trying agent port with client auth 
> >> [30/Jul/2014:00:27:48][http-bio-8443-exec-3]: WizardPanelBase 
> >> updateDomainXML start hostname=ipa.example.com port=443 
> >> [30/Jul/2014:00:27:48][http-bio-8443-exec-3]: updateDomainXML() 
> >> nickname=subsystemCert cert-pki-ca 
> >> [30/Jul/2014:00:27:48][http-bio-8443-exec-3]: WizardPanelBase 
> >> updateDomainXML: status=1
> >> 
> >> And from pki-tomcat/catalina.out: 00:26:53,450  INFO 
> >> (org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher:82)
> >>
> >> 
> - Deploying javax.ws.rs.core.Application: class
> >> com.netscape.ca.CertificateAuthorityApplication 00:26:53,472
> >> INFO 
> >> (org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher:82)
> >>
> >> 
> - Adding singleton provider com.netscape.certsrv.acls.ACLInterceptor
> >> from Application javax.ws.rs.core.Application 00:26:53,473  INFO 
> >> (org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher:82)
> >>
> >> 
> - Adding singleton provider
> >> com.netscape.certsrv.authentication.AuthMethodInterceptor from 
> >> Application javax.ws.rs.core.Application 00:26:53,772 DEBUG
> >> (org.jboss.resteasy.core.SynchronousDispatcher:60) - PathInfo:
> >> /installer/configure AuthInterceptor:
> >> SystemConfigResource.configure() AuthInterceptor: mapping name:
> >> default AuthInterceptor: required auth methods: [*] 
> >> AuthInterceptor: anonymous access allowed [Fatal Error] :1:50:
> >> White spaces are required between publicId and systemId. [Fatal
> >> Error] :1:50: White spaces are required between publicId and 
> >> systemId. [Fatal Error] :1:50: White spaces are required between
> >> publicId and systemId. [Fatal Error] :1:50: White spaces are
> >> required between publicId and systemId. java.io.IOException: 2 
> >> at 
> >> com.netscape.cms.servlet.csadmin.ConfigurationUtils.updateDomainXML(ConfigurationUtils.java:3415)
> >>
> >> 
> at
> >> com.netscape.cms.servlet.csadmin.ConfigurationUtils.updateSecurityDomain(ConfigurationUtils.java:3345)
> >>
> >> 
> at
> >> com.netscape.cms.servlet.csadmin.SystemConfigService.configure(SystemConfigService.java:655)
> >>
> >> 
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at 
> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >>
> >> 
> at
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>
> >> 
> at java.lang.reflect.Method.invoke(Method.java:606)
> >> at 
> >> org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:167)
> >>
> >> 
> at
> >> org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257)
> >>
> >> 
> at
> >> org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222)
> >>
> >> 
> at
> >> org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211)
> >>
> >> 
> at
> >> org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:542)
> >>
> >> 
> at
> >> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:524)
> >>
> >> 
> at
> >> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:126)
> >>
> >> 
> at
> >> org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208)
> >>
> >> 
> at
> >> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55)
> >>
> >> 
> at
> >> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50)
> >>
> >> 
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> >> at 
> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >>
> >> 
> at
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>
> >> 
> at java.lang.reflect.Method.invoke(Method.java:606)
> >> at 
> >> org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:277)
> >>
> >> 
> at
> >> org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:274)
> >>
> >> 
> at java.security.AccessController.doPrivileged(Native Method)
> >> at javax.security.auth.Subject.doAsPrivileged(Subject.java:536) 
> >> at 
> >> org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:309)
> >>
> >> 
> at
> >> org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:169)
> >>
> >> 
> at
> >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:299)
> >>
> >> 
> at
> >> org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:57)
> >>
> >> 
> at
> >> org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:193)
> >>
> >> 
> at
> >> org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:189)
> >>
> >> 
> at java.security.AccessController.doPrivileged(Native Method)
> >> at 
> >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> >>
> >> 
> at
> >> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> >>
> >> 
> at
> >> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> >>
> >> 
> at
> >> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
> >>
> >> 
> at
> >> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> >>
> >> 
> at
> >> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> >>
> >> 
> at
> >> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> >>
> >> 
> at
> >> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
> >>
> >> 
> at
> >> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1024)
> >>
> >> 
> at
> >> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> >>
> >> 
> at
> >> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
> >>
> >> 
> at
> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>
> >> 
> at
> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>
> >> 
> at java.lang.Thread.run(Thread.java:745)
> >> 
> >> 
> > 
> > Is there any indication of what the error is on the master CA? This
> > would likely be in either the debug log or the catalina.out. Also,
> > you should see the access to update the security domain in the 
> > httpd access log on the master.
> > 
> > 
> >> I fixed the db (in case anyone else runs into this issue) by
> >> doing the following:
> >> 
> >> PKCS12Export of the NSS DB in order to get a .p12 file with all
> >> the certificates.
> >> 
> >> use openssl to convert the pkcs12 file to a single file in PEM
> >> format with all of the certificates and the keys.
> >> 
> >> From here unfortunately, you have to manually go in and find the
> >> valid key/cert pairs in the pem file and create new PEM files for
> >> each key pair you intend to import, ocsp, server cert, etc.
> >> Obviously only grab one key pair for each, and only the valid
> >> ones. Openssl does not support mass importing of key/certificate
> >> pairs into a PKCS12 file.
> >> 
> >> Once you have a pem file for each service, you then need to
> >> convert these pem files back into PKCS12 format, one at a time,
> >> using the -name flag to give them friendly names.
> >> 
> >> After this create a new NSS DB using certutil, and import each
> >> PKCS12 file for each service into the DB.
> >> 
> >> I don't know if this is necessary, but I set the flags to be
> >> identical to the original DB for the certs.
> >> 
> >> Now use PKCS12Export to export your newly created NSS DB into a 
> >> cacert.p12 file. You now should have a nice new cacert.p12 file
> >> with only valid certificates.
> >> 
> >> Most of the user space tools for handling NSS and PKCS12 files
> >> are not flexible enough to get what you want done. This could
> >> probably be coded up in a more efficient way.
> >> 
> > 
> > Thanks for the steps above.  We'll be sure to keep them handy in
> > case this happens again, and I think we need to look at the
> > installation code to make sure that it handles cases where multiple
> > certs with the same nick are present.
> > 
> >> Let me know if this stirs any thoughts, -Erinn
> > 
> > 
> 
> Well here is probably the pertinent part of the debug log, though
> there is a lot more when the clone is setting up:
> [31/Jul/2014:13:23:53][TP-Processor3]: AuthMgrName: certUserDBAuthMgr
> [31/Jul/2014:13:23:53][TP-Processor3]: CMSServlet: retrieving SSL
> certificate
> [31/Jul/2014:13:23:53][TP-Processor3]: CMSServlet: certUID=CN=CA
> Subsystem,O=example.COM
> [31/Jul/2014:13:23:53][TP-Processor3]: CertUserDBAuth: started
> [31/Jul/2014:13:23:53][TP-Processor3]: CertUserDBAuth: Retrieving
> client certificate
> [31/Jul/2014:13:23:53][TP-Processor3]: CertUserDBAuth: Got client
> certificate
> [31/Jul/2014:13:23:53][TP-Processor3]: In LdapBoundConnFactory::getConn()
> [31/Jul/2014:13:23:53][TP-Processor3]: masterConn is connected: true
> [31/Jul/2014:13:23:53][TP-Processor3]: getConn: conn is connected true
> [31/Jul/2014:13:23:53][TP-Processor3]: getConn: mNumConns now 2
> [31/Jul/2014:13:23:53][TP-Processor3]: returnConn: mNumConns now 3
> [31/Jul/2014:13:23:53][TP-Processor3]: Authentication: client
> certificate found
> [31/Jul/2014:13:23:53][TP-Processor3]: In LdapBoundConnFactory::getConn()
> [31/Jul/2014:13:23:53][TP-Processor3]: masterConn is connected: true
> [31/Jul/2014:13:23:53][TP-Processor3]: getConn: conn is connected true
> [31/Jul/2014:13:23:53][TP-Processor3]: getConn: mNumConns now 2
> [31/Jul/2014:13:23:53][TP-Processor3]: returnConn: mNumConns now 3
> [31/Jul/2014:13:23:53][TP-Processor3]: SignedAuditEventFactory:
> create()
> message=[AuditEvent=AUTH_FAIL][SubjectID=$Unidentified$][Outcome=Failure][AuthMgr=certUserDBAuthMgr][AttemptedCred=CN=CA
> Subsystem,O=example.COM] authentication failure
> 
> [31/Jul/2014:13:23:53][TP-Processor3]: CMSServlet: curDate=Thu Jul 31
> 13:23:53 GMT 2014 id=caUpdateDomainXML time=11
> 
Lets focus on the above error.  This says that the master was unable to
map the certificate that was presented to a user under ou=users,
o=ipaca.

I would look at the database (for the master) and see what users are
defined.  Check which users have the subsystem certificate defined as
their certificate, and check the description attribute.  That attribute
should contain a string that includes the certificate serial number,
subject DN and issuer, delimited by semicolons.  Check that string and
confirm that the certificate for that user matches the description
delimiter, and that the subsystem certificate is the same as the
subsystem certificate in the replica certdb.

It would also be useful to see what the DS access logs say at the time
this authentication failure occurs.

Ade
> 
> -Erinn
> 