[Freeipa-users] RHEL 7 Upgrade experience so far
Erinn Looney-Triggs
erinn.looneytriggs at gmail.com
Thu Jul 31 13:27:19 UTC 2014
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
On 07/30/2014 02:31 PM, Ade Lee wrote:
> On Tue, 2014-07-29 at 17:49 -0700, Erinn Looney-Triggs wrote:
>>>>
>>
>>>> Ok, well I tried deleting it using certutil it deletes both,
>>>> I tried using keytool to see if it would work any better, no
>>>> dice there. I'll try the rename, but at this point I am not
>>>> holding my breath on that, it seems all operation are a bit
>>>> too coarse. It seems the assumption was being made that there
>>>> would only be one of each nickname. Which frankly makes me
>>>> wonder how any of this kept running after the renewal.
>>>>
>>>> For now I'll see what I can do on a copy of the db using
>>>> python.
>>>
>>> It is a little strange that there are multiple 'caSigningCert
>>> cert-pki-ca' as this is the CA itself. It should be good for
>>> 20 years and isn't something that the current renewal code
>>> handles yet.
>>>
>>> You probably won't have much luck with python-nss. It can
>>> handle reading PKCS#12 files but I don't believe it can write
>>> them (access to key material).
>>>
>>> I'm not sure why certutil didn't do the trick. This should
>>> work, if you want to give it another try. I'm assuming that
>>> /root/cacert.p12 has the latest exported certs, adjust as
>>> necessary:
>>>
>>> # certutil -N -d /tmp/test # pk12util -i /root/cacert.p12 -d
>>> /tmp/test # certutil -D -d /tmp/test -n '<nickname>'
>>>
>>> certutil should delete the oldest cert first, it always has
>>> for me.
>>>
>>> rob
>>>
>>
>> Ok folks I managed to clean up the certificate DB so there is
>> only one valid certificate for each service. Installation
>> continued pass that step and then failed shortly thereafter on
>> configuring the ca. So here is my new error:
>>
>>
>> pkispawn : ERROR ....... Exception from Java Configuration
>> Servlet: Error while updating security domain:
>> java.io.IOException: 2 pkispawn : DEBUG ....... Error Type:
>> HTTPError pkispawn : DEBUG ....... Error Message: 500
>> Server Error: Internal Server Error pkispawn : DEBUG
>> ....... File "/usr/sbin/pkispawn", line 374, in main rv =
>> instance.spawn() File
>> "/usr/lib/python2.7/site-packages/pki/deployment/configuration.py",
>>
>>
line 128, in spawn
>> json.dumps(data, cls=pki.encoder.CustomTypeEncoder)) File
>> "/usr/lib/python2.7/site-packages/pki/deployment/pkihelper.py",
>> line 2998, in configure_pki_data response =
>> client.configure(data) File
>> "/usr/lib/python2.7/site-packages/pki/system.py", line 80, in
>> configure r = self.connection.post('/rest/installer/configure',
>> data, headers) File
>> "/usr/lib/python2.7/site-packages/pki/client.py", line 64, in
>> post r.raise_for_status() File
>> "/usr/lib/python2.7/site-packages/requests/models.py", line 638,
>> in raise_for_status raise http_error
>>
>>
>> 2014-07-30T00:27:48Z CRITICAL failed to configure ca instance
>> Command '/usr/sbin/pkispawn -vv -s CA -f /tmp/tmpqX9SGx' returned
>> non-zero exit status 1 2014-07-30T00:27:48Z DEBUG File
>> "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py",
>>
>>
line 638, in run_script
>> return_value = main_function()
>>
>> File "/usr/sbin/ipa-replica-install", line 667, in main CA =
>> cainstance.install_replica_ca(config)
>>
>> File
>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
>>
>>
line 1678, in install_replica_ca
>> subject_base=config.subject_base)
>>
>> File
>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
>>
>>
line 478, in configure_instance
>> self.start_creation(runtime=210)
>>
>> File
>> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
>> line 364, in start_creation method()
>>
>> File
>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
>>
>>
line 604, in __spawn_instance
>> raise RuntimeError('Configuration of CA failed')
>>
>> 2014-07-30T00:27:48Z DEBUG The ipa-replica-install command
>> failed, exception: RuntimeError: Configuration of CA failed
>>
>> And from the pki-tomcat/ca debug log: isSDHostDomainMaster():
>> Getting domain.xml from CA...
>> [30/Jul/2014:00:27:48][http-bio-8443-exec-3]: getDomainXML start
>> [30/Jul/2014:00:27:48][http-bio-8443-exec-3]: getDomainXML:
>> status=0 [30/Jul/2014:00:27:48][http-bio-8443-exec-3]:
>> getDomainXML: domainInfo=<?xml version="1.0" encoding="UTF-8"
>> standalone="no"?><DomainInfo><Name>IPA</Name><CAList><CA><Host>ipa.example.com</Host><SecurePort>443</SecurePort><SecureAgentPort>443</SecureAgentPort><SecureAdminPort>443</SecureAdminPort><SecureEEClientAuthPort>443</SecureEEClientAuthPort><UnSecurePort>80</UnSecurePort><Clone>FALSE</Clone><SubsystemName>pki-cad</SubsystemName><DomainManager>TRUE</DomainManager></CA><SubsystemCount>1</SubsystemCount></CAList><OCSPList><SubsystemCount>0</SubsystemCount></OCSPList><KRAList><SubsystemCount>0</SubsystemCount></KRAList><RAList><SubsystemCount>0</SubsystemCount></RAList><TKSList><SubsystemCount>0</SubsystemCount></TKSList><TPSList><SubsystemCount>0</SubsystemCount></TPSList></DomainInfo>
>>
>>
[30/Jul/2014:00:27:48][http-bio-8443-exec-3]: Cloning a domain master
>> [30/Jul/2014:00:27:48][http-bio-8443-exec-3]: WizardPanelBase
>> updateDomainXML start hostname=ipa.example.com port=443
>> [30/Jul/2014:00:27:48][http-bio-8443-exec-3]:
>> updateSecurityDomain: failed to update security domain using
>> admin port 443: org.xml.sax.SAXParseException; lineNumber: 1;
>> columnNumber: 50; White spaces are required between publicId and
>> systemId. [30/Jul/2014:00:27:48][http-bio-8443-exec-3]:
>> updateSecurityDomain: now trying agent port with client auth
>> [30/Jul/2014:00:27:48][http-bio-8443-exec-3]: WizardPanelBase
>> updateDomainXML start hostname=ipa.example.com port=443
>> [30/Jul/2014:00:27:48][http-bio-8443-exec-3]: updateDomainXML()
>> nickname=subsystemCert cert-pki-ca
>> [30/Jul/2014:00:27:48][http-bio-8443-exec-3]: WizardPanelBase
>> updateDomainXML: status=1
>>
>> And from pki-tomcat/catalina.out: 00:26:53,450 INFO
>> (org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher:82)
>>
>>
- - Deploying javax.ws.rs.core.Application: class
>> com.netscape.ca.CertificateAuthorityApplication 00:26:53,472
>> INFO
>> (org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher:82)
>>
>>
- - Adding singleton provider com.netscape.certsrv.acls.ACLInterceptor
>> from Application javax.ws.rs.core.Application 00:26:53,473 INFO
>> (org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher:82)
>>
>>
- - Adding singleton provider
>> com.netscape.certsrv.authentication.AuthMethodInterceptor from
>> Application javax.ws.rs.core.Application 00:26:53,772 DEBUG
>> (org.jboss.resteasy.core.SynchronousDispatcher:60) - PathInfo:
>> /installer/configure AuthInterceptor:
>> SystemConfigResource.configure() AuthInterceptor: mapping name:
>> default AuthInterceptor: required auth methods: [*]
>> AuthInterceptor: anonymous access allowed [Fatal Error] :1:50:
>> White spaces are required between publicId and systemId. [Fatal
>> Error] :1:50: White spaces are required between publicId and
>> systemId. [Fatal Error] :1:50: White spaces are required between
>> publicId and systemId. [Fatal Error] :1:50: White spaces are
>> required between publicId and systemId. java.io.IOException: 2
>> at
>> com.netscape.cms.servlet.csadmin.ConfigurationUtils.updateDomainXML(ConfigurationUtils.java:3415)
>>
>>
at
>> com.netscape.cms.servlet.csadmin.ConfigurationUtils.updateSecurityDomain(ConfigurationUtils.java:3345)
>>
>>
at
>> com.netscape.cms.servlet.csadmin.SystemConfigService.configure(SystemConfigService.java:655)
>>
>>
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>
>>
at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>>
at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:167)
>>
>>
at
>> org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257)
>>
>>
at
>> org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222)
>>
>>
at
>> org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211)
>>
>>
at
>> org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:542)
>>
>>
at
>> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:524)
>>
>>
at
>> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:126)
>>
>>
at
>> org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208)
>>
>>
at
>> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55)
>>
>>
at
>> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50)
>>
>>
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>
>>
at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>>
at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:277)
>>
>>
at
>> org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:274)
>>
>>
at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAsPrivileged(Subject.java:536)
>> at
>> org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:309)
>>
>>
at
>> org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:169)
>>
>>
at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:299)
>>
>>
at
>> org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:57)
>>
>>
at
>> org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:193)
>>
>>
at
>> org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:189)
>>
>>
at java.security.AccessController.doPrivileged(Native Method)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
>>
>>
at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>>
>>
at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>>
>>
at
>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
>>
>>
at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>>
>>
at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>>
>>
at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>>
>>
at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
>>
>>
at
>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1024)
>>
>>
at
>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>>
>>
at
>> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
>>
>>
at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>
>>
at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>
>>
at java.lang.Thread.run(Thread.java:745)
>>
>>
>
> Is there any indication of what the error is on the master CA? This
> would likely be in either the debug log or the catalina.out. Also,
> you should see the access to update the security domain in the
> httpd access log on the master.
>
>
>> I fixed the db (in case anyone else runs into this issue) by
>> doing the following:
>>
>> PKCS12Export of the NSS DB in order to get a .p12 file with all
>> the certificates.
>>
>> use openssl to convert the pkcs12 file to a single file in PEM
>> format with all of the certificates and the keys.
>>
>> From here unfortunately, you have to manually go in and find the
>> valid key/cert pairs in the pem file and create new PEM files for
>> each key pair you intend to import, ocsp, server cert, etc.
>> Obviously only grab one key pair for each, and only the valid
>> ones. Openssl does not support mass importing of key/certificate
>> pairs into a PKCS12 file.
>>
>> Once you have a pem file for each service, you then need to
>> convert these pem files back into PKCS12 format, one at a time,
>> using the -name flag to give them friendly names.
>>
>> After this create a new NSS DB using certutil, and import each
>> PKCS12 file for each service into the DB.
>>
>> I don't know if this is necessary, but I set the flags to be
>> identical to the original DB for the certs.
>>
>> Now use PKCS12Export to export your newly created NSS DB into a
>> cacert.p12 file. You now should have a nice new cacert.p12 file
>> with only valid certificates.
>>
>> Most of the user space tools for handling NSS and PKCS12 files
>> are not flexible enough to get what you want done. This could
>> probably be coded up in a more efficient way.
>>
>
> Thanks for the steps above. We'll be sure to keep them handy in
> case this happens again, and I think we need to look at the
> installation code to make sure that it handles cases where multiple
> certs with the same nick are present.
>
>> Let me know if this stirs any thoughts, -Erinn
>
>
Well here is probably the pertinent part of the debug log, though
there is a lot more when the clone is setting up:
[31/Jul/2014:13:23:53][TP-Processor3]: AuthMgrName: certUserDBAuthMgr
[31/Jul/2014:13:23:53][TP-Processor3]: CMSServlet: retrieving SSL
certificate
[31/Jul/2014:13:23:53][TP-Processor3]: CMSServlet: certUID=CN=CA
Subsystem,O=example.COM
[31/Jul/2014:13:23:53][TP-Processor3]: CertUserDBAuth: started
[31/Jul/2014:13:23:53][TP-Processor3]: CertUserDBAuth: Retrieving
client certificate
[31/Jul/2014:13:23:53][TP-Processor3]: CertUserDBAuth: Got client
certificate
[31/Jul/2014:13:23:53][TP-Processor3]: In LdapBoundConnFactory::getConn()
[31/Jul/2014:13:23:53][TP-Processor3]: masterConn is connected: true
[31/Jul/2014:13:23:53][TP-Processor3]: getConn: conn is connected true
[31/Jul/2014:13:23:53][TP-Processor3]: getConn: mNumConns now 2
[31/Jul/2014:13:23:53][TP-Processor3]: returnConn: mNumConns now 3
[31/Jul/2014:13:23:53][TP-Processor3]: Authentication: client
certificate found
[31/Jul/2014:13:23:53][TP-Processor3]: In LdapBoundConnFactory::getConn()
[31/Jul/2014:13:23:53][TP-Processor3]: masterConn is connected: true
[31/Jul/2014:13:23:53][TP-Processor3]: getConn: conn is connected true
[31/Jul/2014:13:23:53][TP-Processor3]: getConn: mNumConns now 2
[31/Jul/2014:13:23:53][TP-Processor3]: returnConn: mNumConns now 3
[31/Jul/2014:13:23:53][TP-Processor3]: SignedAuditEventFactory:
create()
message=[AuditEvent=AUTH_FAIL][SubjectID=$Unidentified$][Outcome=Failure][AuthMgr=certUserDBAuthMgr][AttemptedCred=CN=CA
Subsystem,O=example.COM] authentication failure
[31/Jul/2014:13:23:53][TP-Processor3]: CMSServlet: curDate=Thu Jul 31
13:23:53 GMT 2014 id=caUpdateDomainXML time=11
- -Erinn
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBCAAGBQJT2kQxAAoJEFg7BmJL2iPO0rkH/0ewUiqvfcHMQ9F3ByfqbL12
JkyHkSZRQXaGswro69uIl5q+1Ra5FX9FwtoGuslaxlUNmP367i5CP1EN0zZjhJyj
4NBhypJdiCwnLFPBm2eHnRj0hdfDOBBUjOgsmtspJsy/q2Fzo/rTKp9ftUwtxjDA
RBexiC9hlX6mEQH62zxCT9kUrU2UawH/27ioZvFHkI/zhO70/j9XRihlRNepV99p
gxP1weBvDTWwmYQ7tM438UZCYItcQAzovLIMOVh6HZ7uRYgCTBIsQrHmfiDCJEN9
FLIVr1RE+syBQH8KHyxhXH00zMqO4Q5JRrQATNYw1xuIF6auBIjadI/OO6zlJbk=
=/djw
-----END PGP SIGNATURE-----
More information about the Freeipa-users
mailing list