[Freeipa-users] RHEL 7 Upgrade experience so far

Thu Aug 28 08:58:51 UTC 2014

I have been following this thread with great interest, as I have
encountered similar problems with our migration from 3.0.0-37 on CentOS
6.5 to 3.3.3-28 on CentOS 7. I have been able to solve a few of them
with manual patching, but there is still something going on that will
make the CA replication to fail.

The following changes have been made to the environments:

- On the replica,
/usr/lib/python2.7/site-packages/ipaserver/install/replication.py has
been patched to handle multiple values of nsDS5ReplicaId on the master.

- /usr/share/ipa/html/ca.crt used to contain our local root certificate
as well as the IPA CA-certificate, which caused the replica installation
to fail. The root certificate was removed from this file, the replica
gpg-bundle recreated, and the installation would happily continue.

- /etc/httpd/conf.d/ipa-pki-proxy.conf has been patched to contain the
profileSubmit-patch to the ee port-line
<LocationMatch
"^/ca/ee/ca/checkRequest|^/ca/ee/ca/getCertChain|^/ca/ee/ca/getTokenInfo|^/ca/ee/ca/tokenAuthenticate|^/ca/ocsp|^/ca/ee/ca/updateNumberRange|^/ca/ee/ca/getCRL|^/ca/ee/ca/profileSubmit">

and have also tried with and without the additions to the admin port and
installer-line

<LocationMatch
"^/ca/admin/ca/getCertChain|^/ca/admin/ca/getConfigEntries|^/ca/admin/ca/getCookie|^/ca/admin/ca/getStatus|^/ca/admin/ca/securityDomainLogin|^/ca/admin/ca/getDomainXML|^/ca/rest/installer/installToken|^/ca/admin/ca/updateNumberRange|^/ca/rest/securityDomain/domainInfo|^/ca/rest/account/login|^/ca/admin/ca/tokenAuthenticate|^/ca/admin/ca/updateNumberRange|^/ca/admin/ca/updateDomainXML|^/ca/rest/account/logout|^/ca/rest/securityDomain/installToken">

Checking the log files on the 3.3.3 replica, there are a few error
messages, which I am not sure how to resolve.

/var/log/ipareplica-install.log ends with the following lines:

2014-08-27T14:44:15Z DEBUG Starting external process
2014-08-27T14:44:15Z DEBUG args=/usr/sbin/pkispawn -s CA -f /tmp/tmpxkixl8
2014-08-27T14:45:19Z DEBUG Process finished, return code=1
2014-08-27T14:45:19Z DEBUG stdout=Loading deployment configuration from
/tmp/tmpxkixl8.
Installing CA into /var/lib/pki/pki-tomcat.
Storing deployment configuration into
/etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg.
Installation failed.

2014-08-27T14:45:19Z DEBUG stderr=pkispawn    : WARNING  ....... unable
to validate security domain user/password through REST interface.
Interface not available
pkispawn    : ERROR    ....... Exception from Java Configuration
Servlet: Error while updating security domain: java.io.IOException: 2

2014-08-27T14:45:19Z CRITICAL failed to configure ca instance Command
'/usr/sbin/pkispawn -s CA -f /tmp/tmpxkixl8' returned non-zero exit status 1
2014-08-27T14:45:19Z DEBUG   File
"/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py",
line 638, in run_script
    return_value = main_function()

  File "/usr/sbin/ipa-replica-install", line 667, in main
    CA = cainstance.install_replica_ca(config)

  File
"/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line
1678, in install_replica_ca
    subject_base=config.subject_base)

  File
"/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line
478, in configure_instance
    self.start_creation(runtime=210)

  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
line 364, in start_creation
    method()

  File
"/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line
604, in __spawn_instance
    raise RuntimeError('Configuration of CA failed')

2014-08-27T14:45:19Z DEBUG The ipa-replica-install command failed,
exception: RuntimeError: Configuration of CA failed

/var/log/pki/pki-ca-spawn.20140827164415.log reveals these error messages:

2014-08-27 16:44:16 pkispawn    : INFO     ....... executing 'systemctl
start pki-tomcatd at pki-tomcat.service'
2014-08-27 16:44:18 pkispawn    : DEBUG    ........... No connection -
server may still be down
2014-08-27 16:44:18 pkispawn    : DEBUG    ........... No connection -
exception thrown: [Errno 111] Connection refused
2014-08-27 16:44:26 pkispawn    : DEBUG    ........... <?xml
version="1.0" encoding="UTF-8"
standalone="no"?><XMLResponse><State>0</State><Type>CA</Type><Status>running</Status><Version>10.0.5-3.el7</Version></XMLResponse>
2014-08-27 16:44:27 pkispawn    : INFO     ....... constructing PKI
configuration data.
2014-08-27 16:44:27 pkispawn    : INFO     ....... configuring PKI
configuration data.
2014-08-27 16:45:19 pkispawn    : ERROR    ....... Exception from Java
Configuration Servlet: Error while updating security domain:
java.io.IOException: 2
2014-08-27 16:45:19 pkispawn    : DEBUG    ....... Error Type: HTTPError
2014-08-27 16:45:19 pkispawn    : DEBUG    ....... Error Message: 500
Server Error: Internal Server Error
2014-08-27 16:45:19 pkispawn    : DEBUG    .......   File
"/usr/sbin/pkispawn", line 374, in main
    rv = instance.spawn()
  File
"/usr/lib/python2.7/site-packages/pki/deployment/configuration.py", line
128, in spawn
    json.dumps(data, cls=pki.encoder.CustomTypeEncoder))
  File "/usr/lib/python2.7/site-packages/pki/deployment/pkihelper.py",
line 2998, in configure_pki_data
    response = client.configure(data)
  File "/usr/lib/python2.7/site-packages/pki/system.py", line 80, in
configure
    r = self.connection.post('/rest/installer/configure', data, headers)
  File "/usr/lib/python2.7/site-packages/pki/client.py", line 64, in post
    r.raise_for_status()
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 638,
in raise_for_status
    raise http_error

In /var/log/pki/pki-tomcat/catalina.out one can read:

Aug 27, 2014 4:44:22 PM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deploying web application directory /var/lib/pki/pki-tomcat/webapps/ca
SSLAuthenticatorWithFallback: Creating SSL authenticator with fallback
SSLAuthenticatorWithFallback: Setting container
SSLAuthenticatorWithFallback: Initializing authenticators
SSLAuthenticatorWithFallback: Starting authenticators
CMS Warning: FAILURE: Cannot build CA chain. Error
java.security.cert.CertificateException: Certificate is not a PKCS #11
certificate|FAILURE: authz instance DirAclAuthz initializa
tion failed and skipped, error=Property internaldb.ldapconn.port missing
value|
Server is started.
Aug 27, 2014 4:44:26 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8080"]
Aug 27, 2014 4:44:26 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8443"]
Aug 27, 2014 4:44:26 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-bio-127.0.0.1-8009"]
Aug 27, 2014 4:44:26 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 7872 ms
16:44:27,950  INFO
(org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher:82) -
Deploying javax.ws.rs.core.Application: class
com.netscape.ca.CertificateAuthorityApplication
16:44:27,967  INFO
(org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher:82) -
Adding singleton provider com.netscape.certsrv.acls.ACLInterceptor from
Application javax.ws.rs.core.Application
16:44:27,968  INFO
(org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher:82) -
Adding singleton provider
com.netscape.certsrv.authentication.AuthMethodInterceptor from
Application javax.ws.rs.core.Application
16:44:28,433 DEBUG (org.jboss.resteasy.core.SynchronousDispatcher:60) -
PathInfo: /installer/configure
AuthInterceptor: SystemConfigResource.configure()
AuthInterceptor: mapping name: default
AuthInterceptor: required auth methods: [*]
AuthInterceptor: anonymous access allowed
java.io.IOException: 2
        at
com.netscape.cms.servlet.csadmin.ConfigurationUtils.updateDomainXML(ConfigurationUtils.java:3415)
        at
com.netscape.cms.servlet.csadmin.ConfigurationUtils.updateSecurityDomain(ConfigurationUtils.java:3345)
        at
com.netscape.cms.servlet.csadmin.SystemConfigService.configure(SystemConfigService.java:655)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at
org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:167)
        at
org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257)
        at
org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222)
        at
org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211)
        at
org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:542)
        at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:524)
        at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:126)
        at
org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208)
        at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55)
        at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:277)
        at
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:274)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAsPrivileged(Subject.java:536)
        at
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:309)
        at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:169)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:299)
        at
org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:57)
        at
org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:193)
        at
org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:189)
        at java.security.AccessController.doPrivileged(Native Method)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
        at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
        at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1024)
        at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
        at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)

/var/log/pki/pki-tomcat/ca/debug may give a clue aswell:

[27/Aug/2014:16:45:19][http-bio-8443-exec-3]: isSDHostDomainMaster():
Getting domain.xml from CA...
[27/Aug/2014:16:45:19][http-bio-8443-exec-3]: getDomainXML start
[27/Aug/2014:16:45:19][http-bio-8443-exec-3]: getDomainXML: status=0
[27/Aug/2014:16:45:19][http-bio-8443-exec-3]: getDomainXML:
domainInfo=<?xml version="1.0" encoding="UTF-8"
standalone="no"?><DomainInfo><Name>IPA</Name><CAList><CA><Host>ipa.skalarit.net</Host><SecurePort>443</SecurePort><SecureAgentPort>443</SecureAgentPort><SecureAdminPort>443</SecureAdminPort><SecureEEClientAuthPort>443</SecureEEClientAuthPort><UnSecurePort>80</UnSecurePort><Clone>FALSE</Clone><SubsystemName>pki-cad</SubsystemName><DomainManager>TRUE</DomainManager></CA><SubsystemCount>1</SubsystemCount></CAList><OCSPList><SubsystemCount>0</SubsystemCount></OCSPList><KRAList><SubsystemCount>0</SubsystemCount></KRAList><RAList><SubsystemCount>0</SubsystemCount></RAList><TKSList><SubsystemCount>0</SubsystemCount></TKSList><TPSList><SubsystemCount>0</SubsystemCount></TPSList></DomainInfo>
[27/Aug/2014:16:45:19][http-bio-8443-exec-3]: Cloning a domain master
[27/Aug/2014:16:45:19][http-bio-8443-exec-3]: WizardPanelBase
updateDomainXML start hostname=ipa.skalarit.net port=443
[27/Aug/2014:16:45:19][http-bio-8443-exec-3]: updateSecurityDomain:
failed to update security domain using admin port 443:
java.io.IOException: Failed to get response when updating security domain
[27/Aug/2014:16:45:19][http-bio-8443-exec-3]: updateSecurityDomain: now
trying agent port with client auth
[27/Aug/2014:16:45:19][http-bio-8443-exec-3]: WizardPanelBase
updateDomainXML start hostname=ipa.skalarit.net port=443
[27/Aug/2014:16:45:19][http-bio-8443-exec-3]: updateDomainXML()
nickname=subsystemCert cert-pki-ca
[27/Aug/2014:16:45:19][http-bio-8443-exec-3]: WizardPanelBase
updateDomainXML: status=1

/Nicklas

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 246 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20140828/d0728e4b/attachment.sig>