[Freeipa-devel] [PATCH] 762 Let the framework be able to override the hostname

Wed Jun 22 12:51:04 UTC 2011

Rob Crittenden wrote:
> Martin Kosek wrote:
>> On Fri, 2011-05-27 at 15:39 -0400, Rob Crittenden wrote:
>>> Martin Kosek wrote:
>>>> On Wed, 2011-05-25 at 11:29 -0400, Rob Crittenden wrote:
>>>>> Martin Kosek wrote:
>>>>>> On Fri, 2011-04-01 at 11:47 -0400, Rob Crittenden wrote:
>>>>>>> The hostname is passed in during the server installation. We
>>>>>>> should use
>>>>>>> this hostname for the resulting server as well. It was being
>>>>>>> discarded
>>>>>>> and we always used the system hostname value.
>>>>>>>
>>>>>>> ticket 1052
>>>>>>>
>>>>>>> rob
>>>>>>
>>>>>> I have to NACK this again. I have a problem communicating with IPA
>>>>>> on a
>>>>>> master machine. I reproduced in on 2 different machines. Please,
>>>>>> correct
>>>>>> my steps if I am wrong, I do the following procedure
>>>>>>
>>>>>> 1) I prepare a fresh minimal F-15
>>>>>> 2) Install freeipa-server (current master with your patches)
>>>>>> 3) Add custom hostname to /etc/hosts
>>>>>> 4) Install IPA server:
>>>>>> ipa-server-install -p secret123 -a secret123 --hostname
>>>>>> ipa.idm.lab.bos.redhat.com --setup-dns --forwarder=10.16.255.2
>>>>>> 5) # kinit admin
>>>>>> Password for admin at IDM.LAB.BOS.REDHAT.COM:
>>>>>> 6) # ipa user-show admin
>>>>>> ipa: ERROR: cannot connect to 'any of the configured servers':
>>>>>> https://ipa.idm.lab.bos.redhat.com/ipa/xml,
>>>>>> https://ipa.idm.lab.bos.redhat.com/ipa/xml
>>>>>>
>>>>>> # ping -c 1 ipa.idm.lab.bos.redhat.com
>>>>>> PING ipa.idm.lab.bos.redhat.com (10.16.78.140) 56(84) bytes of data.
>>>>>> 64 bytes from ipa.idm.lab.bos.redhat.com (10.16.78.140): icmp_req=1
>>>>>> ttl=64 time=0.049 ms
>>>>>>
>>>>>> Apache error_log shows relevant errors:
>>>>>>
>>>>>> [Wed May 25 06:42:38 2011] [error] ipa: ERROR: Failed to start
>>>>>> IPA: Unable to retrieve LDAP schema: Invalid credentials:
>>>>>> SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.
>>>>>> Minor code may provide more information (Permission denied)
>>>>>> [Wed May 25 06:42:38 2011] [error] ipa: ERROR: Failed to start
>>>>>> IPA: Unable to retrieve LDAP schema: Invalid credentials:
>>>>>> SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.
>>>>>> Minor code may provide more information (Permission denied)
>>>>>> [Wed May 25 06:43:55 2011] [error] Exception KeyError:
>>>>>> KeyError(140250828974112,) in<module 'threading' from
>>>>>> '/usr/lib64/python2.7/threading.pyc'> ignored
>>>>>> [Wed May 25 06:43:55 2011] [error] Exception KeyError:
>>>>>> KeyError(140250828974112,) in<module 'threading' from
>>>>>> '/usr/lib64/python2.7/threading.pyc'> ignored
>>>>>> [Wed May 25 06:43:55 2011] [error] Exception KeyError:
>>>>>> KeyError(140250828974112,) in<module 'threading' from
>>>>>> '/usr/lib64/python2.7/threading.pyc'> ignored
>>>>>> [Wed May 25 06:43:55 2011] [error] Exception KeyError:
>>>>>> KeyError(140250828974112,) in<module 'threading' from
>>>>>> '/usr/lib64/python2.7/threading.pyc'> ignored
>>>>>> [Wed May 25 06:43:55 2011] [error] Exception KeyError:
>>>>>> KeyError(140250828974112,) in<module 'threading' from
>>>>>> '/usr/lib64/python2.7/threading.pyc'> ignored
>>>>>> [Wed May 25 06:43:55 2011] [error] Exception KeyError:
>>>>>> KeyError(140250828974112,) in<module 'threading' from
>>>>>> '/usr/lib64/python2.7/threading.pyc'> ignored
>>>>>> [Wed May 25 06:43:55 2011] [error] Exception KeyError:
>>>>>> KeyError(140250828974112,) in<module 'threading' from
>>>>>> '/usr/lib64/python2.7/threading.pyc'> ignored
>>>>>> [Wed May 25 06:43:55 2011] [error] Exception KeyError:
>>>>>> KeyError(140250828974112,) in<module 'threading' from
>>>>>> '/usr/lib64/python2.7/threading.pyc'> ignored
>>>>>> [Wed May 25 06:43:55 2011] [error] Exception KeyError:
>>>>>> KeyError(140250828974112,) in<module 'threading' from
>>>>>> '/usr/lib64/python2.7/threading.pyc'> ignored
>>>>>> [Wed May 25 06:43:55 2011] [error] Exception KeyError:
>>>>>> KeyError(140250828974112,) in<module 'threading' from
>>>>>> '/usr/lib64/python2.7/threading.pyc'> ignored
>>>>>> [Wed May 25 06:43:56 2011] [notice] caught SIGTERM, shutting down
>>>>>> [Wed May 25 06:43:56 2011] [notice] SELinux policy enabled; httpd
>>>>>> running as context system_u:system_r:kernel_t:s0
>>>>>> [Wed May 25 06:43:57 2011] [notice] Digest: generating secret for
>>>>>> digest authentication ...
>>>>>> [Wed May 25 06:43:57 2011] [notice] Digest: done
>>>>>> [Wed May 25 06:43:57 2011] [notice] Apache/2.2.17 (Unix) DAV/2
>>>>>> mod_auth_kerb/5.4 mod_nss/2.2.17 NSS/3.12.9.0 mod_wsgi/3.2
>>>>>> Python/2.7.1 configured -- resuming normal operations
>>>>>> [Wed May 25 06:44:04 2011] [error] ipa: INFO: *** PROCESS START ***
>>>>>> [Wed May 25 06:44:04 2011] [error] ipa: INFO: *** PROCESS START ***
>>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140] mod_wsgi
>>>>>> (pid=5192): Exception occurred processing WSGI script
>>>>>> '/usr/share/ipa/wsgi.py'.
>>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140] Traceback
>>>>>> (most recent call last):
>>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140] File
>>>>>> "/usr/share/ipa/wsgi.py", line 48, in application
>>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140] return
>>>>>> api.Backend.session(environ, start_response)
>>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140] File
>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line
>>>>>> 141, in __call__
>>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140]
>>>>>> self.create_context(ccache=environ.get('KRB5CCNAME'))
>>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140] File
>>>>>> "/usr/lib/python2.7/site-packages/ipalib/backend.py", line 110, in
>>>>>> create_context
>>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140]
>>>>>> self.Backend.ldap2.connect(ccache=ccache)
>>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140] File
>>>>>> "/usr/lib/python2.7/site-packages/ipalib/backend.py", line 62, in
>>>>>> connect
>>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140] conn =
>>>>>> self.create_connection(*args, **kw)
>>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140] File
>>>>>> "/usr/lib/python2.7/site-packages/ipalib/encoder.py", line 188, in
>>>>>> new_f
>>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140] return
>>>>>> f(*new_args, **kwargs)
>>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140] File
>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/plugins/ldap2.py",
>>>>>> line 337, in create_connection
>>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140]
>>>>>> _handle_errors(e, **{})
>>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140] File
>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/plugins/ldap2.py",
>>>>>> line 118, in _handle_errors
>>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140] raise
>>>>>> errors.DatabaseError(desc=desc, info=info)
>>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140]
>>>>>> DatabaseError: Local error: SASL(-1): generic failure: GSSAPI
>>>>>> Error: An invalid name was supplied (Hostname cannot be
>>>>>> canonicalized)
>>>>>> [Wed May 25 06:45:26 2011] [error] [client 10.16.78.140] mod_wsgi
>>>>>> (pid=5193): Exception occurred processing WSGI script
>>>>>> '/usr/share/ipa/wsgi.py'.
>>>>>>
>>>>>>
>>>>>> You can check the problem on vm-140.idm.lab.bos.redhat.com if you
>>>>>> want to.
>>>>>>
>>>>>> Martin
>>>>>>
>>>>>
>>>>> The LDAP connection was still using the system hostname value. I
>>>>> added a
>>>>> conn.set_option(_ldap.OPT_HOST_NAME, api.env.host) in the two
>>>>> places we
>>>>> initialize an LDAP connection and that seems to have fixed it.
>>>>>
>>>>> Updated patch attached
>>>>>
>>>>> rob
>>>>
>>>> NACK. The problem on a master is gone. However, now ipa-replica-install
>>>> is failing:
>>>>
>>>> # ipa-replica-install
>>>> /home/mkosek/replica-info-vm-027.idm.lab.bos.redhat.com.gpg
>>>> Directory Manager (existing master) password:
>>>>
>>>> creation of replica failed: Can't contact LDAP server:
>>>>
>>>>
>>>> I found out that the root cause of the failure is in the change you
>>>> just
>>>> made in ldap2.py:
>>>>
>>>> def create_connection(self, ccache=None, bind_dn='', bind_pw='',
>>>> tls_cacertfile=None, tls_certfile=None, tls_keyfile=None,
>>>> debug_level=0):
>>>> ...
>>>> try:
>>>> conn = _ldap.initialize(self.ldap_uri)
>>>> conn.set_option(_ldap.OPT_HOST_NAME, api.env.host)<--
>>>> if ccache is not None:
>>>> os.environ['KRB5CCNAME'] = ccache
>>>> ...
>>>>
>>>> because api.env.host points to the local host and not the remote
>>>> master.
>>>> When I commented this line out, installation continued OK. Then, it
>>>> crashed again with our "favorite" dogtag's "invalid clone_uri"
>>>> exception.
>>>>
>>>> Since we see this error also in other scenarios (not only custom
>>>> --hostname) and the root cause is not in your patch I can ACK you patch
>>>> 762 once the replica install bug is fixed.
>>>>
>>>> Martin
>>>>
>>>
>>> Fixed both of these. We only need to set the hostname when using an
>>> ldapi URI, so fixed both of those.
>>>
>>> I also fixed the Invalid clone_uri bug. The problem was we weren't
>>> passing our new hostname to pkicreate so it was creating a CA for
>>> whatever the value of `hostname` was. There is an environment variable
>>> in pkicreate to pass in the hostname and doing that has fixed the
>>> problem.
>>>
>>> rob
>>
>> Yes, this issue was fixed. It's good you find a way how to deal with
>> clone_uri problem. However, I still hit some issues:
>>
>> 1) I think we have some Kerberos related problems when the custom
>> hostname is used (ipa.idm.lab.bos.redhat.com on a
>> vm-096.idm.lab.bos.redhat.com). Named and SSSD refuses to start on the
>> system.
>>
>> /var/log/messages:
>> May 30 05:04:35 vm-096 named[13932]: listening on IPv4 interface eth0,
>> 10.16.78.96#53
>> May 30 05:04:35 vm-096 named[13932]: generating session key for
>> dynamic DNS
>> May 30 05:04:36 vm-096 named[13932]: Failed to init credentials
>> (Preauthentication failed)
>> May 30 05:04:36 vm-096 named[13932]: loading configuration: failure
>> May 30 05:04:36 vm-096 named[13932]: exiting (due to fatal error)
>> May 30 05:04:36 vm-096 systemd[1]: named.service: control process
>> exited, code=exited status=7
>> May 30 05:04:36 vm-096 systemd[1]: Unit named.service entered failed
>> state.
>> May 30 05:07:41 vm-096 sssd: Starting up
>> May 30 05:07:41 vm-096 sssd[be[idm.lab.bos.redhat.com]]: Starting up
>> May 30 05:07:41 vm-096 sssd[be[idm.lab.bos.redhat.com]]: Error
>> processing keytab file [(null)]: Principal
>> [host/vm-096.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM] was not
>> found. Unable to create GSSAPI-encrypted LDAP connection.
>
> For the named issue I filed a bug against bind-dyndb-ldap for this,
> https://bugzilla.redhat.com/show_bug.cgi?id=710261
>
> This is a similar problem I ran into where when you do an ldapi bind it
> defaults to using the system hostname value.
>
> To fix the sssd problem we just need to set the ipa_hostname option
> (they have lots of nice tuning options!). We just need to decide if we
> always set this value or only at install time when the hostnames differ.
>
>> 2) My dogtag powered replica still refuses to install (happened to me on
>> 2 fresh VMs) with "creation of replica failed: Configuration of CA
>> failed".
>>
>> I investigated the ipareplica-install.log, I found a error that may be
>> relevant. Maybe Ade will recognize some of them.
>>
>> #############################################
>> Attempting to connect to: vm-028.idm.lab.bos.redhat.com:9445
>> Connected.
>> Posting Query =
>> https://vm-028.idm.lab.bos.redhat.com:9445//ca/admin/console/config/wizard?p=9&op=next&xml=true&host=vm-028.idm.lab.bos.redhat.com&port=7389&binddn=cn%3DDirectory+Manager&__bindpwd=XXXXXXXX&basedn=o%3Dipaca&database=ipaca&display=%24displayStr&cloneStartTLS=on
>>
>> RESPONSE STATUS: HTTP/1.1 200 OK
>> RESPONSE HEADER: Server: Apache-Coyote/1.1
>> RESPONSE HEADER: Content-Type: application/xml;charset=UTF-8
>> RESPONSE HEADER: Date: Mon, 30 May 2011 11:26:29 GMT
>> RESPONSE HEADER: Connection: close
>> ...
>> <response>
>> <panel>admin/console/config/databasepanel.vm</panel>
>> <clone>clone</clone>
>> <res/>
>> <portStr>7389</portStr>
>> <bindpwd>(sensitive)</bindpwd>
>> <cloneStartTLS>on</cloneStartTLS>
>> <hostname>vm-028.idm.lab.bos.redhat.com</hostname>
>> <errorString>Master and clone should have the same base DN</errorString>
>>
>>
>> The CA installation fails few error messages later.
>>
>> Providing excerpt of CA logs as they may be relevant:
>>
>> /var/log/pki-ca/catalina.out:
>> ...
>> CMS Warning: FAILURE: Cannot build CA chain. Error
>> java.security.cert.CertificateException: Certificate is not a PKCS #11
>> certificate|FAILURE: authz instance DirAclAuthz initialization failed
>> and skipped, error=Property internaldb.ldapconn.port missing value|
>> ...
>> [Fatal Error] :2:15: Open quote is expected for attribute "BGCOLOR"
>> associated with an element type "BODY".
>>
>> /var/log/pki-ca/system:
>> 2893.main - [30/May/2011:07:25:47 EDT] [3] [3] Cannot build CA chain.
>> Error java.security.cert.CertificateException: Certificate is not a
>> PKCS #11 certificate
>> 2893.main - [30/May/2011:07:25:47 EDT] [13] [3] authz instance
>> DirAclAuthz initialization failed and skipped, error=Property
>> internaldb.ldapconn.port missing value
>>
>> Martin
>>
>
> Haven't had a chance to explore this one yet. It sure would be nice if
> dogtag would tell us what the two differing base DNs are though...

This patch should resolve the remaining issues. It requires a patch to 
bind-dyndb-ldap, I have a candidate patch in 
https://bugzilla.redhat.com/show_bug.cgi?id=710261

rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-rcrit-762-4-host.patch
Type: text/x-diff
Size: 9780 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20110622/ee0bffb4/attachment.bin>