[Freeipa-users] RHEL 7 Upgrade experience so far

Mon Jul 28 14:41:00 UTC 2014

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 07/28/2014 07:17 AM, Rob Crittenden wrote:
> Rob Crittenden wrote:
>> Erinn Looney-Triggs wrote:
>>> On 07/27/2014 12:02 AM, Erinn Looney-Triggs wrote:
>>>> On 07/26/2014 07:12 PM, Erinn Looney-Triggs wrote:
>>>>> On 07/26/2014 05:25 PM, Erinn Looney-Triggs wrote:
>>>>>> Well it hasn't been all the pretty trying to move from
>>>>>> RHEL 6.5 to RHEL 7.
>>> 
>>>>>> I have two servers providing my ipa instances ipa and
>>>>>> ipa2. Given that I don't have a great deal of spare
>>>>>> capacity the plan was to remove ipa2 from the replication
>>>>>> agreement, modify DNS so that only IPA was available in
>>>>>> SRV logs (IPA does not manage DNS at this point, was
>>>>>> waiting for DNSSEC). As well, I would change my sudo-ldap
>>>>>> config files to point to ipa and remove ipa2.
>>> 
>>>>>> Well that all worked well, installed RHEL 7 on the system
>>>>>> and began working through the steps in the upgrade
>>>>>> guide.
>>> 
>>>>>> First major problem was running into this bug: 
>>>>>> https://fedorahosted.org/freeipa/ticket/4375 ValueError:
>>>>>>  nsDS5ReplicaId has 2 values, one expected.
>>> 
>>>>>> Went and patched the replication.py file to get around
>>>>>> that issue, and we moved on.
>>> 
>>>>>> Next up is my current issue: Exception from Java
>>>>>> Configuration Servlet: Clone does not have all the
>>>>>> required certificates.
>>> 
>>>>>> I suspect this is because I am running the CA as a
>>>>>> subordinate to an AD CS instance, but I am unsure at this
>>>>>> point.
>>> 
>>>>>> It has been a haul to get here, despite the short
>>>>>> explanation. It seems that my primary ipa instance is
>>>>>> working on only a hit or miss basis for kerberos tickets
>>>>>> which has made all this a bit of a pain. You can kinit as
>>>>>> admin once it will fail unable to find KDC, try again
>>>>>> another three times, it will work. I have even modified
>>>>>> the krb5.conf file to point directly at the server, thus
>>>>>> bypassing DNS SRV lookups, however, that hasn't worked.
>>> 
>>>>>> Point is, any help would be appreciated on the
>>>>>> aforementioned error.
>>> 
>>>>>> -Erinn
>>> 
>>> 
>>>>> To reply to myself here, I believe the problem may be that
>>>>> I had to renew the CA certificates and as such the
>>>>> certificates in /root/cacert.p12 are no longer valid. It is
>>>>> this file that gets bundled up with whatever else using
>>>>> ipa-replica-prepare, so I will have to create a new one
>>>>> that has the valid certificates in it.
>>> 
>>>>> One way or another though, if it isn't already documented,
>>>>> during a CA renewal this file should probably be updated
>>>>> with the correct certificates.
>>> 
>>>>> -Erinn
>>> 
>>>>> -Erinn
>>> 
>>> 
>>> 
>>>> Well thanks to this: 
>>>> http://www.freeipa.org/page/Howto/Change_Directory_Manager_Password
>>>
>>>>
>>>> 
I have gotten a little further down the road an created a new
>>>> cacert.p12 which looks to be complete.
>>> 
>>>> However, installation still fails in the same place:
>>> 
>>>> 2014-07-27T06:33:04Z DEBUG Starting external process 
>>>> 2014-07-27T06:33:04Z DEBUG args=/usr/sbin/pkispawn -s CA -f 
>>>> /tmp/tmp5QGhUx 2014-07-27T06:33:25Z DEBUG Process finished,
>>>> return code=1 2014-07-27T06:33:25Z DEBUG stdout=Loading
>>>> deployment configuration from /tmp/tmp5QGhUx. Installing CA
>>>> into /var/lib/pki/pki-tomcat. Storing deployment
>>>> configuration into 
>>>> /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg. 
>>>> Installation failed.
>>> 
>>> 
>>>> 2014-07-27T06:33:25Z DEBUG stderr=pkispawn    : WARNING
>>>> ....... unable to validate security domain user/password
>>>> through REST interface. Interface not available pkispawn    :
>>>> ERROR    ....... Exception from Java Configuration Servlet:
>>>> Clone does not have all the required certificates
>>> 
>>>> 2014-07-27T06:33:25Z CRITICAL failed to configure ca
>>>> instance Command '/usr/sbin/pkispawn -s CA -f /tmp/tmp5QGhUx'
>>>> returned non-zero exit status 1 2014-07-27T06:33:25Z DEBUG
>>>> File 
>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py",
>>>
>>>
>>>
>>>> 
line 638, in run_script
>>>> return_value = main_function()
>>> 
>>>> File "/usr/sbin/ipa-replica-install", line 667, in main CA = 
>>>> cainstance.install_replica_ca(config)
>>> 
>>>> File 
>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
>>>
>>>
>>>
>>>> 
line 1678, in install_replica_ca
>>>> subject_base=config.subject_base)
>>> 
>>>> File 
>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
>>>
>>>
>>>
>>>> 
line 478, in configure_instance
>>>> self.start_creation(runtime=210)
>>> 
>>>> File 
>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
>>>>
>>>> 
line 364, in start_creation method()
>>> 
>>>> File 
>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
>>>
>>>
>>>
>>>> 
line 604, in __spawn_instance
>>>> raise RuntimeError('Configuration of CA failed')
>>> 
>>>> 2014-07-27T06:33:25Z DEBUG The ipa-replica-install command
>>>> failed, exception: RuntimeError: Configuration of CA failed
>>> 
>>> 
>>>> So some of the required certificates must be missing still.
>>> 
>>>> Unhelpfully, the ipa-server-install --uninstall process is
>>>> not cleaning up everything after this failure, it leaves the
>>>> CA intact and the next run through the installer believes the
>>>> CA is working so it does not configure it. As such, I guess a
>>>> re-install is necessary or some other steps to truly clean
>>>> everything that I haven't found yet.
>>> 
>>>> -Erinn
>>> 
>>> Continuing on, in order to remove the CA I am manually
>>> running: pkidestroy -s CA -i pki-tomcat
>>> 
>>> And indeed there is a bug:
>>> https://fedorahosted.org/freeipa/ticket/2796
>>> 
>>> Interesting that the installer detects that the CA is
>>> installed, but the uninstaller does not detect it. I guess they
>>> are doing their detection in different ways.
>> 
>> The uninstaller doesn't rely on detection. There is a stored log
>> of what needs to be done. Unfortunately in this case the fact
>> that the CA was configured was added AFTER it was successfully
>> installed and not when we started, so if installation fails it
>> can leave things half-installed but not recorded.
>> 
>>> At this point I wanted to explore how feasible it would be to
>>> have a RHEL 7 replica without the CA replica portion, this
>>> ought to alleviate the KDC issues I seem to be having on the
>>> primary, which I have still to figure out.
>>> 
>>> So any reason not to do that? Would I simply be able to do a 
>>> ipa-ca-install on the rhel 7 system at a future juncture and
>>> then perform the rest of the migration?
>> 
>> This would be a reasonable short-term stop-gap measure though if
>> you can live without a second CA. You would likely have the same
>> problem with ipa-ca-install, at least until we figure out what
>> this missing cert error means.
>> 
>> I've seen that error about missing certs before but I can't
>> recall what it means. I have the vague notion it is a little
>> misleading though, and that something else has already failed. I
>> think we'll need one of the dogtag devs to chime in. I'll poke
>> them out-of-band.
> 
> Ok, start with the debug log on the clone ( 
> /var/log/pki/pki-tomcat/ca/debug ). It should tell you which cert
> is missing or unreadable.
> 
> How did you re-create the PKCS#12 file on the RHEL-6 server? You
> used PKCS12Export, right?
> 
> rob
> 

Correct, I just did the steps as if I was changing the dir manager
password, to re-export the certificates.

To my untrained eye it looks like the server-cert that is failing, but
here are what I believe the pertinent bits from the debug log:

[27/Jul/2014:20:46:24][http-bio-8443-exec-3]: updateNumberRange:
Failed to contact master using admin
portorg.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 50;
White spaces are required between publicId and systemId.
[27/Jul/2014:20:46:24][http-bio-8443-exec-3]: updateNumberRange:
Attempting to contact master using EE port
[27/Jul/2014:20:46:25][http-bio-8443-exec-3]: content from ee
interface =<?xml version="1.0" encoding="UTF-8"
standalone="no"?><XMLResponse><Status>0</Status><beginNumber>66</beginNumber><endNumber>70</endNumber></XMLResponse>
[27/Jul/2014:20:46:25][http-bio-8443-exec-3]: updateNumberRange():
status=0
[27/Jul/2014:20:46:25][http-bio-8443-exec-3]: updateConfigEntries start
[27/Jul/2014:20:46:26][http-bio-8443-exec-3]: updateConfigEntries:
status=0
[27/Jul/2014:20:46:26][http-bio-8443-exec-3]: deleteExistingCerts:
Exception=org.mozilla.jss.crypto.ObjectNotFoundException
[27/Jul/2014:20:46:26][http-bio-8443-exec-3]: deleteExistingCerts:
Exception=org.mozilla.jss.crypto.NoSuchItemOnTokenException
[27/Jul/2014:20:46:26][http-bio-8443-exec-3]: deleteExistingCerts:
Exception=org.mozilla.jss.crypto.ObjectNotFoundException
[27/Jul/2014:20:46:26][http-bio-8443-exec-3]: deleteExistingCerts:
Exception=org.mozilla.jss.crypto.NoSuchItemOnTokenException
[27/Jul/2014:20:46:26][http-bio-8443-exec-3]: deleteExistingCerts:
Exception=org.mozilla.jss.crypto.ObjectNotFoundException
[27/Jul/2014:20:46:26][http-bio-8443-exec-3]: deleteExistingCerts:
Exception=org.mozilla.jss.crypto.NoSuchItemOnTokenException
[27/Jul/2014:20:46:26][http-bio-8443-exec-3]: deleteExistingCerts:
Exception=org.mozilla.jss.crypto.ObjectNotFoundException
[27/Jul/2014:20:46:26][http-bio-8443-exec-3]: deleteExistingCerts:
Exception=org.mozilla.jss.crypto.NoSuchItemOnTokenException
[27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm 'RSA'
[27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm 'RSA'
[27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm 'RSA'
[27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm 'RSA'
[27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm 'RSA'
[27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm 'RSA'
[27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm 'RSA'
[27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm 'RSA'
[27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Ignoring key
CN=ipa.example.com,O=EXAMPLE.COM
[27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm 'RSA'
[27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Not importing
Server-Cert cert-pki-ca
[27/Jul/2014:20:46:26][http-bio-8443-exec-3]: isCertdbCloned:
caSigningCert cert-pki-ca
[27/Jul/2014:20:46:26][http-bio-8443-exec-3]: clone does not have all
the certificates.

Interestingly, when I do:

certutil -L -d /etc/pki/pki-tomcat/alias/ -n "Server-Cert cert-pki-ca"

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 0 (0x0)
        Signature Algorithm: PKCS #1 SHA-1 With RSA Encryption
        Issuer: "CN=ipa2.example.com,O=2014-07-27 20:46:11"
        Validity:
            Not Before: Sun Jul 27 20:46:13 2014
            Not After : Mon Jul 27 20:46:13 2015
        Subject: "CN=ipa2.example.com,O=2014-07-27 20:46:11"
        Subject Public Key Info:
            Public Key Algorithm: PKCS #1 RSA Encryption
            RSA Public Key:
                Modulus:
                    ac:f1:74:8b:d0:fa:83:5a:e9:58:fa:b4:61:dc:d2:0f:
                    66:4e:9f:61:ef:dc:47:0e:40:f8:89:45:7a:9c:1a:bf:
                    87:a3:a3:b3:06:ab:98:f7:3f:58:a4:4e:78:fe:c5:b5:
                    01:33:35:f6:0b:a2:7a:be:40:a2:76:69:61:4a:6f:1e:
                    c5:3f:c4:35:3c:dd:b0:14:c8:cd:37:e2:f6:c7:9f:53:
                    56:83:c6:74:dc:b8:f8:f5:dc:35:3f:e3:e7:f5:74:8f:
                    69:75:56:0b:cb:6e:04:3c:4a:16:67:92:63:14:92:4e:
                    ec:86:77:73:86:81:fe:01:04:2b:c2:61:13:af:70:e7
                Exponent: 65537 (0x10001)
    Signature Algorithm: PKCS #1 SHA-1 With RSA Encryption
    Signature:
        7b:d6:22:fe:df:61:2e:30:c0:76:9f:1e:59:88:7f:14:
        e3:75:e0:7b:0f:67:07:73:ba:79:59:09:4e:86:2b:9a:
        a9:8b:c4:fd:88:c4:fb:a2:1c:d9:61:70:af:55:51:09:
        35:93:f8:4e:d4:fa:7c:a0:68:fe:5a:c0:13:af:33:6a:
        7a:b5:7e:f5:e3:5a:14:b6:53:0d:19:36:ed:e2:cb:38:
        34:55:23:6b:4f:d8:6f:aa:f1:3e:12:1e:98:71:3b:0a:
        29:53:ef:10:39:d3:9e:66:05:e9:9d:aa:1a:b0:4a:9a:
        af:f2:32:85:07:f5:d0:0f:08:04:05:8b:f9:f9:bc:43
    Fingerprint (MD5):
        85:56:1B:40:91:CB:5E:A1:2B:A0:01:68:C8:57:39:B9
    Fingerprint (SHA1):
        54:48:56:07:CC:07:3A:87:A0:6C:D2:5A:7F:2B:99:BF:89:87:27:0E

    Certificate Trust Flags:
        SSL Flags:
            Valid CA
            Trusted CA
            User
            Trusted Client CA
        Email Flags:
            Valid CA
            Trusted CA
            User
        Object Signing Flags:
            Valid CA
            Trusted CA
            User

Which would appear to be a valid certificate, but I may be chasing
down the wrong path.

- -Erinn

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCAAGBQJT1mD3AAoJEFg7BmJL2iPOzDIH/2vHkZYbQv0qPcDcGqDIemdw
AujtdTqqQtJYx3WIky2N/DBhiAn2m+fj2ZPb5jMJrGjeqQsqVBt7Dlmjh87n6qmX
3gO5fGtbrTUtE4cAwr6c2QqrXOEFtsAZPhg5VqS5mug+VD7VQ64HH4EzcMek7o2s
RokqQF9T8vCKQ0zZJ1/uhvQvrG7EmNb9NrQV/l3FsFi7VisbSf0qwfwXmsjFz5GC
Hi79QFTlTi7H1Gcq0ifOTqcZ0N+q/MHiElMMHzRY09zxPXZ3AkOUmv0Wxb0IZdpw
s/PV431YbSPXoeOBL2MtvCr737ThWLVABGl4H8Ib2RncgA+keCsCFf7gnaap7RA=
=Lx6j
-----END PGP SIGNATURE-----