[Freeipa-users] RHEL 7 Upgrade experience so far
Erinn Looney-Triggs
erinn.looneytriggs at gmail.com
Mon Jul 28 18:32:09 UTC 2014
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
On 07/28/2014 11:07 AM, Ade Lee wrote:
> On Mon, 2014-07-28 at 08:26 -0700, Erinn Looney-Triggs wrote:
>> On 07/28/2014 08:04 AM, Ade Lee wrote:
>>> On Mon, 2014-07-28 at 07:41 -0700, Erinn Looney-Triggs wrote:
>>>> On 07/28/2014 07:17 AM, Rob Crittenden wrote:
>>>>> Rob Crittenden wrote:
>>>>>> Erinn Looney-Triggs wrote:
>>>>>>> On 07/27/2014 12:02 AM, Erinn Looney-Triggs wrote:
>>>>>>>> On 07/26/2014 07:12 PM, Erinn Looney-Triggs wrote:
>>>>>>>>> On 07/26/2014 05:25 PM, Erinn Looney-Triggs wrote:
>>>>>>>>>> Well it hasn't been all the pretty trying to
>>>>>>>>>> move from RHEL 6.5 to RHEL 7.
>>>>>>>
>>>>>>>>>> I have two servers providing my ipa instances
>>>>>>>>>> ipa and ipa2. Given that I don't have a great
>>>>>>>>>> deal of spare capacity the plan was to remove
>>>>>>>>>> ipa2 from the replication agreement, modify DNS
>>>>>>>>>> so that only IPA was available in SRV logs (IPA
>>>>>>>>>> does not manage DNS at this point, was waiting
>>>>>>>>>> for DNSSEC). As well, I would change my sudo-ldap
>>>>>>>>>> config files to point to ipa and remove ipa2.
>>>>>>>
>>>>>>>>>> Well that all worked well, installed RHEL 7 on
>>>>>>>>>> the system and began working through the steps in
>>>>>>>>>> the upgrade guide.
>>>>>>>
>>>>>>>>>> First major problem was running into this bug:
>>>>>>>>>> https://fedorahosted.org/freeipa/ticket/4375
>>>>>>>>>> ValueError: nsDS5ReplicaId has 2 values, one
>>>>>>>>>> expected.
>>>>>>>
>>>>>>>>>> Went and patched the replication.py file to get
>>>>>>>>>> around that issue, and we moved on.
>>>>>>>
>>>>>>>>>> Next up is my current issue: Exception from Java
>>>>>>>>>> Configuration Servlet: Clone does not have all
>>>>>>>>>> the required certificates.
>>>>>>>
>>>>>>>>>> I suspect this is because I am running the CA as
>>>>>>>>>> a subordinate to an AD CS instance, but I am
>>>>>>>>>> unsure at this point.
>>>>>>>
>>>>>>>>>> It has been a haul to get here, despite the short
>>>>>>>>>> explanation. It seems that my primary ipa
>>>>>>>>>> instance is working on only a hit or miss basis
>>>>>>>>>> for kerberos tickets which has made all this a
>>>>>>>>>> bit of a pain. You can kinit as admin once it
>>>>>>>>>> will fail unable to find KDC, try again another
>>>>>>>>>> three times, it will work. I have even modified
>>>>>>>>>> the krb5.conf file to point directly at the
>>>>>>>>>> server, thus bypassing DNS SRV lookups, however,
>>>>>>>>>> that hasn't worked.
>>>>>>>
>>>>>>>>>> Point is, any help would be appreciated on the
>>>>>>>>>> aforementioned error.
>>>>>>>
>>>>>>>>>> -Erinn
>>>>>>>
>>>>>>>
>>>>>>>>> To reply to myself here, I believe the problem may
>>>>>>>>> be that I had to renew the CA certificates and as
>>>>>>>>> such the certificates in /root/cacert.p12 are no
>>>>>>>>> longer valid. It is this file that gets bundled up
>>>>>>>>> with whatever else using ipa-replica-prepare, so I
>>>>>>>>> will have to create a new one that has the valid
>>>>>>>>> certificates in it.
>>>>>>>
>>>>>>>>> One way or another though, if it isn't already
>>>>>>>>> documented, during a CA renewal this file should
>>>>>>>>> probably be updated with the correct certificates.
>>>>>>>
>>>>>>>>> -Erinn
>>>>>>>
>>>>>>>>> -Erinn
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Well thanks to this:
>>>>>>>> http://www.freeipa.org/page/Howto/Change_Directory_Manager_Password
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>
>>>>>>>>
>>
>>>>>>>>
I have gotten a little further down the road an created a new
>>>>>>>> cacert.p12 which looks to be complete.
>>>>>>>
>>>>>>>> However, installation still fails in the same place:
>>>>>>>
>>>>>>>> 2014-07-27T06:33:04Z DEBUG Starting external process
>>>>>>>> 2014-07-27T06:33:04Z DEBUG args=/usr/sbin/pkispawn
>>>>>>>> -s CA -f /tmp/tmp5QGhUx 2014-07-27T06:33:25Z DEBUG
>>>>>>>> Process finished, return code=1 2014-07-27T06:33:25Z
>>>>>>>> DEBUG stdout=Loading deployment configuration from
>>>>>>>> /tmp/tmp5QGhUx. Installing CA into
>>>>>>>> /var/lib/pki/pki-tomcat. Storing deployment
>>>>>>>> configuration into
>>>>>>>> /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg.
>>>>>>>> Installation failed.
>>>>>>>
>>>>>>>
>>>>>>>> 2014-07-27T06:33:25Z DEBUG stderr=pkispawn :
>>>>>>>> WARNING ....... unable to validate security domain
>>>>>>>> user/password through REST interface. Interface not
>>>>>>>> available pkispawn : ERROR ....... Exception from
>>>>>>>> Java Configuration Servlet: Clone does not have all
>>>>>>>> the required certificates
>>>>>>>
>>>>>>>> 2014-07-27T06:33:25Z CRITICAL failed to configure ca
>>>>>>>> instance Command '/usr/sbin/pkispawn -s CA -f
>>>>>>>> /tmp/tmp5QGhUx' returned non-zero exit status 1
>>>>>>>> 2014-07-27T06:33:25Z DEBUG File
>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py",
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>
>>>>>>>>
>>
>>>>>>>>
line 638, in run_script
>>>>>>>> return_value = main_function()
>>>>>>>
>>>>>>>> File "/usr/sbin/ipa-replica-install", line 667, in
>>>>>>>> main CA = cainstance.install_replica_ca(config)
>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>
>>>>>>>>
>>
>>>>>>>>
line 1678, in install_replica_ca
>>>>>>>> subject_base=config.subject_base)
>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>
>>>>>>>>
>>
>>>>>>>>
line 478, in configure_instance
>>>>>>>> self.start_creation(runtime=210)
>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
>>>>>>>>
>>>>>>>>
>>>>
>>>>>>>>
>>
>>>>>>>>
line 364, in start_creation method()
>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>
>>>>>>>>
>>
>>>>>>>>
line 604, in __spawn_instance
>>>>>>>> raise RuntimeError('Configuration of CA failed')
>>>>>>>
>>>>>>>> 2014-07-27T06:33:25Z DEBUG The ipa-replica-install
>>>>>>>> command failed, exception: RuntimeError:
>>>>>>>> Configuration of CA failed
>>>>>>>
>>>>>>>
>>>>>>>> So some of the required certificates must be missing
>>>>>>>> still.
>>>>>>>
>>>>>>>> Unhelpfully, the ipa-server-install --uninstall
>>>>>>>> process is not cleaning up everything after this
>>>>>>>> failure, it leaves the CA intact and the next run
>>>>>>>> through the installer believes the CA is working so
>>>>>>>> it does not configure it. As such, I guess a
>>>>>>>> re-install is necessary or some other steps to truly
>>>>>>>> clean everything that I haven't found yet.
>>>>>>>
>>>>>>>> -Erinn
>>>>>>>
>>>>>>> Continuing on, in order to remove the CA I am manually
>>>>>>> running: pkidestroy -s CA -i pki-tomcat
>>>>>>>
>>>>>>> And indeed there is a bug:
>>>>>>> https://fedorahosted.org/freeipa/ticket/2796
>>>>>>>
>>>>>>> Interesting that the installer detects that the CA is
>>>>>>> installed, but the uninstaller does not detect it. I
>>>>>>> guess they are doing their detection in different
>>>>>>> ways.
>>>>>>
>>>>>> The uninstaller doesn't rely on detection. There is a
>>>>>> stored log of what needs to be done. Unfortunately in
>>>>>> this case the fact that the CA was configured was added
>>>>>> AFTER it was successfully installed and not when we
>>>>>> started, so if installation fails it can leave things
>>>>>> half-installed but not recorded.
>>>>>>
>>>>>>> At this point I wanted to explore how feasible it would
>>>>>>> be to have a RHEL 7 replica without the CA replica
>>>>>>> portion, this ought to alleviate the KDC issues I seem
>>>>>>> to be having on the primary, which I have still to
>>>>>>> figure out.
>>>>>>>
>>>>>>> So any reason not to do that? Would I simply be able to
>>>>>>> do a ipa-ca-install on the rhel 7 system at a future
>>>>>>> juncture and then perform the rest of the migration?
>>>>>>
>>>>>> This would be a reasonable short-term stop-gap measure
>>>>>> though if you can live without a second CA. You would
>>>>>> likely have the same problem with ipa-ca-install, at
>>>>>> least until we figure out what this missing cert error
>>>>>> means.
>>>>>>
>>>>>> I've seen that error about missing certs before but I
>>>>>> can't recall what it means. I have the vague notion it is
>>>>>> a little misleading though, and that something else has
>>>>>> already failed. I think we'll need one of the dogtag devs
>>>>>> to chime in. I'll poke them out-of-band.
>>>>>
>>>>> Ok, start with the debug log on the clone (
>>>>> /var/log/pki/pki-tomcat/ca/debug ). It should tell you
>>>>> which cert is missing or unreadable.
>>>>>
>>>>> How did you re-create the PKCS#12 file on the RHEL-6
>>>>> server? You used PKCS12Export, right?
>>>>>
>>>>> rob
>>>>>
>>>>
>>>> Correct, I just did the steps as if I was changing the dir
>>>> manager password, to re-export the certificates.
>>>>
>>>> To my untrained eye it looks like the server-cert that is
>>>> failing, but here are what I believe the pertinent bits from
>>>> the debug log:
>>>>
>>>> [27/Jul/2014:20:46:24][http-bio-8443-exec-3]:
>>>> updateNumberRange: Failed to contact master using admin
>>>> portorg.xml.sax.SAXParseException; lineNumber: 1;
>>>> columnNumber: 50; White spaces are required between publicId
>>>> and systemId. [27/Jul/2014:20:46:24][http-bio-8443-exec-3]:
>>>> updateNumberRange: Attempting to contact master using EE port
>>>> [27/Jul/2014:20:46:25][http-bio-8443-exec-3]: content from
>>>> ee interface =<?xml version="1.0" encoding="UTF-8"
>>>> standalone="no"?><XMLResponse><Status>0</Status><beginNumber>66</beginNumber><endNumber>70</endNumber></XMLResponse>
>>>>
>>>>
>>
>>>>
[27/Jul/2014:20:46:25][http-bio-8443-exec-3]: updateNumberRange():
>>>> status=0 [27/Jul/2014:20:46:25][http-bio-8443-exec-3]:
>>>> updateConfigEntries start
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]:
>>>> updateConfigEntries: status=0
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]:
>>>> deleteExistingCerts:
>>>> Exception=org.mozilla.jss.crypto.ObjectNotFoundException
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]:
>>>> deleteExistingCerts:
>>>> Exception=org.mozilla.jss.crypto.NoSuchItemOnTokenException
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]:
>>>> deleteExistingCerts:
>>>> Exception=org.mozilla.jss.crypto.ObjectNotFoundException
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]:
>>>> deleteExistingCerts:
>>>> Exception=org.mozilla.jss.crypto.NoSuchItemOnTokenException
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]:
>>>> deleteExistingCerts:
>>>> Exception=org.mozilla.jss.crypto.ObjectNotFoundException
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]:
>>>> deleteExistingCerts:
>>>> Exception=org.mozilla.jss.crypto.NoSuchItemOnTokenException
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]:
>>>> deleteExistingCerts:
>>>> Exception=org.mozilla.jss.crypto.ObjectNotFoundException
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]:
>>>> deleteExistingCerts:
>>>> Exception=org.mozilla.jss.crypto.NoSuchItemOnTokenException
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm
>>>> 'RSA' [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key
>>>> Algorithm 'RSA' [27/Jul/2014:20:46:26][http-bio-8443-exec-3]:
>>>> Key Algorithm 'RSA'
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm
>>>> 'RSA' [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key
>>>> Algorithm 'RSA' [27/Jul/2014:20:46:26][http-bio-8443-exec-3]:
>>>> Key Algorithm 'RSA'
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm
>>>> 'RSA' [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key
>>>> Algorithm 'RSA' [27/Jul/2014:20:46:26][http-bio-8443-exec-3]:
>>>> Ignoring key CN=ipa.example.com,O=EXAMPLE.COM
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm
>>>> 'RSA' [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Not
>>>> importing Server-Cert cert-pki-ca
>>>
>>> The above line is absolutely correct. The server cert is not
>>> imported into the clone. The clone will generate its own
>>> server cert.
>>>
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: isCertdbCloned:
>>>> caSigningCert cert-pki-ca
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: clone does not
>>>> have all the certificates.
>>>>
>>> This actually indicates to me that the signing cert is not
>>> present in the P12 replica file, or that some error has been
>>> encountered in reading the cert.
>>>
>>> Also, check in the journal to see if any exceptions have been
>>> thrown. journalctl -u pki-tomcatd at pki-tomcat.service
>>>
>>> Ade
>>>
>>
>> No exceptions thrown in the journal.
>>
>> When investigating the cacert.p12 file that is bundled up for
>> the replica's I see two caSigningCert's. One is the older one,
>> before I renewed and one is the new, valid, post renewal one. Are
>> these the certs that are used or are they requested from the
>> primary much like the servercert?
>
> I think the problem might be that there are two caSigningCerts,
> with presumably the same nickname. We need to try and generate a
> p12 file without the old pre-renewal signing cert.
>
> Does the master certdb contain both certs with the same nickname?
> If so, you could try to remove the old signing cert from the
> master certdb and then regenerate the pkcs12 using PKCS12Export.
>
> Here is one way you could try to do this: 1. Export the caSigning
> cert from the certdb using pk12util. Check that only the latest
> cert/key has been exported. Make sure to note down the exact cert
> nickname. If so, then continue .. 2. Shut down the CA. 3.
> IMPORTANT: Back up the certdb. 4. Delete the caSigning cert from
> the certdb using certutil. You may have to do this twice to remove
> both certs. 5. Re-import the saved caSigningCert using pk12util. 6.
> Restart the CA.
>
>>
>> However, when examining the /etc/pki/pki-tomcat/alias db there is
>> no casigningcert, hence I suppose the failure. So somewhere in
>> there it is getting lost, I'll keep looking but throw me any
>> ideas.
>>
> By this, you are implying looking at the clone certdb, right? The
> cert should certainly still exist on the master. The cert not
> being in the clone certdb is because it fails to be imported from
> the PKCS12 file.
>
>
>> -Erinn
>>
>>
>>
>
>
Using the following:
pk12util -o /root/casigningcert.p12 -d /var/lib/pki-ca/alias/ -n
'caSigningCert cert-pki-ca'
I end up with both the old and the new certs, as well as two copies of
the root CA.
Any other ways to change around the DB? I don't know NSS tools all
that well.
- -Erinn
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBCAAGBQJT1pcjAAoJEFg7BmJL2iPOrE0IAIzfjjmmK+lmMfA7q869sAEq
hF30gceUy+XRKFZZxZsgg8HVnmr3UmBT1f3JY19ADkW4HQ5PkWr7/zDGIkoSTfvT
WTn9wnYtqj/zg5ZQH6wOtlSoyu1hBJXKgwJQn0RNiGdeHaVYh6mtLT+/Z4Dsgdh1
+KP9fQIlGTwLmYtrEHSbRfjQ3bH3slIGzcuNjiDmH37kpkEwVkhfKh0+QOvdko4p
HSX1ulNRbKEDgc08BXHUpwtuJ+R/bs96DM0jF9/PjgdDgI2J+Q3j6BJoLMqNCqNB
P6OwJlY6Guq3PotVBeSSrZG4CzHYHi/xweW9mN/vAyIBSJk6QwkswtKF8EKLW+I=
=eltq
-----END PGP SIGNATURE-----
More information about the Freeipa-users
mailing list