[Freeipa-users] RHEL 7 Upgrade experience so far

Erinn Looney-Triggs erinn.looneytriggs at gmail.com
Mon Jul 28 18:32:09 UTC 2014


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 07/28/2014 11:07 AM, Ade Lee wrote:
> On Mon, 2014-07-28 at 08:26 -0700, Erinn Looney-Triggs wrote:
>> On 07/28/2014 08:04 AM, Ade Lee wrote:
>>> On Mon, 2014-07-28 at 07:41 -0700, Erinn Looney-Triggs wrote:
>>>> On 07/28/2014 07:17 AM, Rob Crittenden wrote:
>>>>> Rob Crittenden wrote:
>>>>>> Erinn Looney-Triggs wrote:
>>>>>>> On 07/27/2014 12:02 AM, Erinn Looney-Triggs wrote:
>>>>>>>> On 07/26/2014 07:12 PM, Erinn Looney-Triggs wrote:
>>>>>>>>> On 07/26/2014 05:25 PM, Erinn Looney-Triggs wrote:
>>>>>>>>>> Well it hasn't been all the pretty trying to
>>>>>>>>>> move from RHEL 6.5 to RHEL 7.
>>>>>>> 
>>>>>>>>>> I have two servers providing my ipa instances
>>>>>>>>>> ipa and ipa2. Given that I don't have a great
>>>>>>>>>> deal of spare capacity the plan was to remove
>>>>>>>>>> ipa2 from the replication agreement, modify DNS
>>>>>>>>>> so that only IPA was available in SRV logs (IPA
>>>>>>>>>> does not manage DNS at this point, was waiting
>>>>>>>>>> for DNSSEC). As well, I would change my sudo-ldap
>>>>>>>>>> config files to point to ipa and remove ipa2.
>>>>>>> 
>>>>>>>>>> Well that all worked well, installed RHEL 7 on
>>>>>>>>>> the system and began working through the steps in
>>>>>>>>>> the upgrade guide.
>>>>>>> 
>>>>>>>>>> First major problem was running into this bug: 
>>>>>>>>>> https://fedorahosted.org/freeipa/ticket/4375 
>>>>>>>>>> ValueError: nsDS5ReplicaId has 2 values, one 
>>>>>>>>>> expected.
>>>>>>> 
>>>>>>>>>> Went and patched the replication.py file to get 
>>>>>>>>>> around that issue, and we moved on.
>>>>>>> 
>>>>>>>>>> Next up is my current issue: Exception from Java
>>>>>>>>>>  Configuration Servlet: Clone does not have all
>>>>>>>>>> the required certificates.
>>>>>>> 
>>>>>>>>>> I suspect this is because I am running the CA as
>>>>>>>>>> a subordinate to an AD CS instance, but I am
>>>>>>>>>> unsure at this point.
>>>>>>> 
>>>>>>>>>> It has been a haul to get here, despite the short
>>>>>>>>>>  explanation. It seems that my primary ipa
>>>>>>>>>> instance is working on only a hit or miss basis
>>>>>>>>>> for kerberos tickets which has made all this a
>>>>>>>>>> bit of a pain. You can kinit as admin once it
>>>>>>>>>> will fail unable to find KDC, try again another
>>>>>>>>>> three times, it will work. I have even modified
>>>>>>>>>> the krb5.conf file to point directly at the
>>>>>>>>>> server, thus bypassing DNS SRV lookups, however,
>>>>>>>>>> that hasn't worked.
>>>>>>> 
>>>>>>>>>> Point is, any help would be appreciated on the 
>>>>>>>>>> aforementioned error.
>>>>>>> 
>>>>>>>>>> -Erinn
>>>>>>> 
>>>>>>> 
>>>>>>>>> To reply to myself here, I believe the problem may
>>>>>>>>> be that I had to renew the CA certificates and as
>>>>>>>>> such the certificates in /root/cacert.p12 are no
>>>>>>>>> longer valid. It is this file that gets bundled up
>>>>>>>>> with whatever else using ipa-replica-prepare, so I
>>>>>>>>> will have to create a new one that has the valid
>>>>>>>>> certificates in it.
>>>>>>> 
>>>>>>>>> One way or another though, if it isn't already 
>>>>>>>>> documented, during a CA renewal this file should 
>>>>>>>>> probably be updated with the correct certificates.
>>>>>>> 
>>>>>>>>> -Erinn
>>>>>>> 
>>>>>>>>> -Erinn
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> Well thanks to this: 
>>>>>>>> http://www.freeipa.org/page/Howto/Change_Directory_Manager_Password
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>
>>>>>>>>
>>
>>>>>>>> 
I have gotten a little further down the road an created a new
>>>>>>>> cacert.p12 which looks to be complete.
>>>>>>> 
>>>>>>>> However, installation still fails in the same place:
>>>>>>> 
>>>>>>>> 2014-07-27T06:33:04Z DEBUG Starting external process
>>>>>>>>  2014-07-27T06:33:04Z DEBUG args=/usr/sbin/pkispawn
>>>>>>>> -s CA -f /tmp/tmp5QGhUx 2014-07-27T06:33:25Z DEBUG
>>>>>>>> Process finished, return code=1 2014-07-27T06:33:25Z
>>>>>>>> DEBUG stdout=Loading deployment configuration from 
>>>>>>>> /tmp/tmp5QGhUx. Installing CA into 
>>>>>>>> /var/lib/pki/pki-tomcat. Storing deployment
>>>>>>>> configuration into 
>>>>>>>> /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg.
>>>>>>>>  Installation failed.
>>>>>>> 
>>>>>>> 
>>>>>>>> 2014-07-27T06:33:25Z DEBUG stderr=pkispawn    :
>>>>>>>> WARNING ....... unable to validate security domain
>>>>>>>> user/password through REST interface. Interface not
>>>>>>>> available pkispawn : ERROR    ....... Exception from
>>>>>>>> Java Configuration Servlet: Clone does not have all
>>>>>>>> the required certificates
>>>>>>> 
>>>>>>>> 2014-07-27T06:33:25Z CRITICAL failed to configure ca
>>>>>>>>  instance Command '/usr/sbin/pkispawn -s CA -f 
>>>>>>>> /tmp/tmp5QGhUx' returned non-zero exit status 1 
>>>>>>>> 2014-07-27T06:33:25Z DEBUG File 
>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py",
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>
>>>>>>>>
>>
>>>>>>>> 
line 638, in run_script
>>>>>>>> return_value = main_function()
>>>>>>> 
>>>>>>>> File "/usr/sbin/ipa-replica-install", line 667, in
>>>>>>>> main CA = cainstance.install_replica_ca(config)
>>>>>>> 
>>>>>>>> File 
>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>
>>>>>>>>
>>
>>>>>>>> 
line 1678, in install_replica_ca
>>>>>>>> subject_base=config.subject_base)
>>>>>>> 
>>>>>>>> File 
>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>
>>>>>>>>
>>
>>>>>>>> 
line 478, in configure_instance
>>>>>>>> self.start_creation(runtime=210)
>>>>>>> 
>>>>>>>> File 
>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
>>>>>>>>
>>>>>>>>
>>>>
>>>>>>>>
>>
>>>>>>>> 
line 364, in start_creation method()
>>>>>>> 
>>>>>>>> File 
>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>
>>>>>>>>
>>
>>>>>>>> 
line 604, in __spawn_instance
>>>>>>>> raise RuntimeError('Configuration of CA failed')
>>>>>>> 
>>>>>>>> 2014-07-27T06:33:25Z DEBUG The ipa-replica-install 
>>>>>>>> command failed, exception: RuntimeError:
>>>>>>>> Configuration of CA failed
>>>>>>> 
>>>>>>> 
>>>>>>>> So some of the required certificates must be missing 
>>>>>>>> still.
>>>>>>> 
>>>>>>>> Unhelpfully, the ipa-server-install --uninstall
>>>>>>>> process is not cleaning up everything after this
>>>>>>>> failure, it leaves the CA intact and the next run
>>>>>>>> through the installer believes the CA is working so
>>>>>>>> it does not configure it. As such, I guess a
>>>>>>>> re-install is necessary or some other steps to truly
>>>>>>>> clean everything that I haven't found yet.
>>>>>>> 
>>>>>>>> -Erinn
>>>>>>> 
>>>>>>> Continuing on, in order to remove the CA I am manually
>>>>>>>  running: pkidestroy -s CA -i pki-tomcat
>>>>>>> 
>>>>>>> And indeed there is a bug: 
>>>>>>> https://fedorahosted.org/freeipa/ticket/2796
>>>>>>> 
>>>>>>> Interesting that the installer detects that the CA is 
>>>>>>> installed, but the uninstaller does not detect it. I
>>>>>>> guess they are doing their detection in different
>>>>>>> ways.
>>>>>> 
>>>>>> The uninstaller doesn't rely on detection. There is a
>>>>>> stored log of what needs to be done. Unfortunately in
>>>>>> this case the fact that the CA was configured was added
>>>>>> AFTER it was successfully installed and not when we
>>>>>> started, so if installation fails it can leave things
>>>>>> half-installed but not recorded.
>>>>>> 
>>>>>>> At this point I wanted to explore how feasible it would
>>>>>>> be to have a RHEL 7 replica without the CA replica
>>>>>>> portion, this ought to alleviate the KDC issues I seem
>>>>>>> to be having on the primary, which I have still to
>>>>>>> figure out.
>>>>>>> 
>>>>>>> So any reason not to do that? Would I simply be able to
>>>>>>> do a ipa-ca-install on the rhel 7 system at a future
>>>>>>> juncture and then perform the rest of the migration?
>>>>>> 
>>>>>> This would be a reasonable short-term stop-gap measure
>>>>>> though if you can live without a second CA. You would
>>>>>> likely have the same problem with ipa-ca-install, at
>>>>>> least until we figure out what this missing cert error
>>>>>> means.
>>>>>> 
>>>>>> I've seen that error about missing certs before but I
>>>>>> can't recall what it means. I have the vague notion it is
>>>>>> a little misleading though, and that something else has
>>>>>> already failed. I think we'll need one of the dogtag devs
>>>>>> to chime in. I'll poke them out-of-band.
>>>>> 
>>>>> Ok, start with the debug log on the clone ( 
>>>>> /var/log/pki/pki-tomcat/ca/debug ). It should tell you
>>>>> which cert is missing or unreadable.
>>>>> 
>>>>> How did you re-create the PKCS#12 file on the RHEL-6
>>>>> server? You used PKCS12Export, right?
>>>>> 
>>>>> rob
>>>>> 
>>>> 
>>>> Correct, I just did the steps as if I was changing the dir 
>>>> manager password, to re-export the certificates.
>>>> 
>>>> To my untrained eye it looks like the server-cert that is 
>>>> failing, but here are what I believe the pertinent bits from
>>>> the debug log:
>>>> 
>>>> [27/Jul/2014:20:46:24][http-bio-8443-exec-3]:
>>>> updateNumberRange: Failed to contact master using admin 
>>>> portorg.xml.sax.SAXParseException; lineNumber: 1;
>>>> columnNumber: 50; White spaces are required between publicId
>>>> and systemId. [27/Jul/2014:20:46:24][http-bio-8443-exec-3]:
>>>> updateNumberRange: Attempting to contact master using EE port
>>>>  [27/Jul/2014:20:46:25][http-bio-8443-exec-3]: content from
>>>> ee interface =<?xml version="1.0" encoding="UTF-8" 
>>>> standalone="no"?><XMLResponse><Status>0</Status><beginNumber>66</beginNumber><endNumber>70</endNumber></XMLResponse>
>>>>
>>>>
>>
>>>> 
[27/Jul/2014:20:46:25][http-bio-8443-exec-3]: updateNumberRange():
>>>> status=0 [27/Jul/2014:20:46:25][http-bio-8443-exec-3]: 
>>>> updateConfigEntries start 
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: 
>>>> updateConfigEntries: status=0 
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: 
>>>> deleteExistingCerts: 
>>>> Exception=org.mozilla.jss.crypto.ObjectNotFoundException 
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: 
>>>> deleteExistingCerts: 
>>>> Exception=org.mozilla.jss.crypto.NoSuchItemOnTokenException 
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: 
>>>> deleteExistingCerts: 
>>>> Exception=org.mozilla.jss.crypto.ObjectNotFoundException 
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: 
>>>> deleteExistingCerts: 
>>>> Exception=org.mozilla.jss.crypto.NoSuchItemOnTokenException 
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: 
>>>> deleteExistingCerts: 
>>>> Exception=org.mozilla.jss.crypto.ObjectNotFoundException 
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: 
>>>> deleteExistingCerts: 
>>>> Exception=org.mozilla.jss.crypto.NoSuchItemOnTokenException 
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: 
>>>> deleteExistingCerts: 
>>>> Exception=org.mozilla.jss.crypto.ObjectNotFoundException 
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: 
>>>> deleteExistingCerts: 
>>>> Exception=org.mozilla.jss.crypto.NoSuchItemOnTokenException 
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm 
>>>> 'RSA' [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key
>>>> Algorithm 'RSA' [27/Jul/2014:20:46:26][http-bio-8443-exec-3]:
>>>> Key Algorithm 'RSA'
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm 
>>>> 'RSA' [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key
>>>> Algorithm 'RSA' [27/Jul/2014:20:46:26][http-bio-8443-exec-3]:
>>>> Key Algorithm 'RSA'
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm 
>>>> 'RSA' [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key
>>>> Algorithm 'RSA' [27/Jul/2014:20:46:26][http-bio-8443-exec-3]:
>>>> Ignoring key CN=ipa.example.com,O=EXAMPLE.COM 
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm 
>>>> 'RSA' [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Not 
>>>> importing Server-Cert cert-pki-ca
>>> 
>>> The above line is absolutely correct.  The server cert is not 
>>> imported into the clone.  The clone will generate its own
>>> server cert.
>>> 
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: isCertdbCloned:
>>>>  caSigningCert cert-pki-ca 
>>>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: clone does not
>>>> have all the certificates.
>>>> 
>>> This actually indicates to me that the signing cert is not
>>> present in the P12 replica file, or that some error has been
>>> encountered in reading the cert.
>>> 
>>> Also, check in the journal to see if any exceptions have been 
>>> thrown. journalctl -u pki-tomcatd at pki-tomcat.service
>>> 
>>> Ade
>>> 
>> 
>> No exceptions thrown in the journal.
>> 
>> When investigating the cacert.p12 file that is bundled up for
>> the replica's I see two caSigningCert's. One is the older one,
>> before I renewed and one is the new, valid, post renewal one. Are
>> these the certs that are used or are they requested from the
>> primary much like the servercert?
> 
> I think the problem might be that there are two caSigningCerts,
> with presumably the same nickname.  We need to try and generate a
> p12 file without the old pre-renewal signing cert.
> 
> Does the master certdb contain both certs with the same nickname? 
> If so, you could try to remove the old signing cert from the
> master certdb and then regenerate the pkcs12 using PKCS12Export.
> 
> Here is one way you could try to do this: 1. Export the caSigning
> cert from the certdb using pk12util.  Check that only the latest
> cert/key has been exported.  Make sure to note down the exact cert
> nickname.  If so, then continue .. 2. Shut down the CA. 3.
> IMPORTANT: Back up the certdb. 4. Delete the caSigning cert from
> the certdb using certutil.  You may have to do this twice to remove
> both certs. 5. Re-import the saved caSigningCert using pk12util. 6.
> Restart the CA.
> 
>> 
>> However, when examining the /etc/pki/pki-tomcat/alias db there is
>> no casigningcert, hence I suppose the failure. So somewhere in
>> there it is getting lost, I'll keep looking but throw me any
>> ideas.
>> 
> By this, you are implying looking at the clone certdb, right?  The
> cert should certainly still exist on the master.  The cert not
> being in the clone certdb is because it fails to be imported from
> the PKCS12 file.
> 
> 
>> -Erinn
>> 
>> 
>> 
> 
> 

Using the following:
pk12util -o /root/casigningcert.p12 -d /var/lib/pki-ca/alias/ -n
'caSigningCert cert-pki-ca'

I end up with both the old and the new certs, as well as two copies of
the root CA.

Any other ways to change around the DB? I don't know NSS tools all
that well.

- -Erinn
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCAAGBQJT1pcjAAoJEFg7BmJL2iPOrE0IAIzfjjmmK+lmMfA7q869sAEq
hF30gceUy+XRKFZZxZsgg8HVnmr3UmBT1f3JY19ADkW4HQ5PkWr7/zDGIkoSTfvT
WTn9wnYtqj/zg5ZQH6wOtlSoyu1hBJXKgwJQn0RNiGdeHaVYh6mtLT+/Z4Dsgdh1
+KP9fQIlGTwLmYtrEHSbRfjQ3bH3slIGzcuNjiDmH37kpkEwVkhfKh0+QOvdko4p
HSX1ulNRbKEDgc08BXHUpwtuJ+R/bs96DM0jF9/PjgdDgI2J+Q3j6BJoLMqNCqNB
P6OwJlY6Guq3PotVBeSSrZG4CzHYHi/xweW9mN/vAyIBSJk6QwkswtKF8EKLW+I=
=eltq
-----END PGP SIGNATURE-----




More information about the Freeipa-users mailing list