[Freeipa-users] RHEL 7 Upgrade experience so far

Erinn Looney-Triggs erinn.looneytriggs at gmail.com
Mon Jul 28 15:26:56 UTC 2014 

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 07/28/2014 08:04 AM, Ade Lee wrote:
> On Mon, 2014-07-28 at 07:41 -0700, Erinn Looney-Triggs wrote:
>> On 07/28/2014 07:17 AM, Rob Crittenden wrote:
>>> Rob Crittenden wrote:
>>>> Erinn Looney-Triggs wrote:
>>>>> On 07/27/2014 12:02 AM, Erinn Looney-Triggs wrote:
>>>>>> On 07/26/2014 07:12 PM, Erinn Looney-Triggs wrote:
>>>>>>> On 07/26/2014 05:25 PM, Erinn Looney-Triggs wrote:
>>>>>>>> Well it hasn't been all the pretty trying to move
>>>>>>>> from RHEL 6.5 to RHEL 7.
>>>>> 
>>>>>>>> I have two servers providing my ipa instances ipa
>>>>>>>> and ipa2. Given that I don't have a great deal of
>>>>>>>> spare capacity the plan was to remove ipa2 from the
>>>>>>>> replication agreement, modify DNS so that only IPA
>>>>>>>> was available in SRV logs (IPA does not manage DNS at
>>>>>>>> this point, was waiting for DNSSEC). As well, I would
>>>>>>>> change my sudo-ldap config files to point to ipa and
>>>>>>>> remove ipa2.
>>>>> 
>>>>>>>> Well that all worked well, installed RHEL 7 on the
>>>>>>>> system and began working through the steps in the
>>>>>>>> upgrade guide.
>>>>> 
>>>>>>>> First major problem was running into this bug: 
>>>>>>>> https://fedorahosted.org/freeipa/ticket/4375
>>>>>>>> ValueError: nsDS5ReplicaId has 2 values, one
>>>>>>>> expected.
>>>>> 
>>>>>>>> Went and patched the replication.py file to get
>>>>>>>> around that issue, and we moved on.
>>>>> 
>>>>>>>> Next up is my current issue: Exception from Java 
>>>>>>>> Configuration Servlet: Clone does not have all the 
>>>>>>>> required certificates.
>>>>> 
>>>>>>>> I suspect this is because I am running the CA as a 
>>>>>>>> subordinate to an AD CS instance, but I am unsure at
>>>>>>>> this point.
>>>>> 
>>>>>>>> It has been a haul to get here, despite the short 
>>>>>>>> explanation. It seems that my primary ipa instance
>>>>>>>> is working on only a hit or miss basis for kerberos
>>>>>>>> tickets which has made all this a bit of a pain. You
>>>>>>>> can kinit as admin once it will fail unable to find
>>>>>>>> KDC, try again another three times, it will work. I
>>>>>>>> have even modified the krb5.conf file to point
>>>>>>>> directly at the server, thus bypassing DNS SRV
>>>>>>>> lookups, however, that hasn't worked.
>>>>> 
>>>>>>>> Point is, any help would be appreciated on the 
>>>>>>>> aforementioned error.
>>>>> 
>>>>>>>> -Erinn
>>>>> 
>>>>> 
>>>>>>> To reply to myself here, I believe the problem may be
>>>>>>> that I had to renew the CA certificates and as such
>>>>>>> the certificates in /root/cacert.p12 are no longer
>>>>>>> valid. It is this file that gets bundled up with
>>>>>>> whatever else using ipa-replica-prepare, so I will have
>>>>>>> to create a new one that has the valid certificates in
>>>>>>> it.
>>>>> 
>>>>>>> One way or another though, if it isn't already
>>>>>>> documented, during a CA renewal this file should
>>>>>>> probably be updated with the correct certificates.
>>>>> 
>>>>>>> -Erinn
>>>>> 
>>>>>>> -Erinn
>>>>> 
>>>>> 
>>>>> 
>>>>>> Well thanks to this: 
>>>>>> http://www.freeipa.org/page/Howto/Change_Directory_Manager_Password
>>>>>
>>>>>>
>>>>>>
>>
>>>>>> 
I have gotten a little further down the road an created a new
>>>>>> cacert.p12 which looks to be complete.
>>>>> 
>>>>>> However, installation still fails in the same place:
>>>>> 
>>>>>> 2014-07-27T06:33:04Z DEBUG Starting external process 
>>>>>> 2014-07-27T06:33:04Z DEBUG args=/usr/sbin/pkispawn -s CA
>>>>>> -f /tmp/tmp5QGhUx 2014-07-27T06:33:25Z DEBUG Process
>>>>>> finished, return code=1 2014-07-27T06:33:25Z DEBUG
>>>>>> stdout=Loading deployment configuration from
>>>>>> /tmp/tmp5QGhUx. Installing CA into
>>>>>> /var/lib/pki/pki-tomcat. Storing deployment configuration
>>>>>> into 
>>>>>> /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg. 
>>>>>> Installation failed.
>>>>> 
>>>>> 
>>>>>> 2014-07-27T06:33:25Z DEBUG stderr=pkispawn    : WARNING 
>>>>>> ....... unable to validate security domain user/password 
>>>>>> through REST interface. Interface not available pkispawn
>>>>>> : ERROR    ....... Exception from Java Configuration
>>>>>> Servlet: Clone does not have all the required
>>>>>> certificates
>>>>> 
>>>>>> 2014-07-27T06:33:25Z CRITICAL failed to configure ca 
>>>>>> instance Command '/usr/sbin/pkispawn -s CA -f
>>>>>> /tmp/tmp5QGhUx' returned non-zero exit status 1
>>>>>> 2014-07-27T06:33:25Z DEBUG File 
>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py",
>>>>>
>>>>>
>>>>>
>>>>>>
>>
>>>>>> 
line 638, in run_script
>>>>>> return_value = main_function()
>>>>> 
>>>>>> File "/usr/sbin/ipa-replica-install", line 667, in main
>>>>>> CA = cainstance.install_replica_ca(config)
>>>>> 
>>>>>> File 
>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
>>>>>
>>>>>
>>>>>
>>>>>>
>>
>>>>>> 
line 1678, in install_replica_ca
>>>>>> subject_base=config.subject_base)
>>>>> 
>>>>>> File 
>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
>>>>>
>>>>>
>>>>>
>>>>>>
>>
>>>>>> 
line 478, in configure_instance
>>>>>> self.start_creation(runtime=210)
>>>>> 
>>>>>> File 
>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
>>>>>>
>>>>>>
>>
>>>>>> 
line 364, in start_creation method()
>>>>> 
>>>>>> File 
>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
>>>>>
>>>>>
>>>>>
>>>>>>
>>
>>>>>> 
line 604, in __spawn_instance
>>>>>> raise RuntimeError('Configuration of CA failed')
>>>>> 
>>>>>> 2014-07-27T06:33:25Z DEBUG The ipa-replica-install
>>>>>> command failed, exception: RuntimeError: Configuration of
>>>>>> CA failed
>>>>> 
>>>>> 
>>>>>> So some of the required certificates must be missing
>>>>>> still.
>>>>> 
>>>>>> Unhelpfully, the ipa-server-install --uninstall process
>>>>>> is not cleaning up everything after this failure, it
>>>>>> leaves the CA intact and the next run through the
>>>>>> installer believes the CA is working so it does not
>>>>>> configure it. As such, I guess a re-install is necessary
>>>>>> or some other steps to truly clean everything that I
>>>>>> haven't found yet.
>>>>> 
>>>>>> -Erinn
>>>>> 
>>>>> Continuing on, in order to remove the CA I am manually 
>>>>> running: pkidestroy -s CA -i pki-tomcat
>>>>> 
>>>>> And indeed there is a bug: 
>>>>> https://fedorahosted.org/freeipa/ticket/2796
>>>>> 
>>>>> Interesting that the installer detects that the CA is 
>>>>> installed, but the uninstaller does not detect it. I guess
>>>>> they are doing their detection in different ways.
>>>> 
>>>> The uninstaller doesn't rely on detection. There is a stored
>>>> log of what needs to be done. Unfortunately in this case the
>>>> fact that the CA was configured was added AFTER it was
>>>> successfully installed and not when we started, so if
>>>> installation fails it can leave things half-installed but not
>>>> recorded.
>>>> 
>>>>> At this point I wanted to explore how feasible it would be
>>>>> to have a RHEL 7 replica without the CA replica portion,
>>>>> this ought to alleviate the KDC issues I seem to be having
>>>>> on the primary, which I have still to figure out.
>>>>> 
>>>>> So any reason not to do that? Would I simply be able to do
>>>>> a ipa-ca-install on the rhel 7 system at a future juncture
>>>>> and then perform the rest of the migration?
>>>> 
>>>> This would be a reasonable short-term stop-gap measure though
>>>> if you can live without a second CA. You would likely have
>>>> the same problem with ipa-ca-install, at least until we
>>>> figure out what this missing cert error means.
>>>> 
>>>> I've seen that error about missing certs before but I can't 
>>>> recall what it means. I have the vague notion it is a little 
>>>> misleading though, and that something else has already
>>>> failed. I think we'll need one of the dogtag devs to chime
>>>> in. I'll poke them out-of-band.
>>> 
>>> Ok, start with the debug log on the clone ( 
>>> /var/log/pki/pki-tomcat/ca/debug ). It should tell you which
>>> cert is missing or unreadable.
>>> 
>>> How did you re-create the PKCS#12 file on the RHEL-6 server?
>>> You used PKCS12Export, right?
>>> 
>>> rob
>>> 
>> 
>> Correct, I just did the steps as if I was changing the dir
>> manager password, to re-export the certificates.
>> 
>> To my untrained eye it looks like the server-cert that is
>> failing, but here are what I believe the pertinent bits from the
>> debug log:
>> 
>> [27/Jul/2014:20:46:24][http-bio-8443-exec-3]: updateNumberRange: 
>> Failed to contact master using admin 
>> portorg.xml.sax.SAXParseException; lineNumber: 1; columnNumber:
>> 50; White spaces are required between publicId and systemId. 
>> [27/Jul/2014:20:46:24][http-bio-8443-exec-3]: updateNumberRange: 
>> Attempting to contact master using EE port 
>> [27/Jul/2014:20:46:25][http-bio-8443-exec-3]: content from ee 
>> interface =<?xml version="1.0" encoding="UTF-8" 
>> standalone="no"?><XMLResponse><Status>0</Status><beginNumber>66</beginNumber><endNumber>70</endNumber></XMLResponse>
>>
>> 
[27/Jul/2014:20:46:25][http-bio-8443-exec-3]: updateNumberRange():
>> status=0 [27/Jul/2014:20:46:25][http-bio-8443-exec-3]:
>> updateConfigEntries start 
>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]:
>> updateConfigEntries: status=0 
>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]:
>> deleteExistingCerts: 
>> Exception=org.mozilla.jss.crypto.ObjectNotFoundException 
>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]:
>> deleteExistingCerts: 
>> Exception=org.mozilla.jss.crypto.NoSuchItemOnTokenException 
>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]:
>> deleteExistingCerts: 
>> Exception=org.mozilla.jss.crypto.ObjectNotFoundException 
>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]:
>> deleteExistingCerts: 
>> Exception=org.mozilla.jss.crypto.NoSuchItemOnTokenException 
>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]:
>> deleteExistingCerts: 
>> Exception=org.mozilla.jss.crypto.ObjectNotFoundException 
>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]:
>> deleteExistingCerts: 
>> Exception=org.mozilla.jss.crypto.NoSuchItemOnTokenException 
>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]:
>> deleteExistingCerts: 
>> Exception=org.mozilla.jss.crypto.ObjectNotFoundException 
>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]:
>> deleteExistingCerts: 
>> Exception=org.mozilla.jss.crypto.NoSuchItemOnTokenException 
>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm
>> 'RSA' [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm
>> 'RSA' [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm
>> 'RSA' [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm
>> 'RSA' [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm
>> 'RSA' [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm
>> 'RSA' [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm
>> 'RSA' [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm
>> 'RSA' [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Ignoring key 
>> CN=ipa.example.com,O=EXAMPLE.COM 
>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm
>> 'RSA' [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Not
>> importing Server-Cert cert-pki-ca
> 
> The above line is absolutely correct.  The server cert is not
> imported into the clone.  The clone will generate its own server
> cert.
> 
>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: isCertdbCloned: 
>> caSigningCert cert-pki-ca 
>> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: clone does not have
>> all the certificates.
>> 
> This actually indicates to me that the signing cert is not present
> in the P12 replica file, or that some error has been encountered in
> reading the cert.
> 
> Also, check in the journal to see if any exceptions have been
> thrown. journalctl -u pki-tomcatd at pki-tomcat.service
> 
> Ade
> 

No exceptions thrown in the journal.

When investigating the cacert.p12 file that is bundled up for the
replica's I see two caSigningCert's. One is the older one, before I
renewed and one is the new, valid, post renewal one. Are these the
certs that are used or are they requested from the primary much like
the servercert?

However, when examining the /etc/pki/pki-tomcat/alias db there is no
casigningcert, hence I suppose the failure. So somewhere in there it
is getting lost, I'll keep looking but throw me any ideas.

- -Erinn



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCAAGBQJT1mu7AAoJEFg7BmJL2iPOmjwH/1pG2Sd8IPqfOzyN16hkibpz
ibvgLSUGb48vNhJayeooBwsNW4UALIVX9zFibXzUtGZbpj3Z7HdKqWFpFqPVNh/G
UGJXpZsIq+GAXhpXbNod0fefsq9LdE50okMa+xqV8GpvjOSl05DqejCVrq+pEFb0
vbiBEeMboyWNWkPtYPMl+QfgKJ7E0KOtdpGEI7eIEQUQjiNvnT7ppOnVo7o3/Vtb
tpHxDVWQKSB9KMjGIxBYDqirHi8ndS/T4cyY6FYeHEW1ICe56avAORDdo5N4uhBI
ADbvj2xStCRT9WDZdpjhnLcr5/fH2Xd6U9J59FMVzNCWLjR+pHPF0lqY8oOv3ys=
=1I8f
-----END PGP SIGNATURE-----

More information about the Freeipa-users mailing list