[Freeipa-users] RHEL 7 Upgrade experience so far

[Ade Lee]

On Mon, 2014-07-28 at 07:41 -0700, Erinn Looney-Triggs wrote:
> On 07/28/2014 07:17 AM, Rob Crittenden wrote:
> > Rob Crittenden wrote:
> >> Erinn Looney-Triggs wrote:
> >>> On 07/27/2014 12:02 AM, Erinn Looney-Triggs wrote:
> >>>> On 07/26/2014 07:12 PM, Erinn Looney-Triggs wrote:
> >>>>> On 07/26/2014 05:25 PM, Erinn Looney-Triggs wrote:
> >>>>>> Well it hasn't been all the pretty trying to move from
> >>>>>> RHEL 6.5 to RHEL 7.
> >>> 
> >>>>>> I have two servers providing my ipa instances ipa and
> >>>>>> ipa2. Given that I don't have a great deal of spare
> >>>>>> capacity the plan was to remove ipa2 from the replication
> >>>>>> agreement, modify DNS so that only IPA was available in
> >>>>>> SRV logs (IPA does not manage DNS at this point, was
> >>>>>> waiting for DNSSEC). As well, I would change my sudo-ldap
> >>>>>> config files to point to ipa and remove ipa2.
> >>> 
> >>>>>> Well that all worked well, installed RHEL 7 on the system
> >>>>>> and began working through the steps in the upgrade
> >>>>>> guide.
> >>> 
> >>>>>> First major problem was running into this bug: 
> >>>>>> https://fedorahosted.org/freeipa/ticket/4375 ValueError:
> >>>>>>  nsDS5ReplicaId has 2 values, one expected.
> >>> 
> >>>>>> Went and patched the replication.py file to get around
> >>>>>> that issue, and we moved on.
> >>> 
> >>>>>> Next up is my current issue: Exception from Java
> >>>>>> Configuration Servlet: Clone does not have all the
> >>>>>> required certificates.
> >>> 
> >>>>>> I suspect this is because I am running the CA as a
> >>>>>> subordinate to an AD CS instance, but I am unsure at this
> >>>>>> point.
> >>> 
> >>>>>> It has been a haul to get here, despite the short
> >>>>>> explanation. It seems that my primary ipa instance is
> >>>>>> working on only a hit or miss basis for kerberos tickets
> >>>>>> which has made all this a bit of a pain. You can kinit as
> >>>>>> admin once it will fail unable to find KDC, try again
> >>>>>> another three times, it will work. I have even modified
> >>>>>> the krb5.conf file to point directly at the server, thus
> >>>>>> bypassing DNS SRV lookups, however, that hasn't worked.
> >>> 
> >>>>>> Point is, any help would be appreciated on the
> >>>>>> aforementioned error.
> >>> 
> >>>>>> -Erinn
> >>> 
> >>> 
> >>>>> To reply to myself here, I believe the problem may be that
> >>>>> I had to renew the CA certificates and as such the
> >>>>> certificates in /root/cacert.p12 are no longer valid. It is
> >>>>> this file that gets bundled up with whatever else using
> >>>>> ipa-replica-prepare, so I will have to create a new one
> >>>>> that has the valid certificates in it.
> >>> 
> >>>>> One way or another though, if it isn't already documented,
> >>>>> during a CA renewal this file should probably be updated
> >>>>> with the correct certificates.
> >>> 
> >>>>> -Erinn
> >>> 
> >>>>> -Erinn
> >>> 
> >>> 
> >>> 
> >>>> Well thanks to this: 
> >>>> http://www.freeipa.org/page/Howto/Change_Directory_Manager_Password
> >>>
> >>>>
> >>>> 
> I have gotten a little further down the road an created a new
> >>>> cacert.p12 which looks to be complete.
> >>> 
> >>>> However, installation still fails in the same place:
> >>> 
> >>>> 2014-07-27T06:33:04Z DEBUG Starting external process 
> >>>> 2014-07-27T06:33:04Z DEBUG args=/usr/sbin/pkispawn -s CA -f 
> >>>> /tmp/tmp5QGhUx 2014-07-27T06:33:25Z DEBUG Process finished,
> >>>> return code=1 2014-07-27T06:33:25Z DEBUG stdout=Loading
> >>>> deployment configuration from /tmp/tmp5QGhUx. Installing CA
> >>>> into /var/lib/pki/pki-tomcat. Storing deployment
> >>>> configuration into 
> >>>> /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg. 
> >>>> Installation failed.
> >>> 
> >>> 
> >>>> 2014-07-27T06:33:25Z DEBUG stderr=pkispawn    : WARNING
> >>>> ....... unable to validate security domain user/password
> >>>> through REST interface. Interface not available pkispawn    :
> >>>> ERROR    ....... Exception from Java Configuration Servlet:
> >>>> Clone does not have all the required certificates
> >>> 
> >>>> 2014-07-27T06:33:25Z CRITICAL failed to configure ca
> >>>> instance Command '/usr/sbin/pkispawn -s CA -f /tmp/tmp5QGhUx'
> >>>> returned non-zero exit status 1 2014-07-27T06:33:25Z DEBUG
> >>>> File 
> >>>> "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py",
> >>>
> >>>
> >>>
> >>>> 
> line 638, in run_script
> >>>> return_value = main_function()
> >>> 
> >>>> File "/usr/sbin/ipa-replica-install", line 667, in main CA = 
> >>>> cainstance.install_replica_ca(config)
> >>> 
> >>>> File 
> >>>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
> >>>
> >>>
> >>>
> >>>> 
> line 1678, in install_replica_ca
> >>>> subject_base=config.subject_base)
> >>> 
> >>>> File 
> >>>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
> >>>
> >>>
> >>>
> >>>> 
> line 478, in configure_instance
> >>>> self.start_creation(runtime=210)
> >>> 
> >>>> File 
> >>>> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
> >>>>
> >>>> 
> line 364, in start_creation method()
> >>> 
> >>>> File 
> >>>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
> >>>
> >>>
> >>>
> >>>> 
> line 604, in __spawn_instance
> >>>> raise RuntimeError('Configuration of CA failed')
> >>> 
> >>>> 2014-07-27T06:33:25Z DEBUG The ipa-replica-install command
> >>>> failed, exception: RuntimeError: Configuration of CA failed
> >>> 
> >>> 
> >>>> So some of the required certificates must be missing still.
> >>> 
> >>>> Unhelpfully, the ipa-server-install --uninstall process is
> >>>> not cleaning up everything after this failure, it leaves the
> >>>> CA intact and the next run through the installer believes the
> >>>> CA is working so it does not configure it. As such, I guess a
> >>>> re-install is necessary or some other steps to truly clean
> >>>> everything that I haven't found yet.
> >>> 
> >>>> -Erinn
> >>> 
> >>> Continuing on, in order to remove the CA I am manually
> >>> running: pkidestroy -s CA -i pki-tomcat
> >>> 
> >>> And indeed there is a bug:
> >>> https://fedorahosted.org/freeipa/ticket/2796
> >>> 
> >>> Interesting that the installer detects that the CA is
> >>> installed, but the uninstaller does not detect it. I guess they
> >>> are doing their detection in different ways.
> >> 
> >> The uninstaller doesn't rely on detection. There is a stored log
> >> of what needs to be done. Unfortunately in this case the fact
> >> that the CA was configured was added AFTER it was successfully
> >> installed and not when we started, so if installation fails it
> >> can leave things half-installed but not recorded.
> >> 
> >>> At this point I wanted to explore how feasible it would be to
> >>> have a RHEL 7 replica without the CA replica portion, this
> >>> ought to alleviate the KDC issues I seem to be having on the
> >>> primary, which I have still to figure out.
> >>> 
> >>> So any reason not to do that? Would I simply be able to do a 
> >>> ipa-ca-install on the rhel 7 system at a future juncture and
> >>> then perform the rest of the migration?
> >> 
> >> This would be a reasonable short-term stop-gap measure though if
> >> you can live without a second CA. You would likely have the same
> >> problem with ipa-ca-install, at least until we figure out what
> >> this missing cert error means.
> >> 
> >> I've seen that error about missing certs before but I can't
> >> recall what it means. I have the vague notion it is a little
> >> misleading though, and that something else has already failed. I
> >> think we'll need one of the dogtag devs to chime in. I'll poke
> >> them out-of-band.
> > 
> > Ok, start with the debug log on the clone ( 
> > /var/log/pki/pki-tomcat/ca/debug ). It should tell you which cert
> > is missing or unreadable.
> > 
> > How did you re-create the PKCS#12 file on the RHEL-6 server? You
> > used PKCS12Export, right?
> > 
> > rob
> > 
> 
> Correct, I just did the steps as if I was changing the dir manager
> password, to re-export the certificates.
> 
> To my untrained eye it looks like the server-cert that is failing, but
> here are what I believe the pertinent bits from the debug log:
> 
> [27/Jul/2014:20:46:24][http-bio-8443-exec-3]: updateNumberRange:
> Failed to contact master using admin
> portorg.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 50;
> White spaces are required between publicId and systemId.
> [27/Jul/2014:20:46:24][http-bio-8443-exec-3]: updateNumberRange:
> Attempting to contact master using EE port
> [27/Jul/2014:20:46:25][http-bio-8443-exec-3]: content from ee
> interface =<?xml version="1.0" encoding="UTF-8"
> standalone="no"?><XMLResponse><Status>0</Status><beginNumber>66</beginNumber><endNumber>70</endNumber></XMLResponse>
> [27/Jul/2014:20:46:25][http-bio-8443-exec-3]: updateNumberRange():
> status=0
> [27/Jul/2014:20:46:25][http-bio-8443-exec-3]: updateConfigEntries start
> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: updateConfigEntries:
> status=0
> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: deleteExistingCerts:
> Exception=org.mozilla.jss.crypto.ObjectNotFoundException
> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: deleteExistingCerts:
> Exception=org.mozilla.jss.crypto.NoSuchItemOnTokenException
> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: deleteExistingCerts:
> Exception=org.mozilla.jss.crypto.ObjectNotFoundException
> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: deleteExistingCerts:
> Exception=org.mozilla.jss.crypto.NoSuchItemOnTokenException
> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: deleteExistingCerts:
> Exception=org.mozilla.jss.crypto.ObjectNotFoundException
> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: deleteExistingCerts:
> Exception=org.mozilla.jss.crypto.NoSuchItemOnTokenException
> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: deleteExistingCerts:
> Exception=org.mozilla.jss.crypto.ObjectNotFoundException
> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: deleteExistingCerts:
> Exception=org.mozilla.jss.crypto.NoSuchItemOnTokenException
> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm 'RSA'
> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm 'RSA'
> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm 'RSA'
> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm 'RSA'
> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm 'RSA'
> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm 'RSA'
> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm 'RSA'
> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm 'RSA'
> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Ignoring key
> CN=ipa.example.com,O=EXAMPLE.COM
> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Key Algorithm 'RSA'
> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: Not importing
> Server-Cert cert-pki-ca

The above line is absolutely correct.  The server cert is not imported
into the clone.  The clone will generate its own server cert.

> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: isCertdbCloned:
> caSigningCert cert-pki-ca
> [27/Jul/2014:20:46:26][http-bio-8443-exec-3]: clone does not have all
> the certificates.
> 
This actually indicates to me that the signing cert is not present in
the P12 replica file, or that some error has been encountered in reading
the cert.

Also, check in the journal to see if any exceptions have been thrown.
journalctl -u pki-tomcatd at pki-tomcat.service

Ade

> Interestingly, when I do:
> 
> certutil -L -d /etc/pki/pki-tomcat/alias/ -n "Server-Cert cert-pki-ca"
> 
> Certificate:
>     Data:
>         Version: 3 (0x2)
>         Serial Number: 0 (0x0)
>         Signature Algorithm: PKCS #1 SHA-1 With RSA Encryption
>         Issuer: "CN=ipa2.example.com,O=2014-07-27 20:46:11"
>         Validity:
>             Not Before: Sun Jul 27 20:46:13 2014
>             Not After : Mon Jul 27 20:46:13 2015
>         Subject: "CN=ipa2.example.com,O=2014-07-27 20:46:11"
>         Subject Public Key Info:
>             Public Key Algorithm: PKCS #1 RSA Encryption
>             RSA Public Key:
>                 Modulus:
>                     ac:f1:74:8b:d0:fa:83:5a:e9:58:fa:b4:61:dc:d2:0f:
>                     66:4e:9f:61:ef:dc:47:0e:40:f8:89:45:7a:9c:1a:bf:
>                     87:a3:a3:b3:06:ab:98:f7:3f:58:a4:4e:78:fe:c5:b5:
>                     01:33:35:f6:0b:a2:7a:be:40:a2:76:69:61:4a:6f:1e:
>                     c5:3f:c4:35:3c:dd:b0:14:c8:cd:37:e2:f6:c7:9f:53:
>                     56:83:c6:74:dc:b8:f8:f5:dc:35:3f:e3:e7:f5:74:8f:
>                     69:75:56:0b:cb:6e:04:3c:4a:16:67:92:63:14:92:4e:
>                     ec:86:77:73:86:81:fe:01:04:2b:c2:61:13:af:70:e7
>                 Exponent: 65537 (0x10001)
>     Signature Algorithm: PKCS #1 SHA-1 With RSA Encryption
>     Signature:
>         7b:d6:22:fe:df:61:2e:30:c0:76:9f:1e:59:88:7f:14:
>         e3:75:e0:7b:0f:67:07:73:ba:79:59:09:4e:86:2b:9a:
>         a9:8b:c4:fd:88:c4:fb:a2:1c:d9:61:70:af:55:51:09:
>         35:93:f8:4e:d4:fa:7c:a0:68:fe:5a:c0:13:af:33:6a:
>         7a:b5:7e:f5:e3:5a:14:b6:53:0d:19:36:ed:e2:cb:38:
>         34:55:23:6b:4f:d8:6f:aa:f1:3e:12:1e:98:71:3b:0a:
>         29:53:ef:10:39:d3:9e:66:05:e9:9d:aa:1a:b0:4a:9a:
>         af:f2:32:85:07:f5:d0:0f:08:04:05:8b:f9:f9:bc:43
>     Fingerprint (MD5):
>         85:56:1B:40:91:CB:5E:A1:2B:A0:01:68:C8:57:39:B9
>     Fingerprint (SHA1):
>         54:48:56:07:CC:07:3A:87:A0:6C:D2:5A:7F:2B:99:BF:89:87:27:0E
> 
>     Certificate Trust Flags:
>         SSL Flags:
>             Valid CA
>             Trusted CA
>             User
>             Trusted Client CA
>         Email Flags:
>             Valid CA
>             Trusted CA
>             User
>         Object Signing Flags:
>             Valid CA
>             Trusted CA
>             User
> 
> Which would appear to be a valid certificate, but I may be chasing
> down the wrong path.
> 
> -Erinn
>