[Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18

[Ade Lee]

On Wed, 2012-09-05 at 16:20 -0400, Rob Crittenden wrote:
> Martin Kosek wrote:
> > On 08/31/2012 04:53 PM, Petr Viktorin wrote:
> >> On 08/28/2012 03:40 PM, Petr Viktorin wrote:
> >>> On 08/17/2012 06:04 PM, Ade Lee wrote:
> >>>> On Fri, 2012-08-17 at 09:34 -0400, Ade Lee wrote:
> >>>>> On Thu, 2012-08-16 at 18:45 +0200, Martin Kosek wrote:
> >>>>>> On 08/16/2012 01:28 PM, Ade Lee wrote:
> >>>>>>> Patch attached this time.  I should know better than to do this in the
> >>>>>>> middle of the night ..
> >>>>>>>
> >>>>>>> On Thu, 2012-08-16 at 09:12 +0200, Martin Kosek wrote:
> >>>>>>>> On 08/16/2012 07:53 AM, Ade Lee wrote:
> >>>>>>>>> On Wed, 2012-08-15 at 23:41 -0400, Ade Lee wrote:
> >>>>>>>>>> On Wed, 2012-08-15 at 16:34 +0200, Martin Kosek wrote:
> >>>>>>>>>>> On 08/15/2012 03:54 PM, Ade Lee wrote:
> >>>>>>>>>>>> On Wed, 2012-08-15 at 13:24 +0200, Martin Kosek wrote:
> >>>>>>>>>>>>> On 08/08/2012 10:05 PM, Ade Lee wrote:
> >>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Dogtag 10 is being released on f18, and has a number of
> >>>>>>>>>>>>>> changes that
> >>>>>>>>>>>>>> will affect IPA.  In particular, the following changes will
> >>>>>>>>>>>>>> affect
> >>>>>>>>>>>>>> current IPA code.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> * The directory layout of the dogtag instance has changed.
> >>>>>>>>>>>>>> Instead of
> >>>>>>>>>>>>>> using separate tomcat instances to host different
> >>>>>>>>>>>>>> subsystems, the
> >>>>>>>>>>>>>> standard dogtag installation will allow one to install a CA.
> >>>>>>>>>>>>>> KRA, OCSP
> >>>>>>>>>>>>>> and TKS within the same instance.  There have been
> >>>>>>>>>>>>>> corresponding changes
> >>>>>>>>>>>>>> in the directory layout, as well as the default instance name
> >>>>>>>>>>>>>> (pki-tomcat instead of pki-ca), and startup daemon
> >>>>>>>>>>>>>> (pki-tomcatd, instead
> >>>>>>>>>>>>>> of pki-cad, pki-krad etc.)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> * The default instance will use only four ports (HTTPS,
> >>>>>>>>>>>>>> HTTP, AJP and
> >>>>>>>>>>>>>> tomcat shutdown port) rather than the 6 previously used.
> >>>>>>>>>>>>>> The default
> >>>>>>>>>>>>>> ports will be changed to the standard tomcat ports.  As
> >>>>>>>>>>>>>> these ports are
> >>>>>>>>>>>>>> local to the ipa server machine, this should not cause too much
> >>>>>>>>>>>>>> disruption.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> * There is a new single step installer written in python.
> >>>>>>>>>>>>>> (pkispawn/destroy) vs. pkicreate/pkisilent/pkiremove.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> * Dogtag 10 runs on tomcat7 - with a new corresponding
> >>>>>>>>>>>>>> version of
> >>>>>>>>>>>>>> tomcatjss.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> The attached patch integrates all the above changes in IPA
> >>>>>>>>>>>>>> installation
> >>>>>>>>>>>>>> and maintenance code.  Once the patch is applied, users will
> >>>>>>>>>>>>>> be able to:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 1. run ipa-server-install to completion on f18 with dogtag 10.
> >>>>>>>>>>>>>> 2. install a new replica on f18 on dogtag 10.
> >>>>>>>>>>>>>> 3. upgrade an f17 machine with an existing IPA instance to
> >>>>>>>>>>>>>> f18/ dogtag
> >>>>>>>>>>>>>> 10 - and have that old-style dogtag instance continue to run
> >>>>>>>>>>>>>> correctly.
> >>>>>>>>>>>>>> This will require the installation of the latest version of
> >>>>>>>>>>>>>> tomcatjss as
> >>>>>>>>>>>>>> well as the installation of tomcat6.  The old-style instance
> >>>>>>>>>>>>>> will
> >>>>>>>>>>>>>> continue to use tomcat6.
> >>>>>>>>>>>>>> 4. in addition, the new cert renewal code has been patched
> >>>>>>>>>>>>>> and should
> >>>>>>>>>>>>>> continue to work.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> What is not yet completed / supported:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 1. Installation with an external CA is not yet completed in
> >>>>>>>>>>>>>> the new
> >>>>>>>>>>>>>> installer.  We plan to complete this soon.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 2. There is some IPA upgrade code that has not yet been touched
> >>>>>>>>>>>>>> (install/tools/ipa-upgradeconfig).
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 3. A script needs to be written to allow admins to convert
> >>>>>>>>>>>>>> their
> >>>>>>>>>>>>>> old-style dogtag instances to new style instances, as well
> >>>>>>>>>>>>>> as code to
> >>>>>>>>>>>>>> periodically prompt admins to do this.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 4. Installation of old-style instances using
> >>>>>>>>>>>>>> pkicreate/pkisilent on
> >>>>>>>>>>>>>> dogtag 10 will no longer be supported, and will be disabled
> >>>>>>>>>>>>>> soon.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 5.  The pki-selinux policy has been updated to reflect these
> >>>>>>>>>>>>>> changes,
> >>>>>>>>>>>>>> but is still in flux.  In fact, it is our intention to place
> >>>>>>>>>>>>>> the dogtag
> >>>>>>>>>>>>>> selinux policy in the base selinux policy for f18.  In the
> >>>>>>>>>>>>>> meantime, it
> >>>>>>>>>>>>>> may be necessary to run installs in permissive mode.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> The dogtag 10 code will be released shortly into f18.  Prior
> >>>>>>>>>>>>>> to that
> >>>>>>>>>>>>>> though, we have placed the new dogtag 10 and tomcatjss code
> >>>>>>>>>>>>>> in a
> >>>>>>>>>>>>>> developer repo that is located at
> >>>>>>>>>>>>>> http://nkinder.fedorapeople.org/dogtag-devel/
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Testing can be done on both f18 and f17 - although the
> >>>>>>>>>>>>>> target platform -
> >>>>>>>>>>>>>> and the only platform for which official builds will be
> >>>>>>>>>>>>>> created is f18.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>> Ade
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Ade,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for the patch, I started with review and integration
> >>>>>>>>>>>>> tests (currently
> >>>>>>>>>>>>> running on Fedora 17 with Nathan's repo).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Installation on single master was smooth, it worked just
> >>>>>>>>>>>>> fine, even with
> >>>>>>>>>>>>> enforced SELinux, without any error - kudos to you and the
> >>>>>>>>>>>>> whole dogtag team.
> >>>>>>>>>>>>> The resulting logs and the structure of your log directory
> >>>>>>>>>>>>> seems improved. I
> >>>>>>>>>>>>> believe that the brand new Python installers will make it
> >>>>>>>>>>>>> easier to debug
> >>>>>>>>>>>>> issues with dogtag installation when they come.  When I tried
> >>>>>>>>>>>>> our unit tests or
> >>>>>>>>>>>>> some simple cert operation, it worked fine as well.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Now the bad news, or rather few issues and suggestions I found:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> 1) As we already discussed on IRC, tomcat 7 was not pulled
> >>>>>>>>>>>>> automatically on
> >>>>>>>>>>>>> Fedora 17 when I updated pki-ca, you somewhere miss a Requires.
> >>>>>>>>>>>>>
> >>>>>>>>>>>> We have a dogtag patch that is currently in review that will
> >>>>>>>>>>>> address
> >>>>>>>>>>>> this.  Once this is in, tomcatjss >=7.0.0 will be required for
> >>>>>>>>>>>> f17+,
> >>>>>>>>>>>> rather than f18+
> >>>>>>>>>>>>
> >>>>>>>>>>>>> 2) I had installed IPA with dogtag10 on master. However, CA
> >>>>>>>>>>>>> installation on a
> >>>>>>>>>>>>> replica (ipa-ca-install) with dogtag9 failed - is this
> >>>>>>>>>>>>> expectable?
> >>>>>>>>>>>>>
> >>>>>>>>>>>> Yes.  The current IPA patch is designed to work with dogtag 10
> >>>>>>>>>>>> only,
> >>>>>>>>>>>> which will be officially available on f18+.  So if you update
> >>>>>>>>>>>> to dogtag
> >>>>>>>>>>>> 10, you must have this patch and visa versa.  We probably need
> >>>>>>>>>>>> to add
> >>>>>>>>>>>> the relevant requires to IPA to enforce this.
> >>>>>>>>>>>>
> >>>>>>>>>>>> If you have an existing dogtag 9 instance, and you upgrade to
> >>>>>>>>>>>> the new
> >>>>>>>>>>>> dogtag 10 and patched IPA, then that instance will continue to
> >>>>>>>>>>>> work.
> >>>>>>>>>>>> But any new instances would be created using dogtag 10.
> >>>>>>>>>>>>
> >>>>>>>>>>>>> 3) I had installed IPA with dogtag10 on master. Replica had
> >>>>>>>>>>>>> dogtag10 as well
> >>>>>>>>>>>>> and I got the following error:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> # ipa-ca-install
> >>>>>>>>>>>>> /home/mkosek/replica-info-vm-114.idm.lab.bos.redhat.com.gpg
> >>>>>>>>>>>>> ...
> >>>>>>>>>>>>> Configuring certificate server: Estimated time 3 minutes 30
> >>>>>>>>>>>>> seconds
> >>>>>>>>>>>>>     [1/14]: creating certificate server user
> >>>>>>>>>>>>>     [2/14]: configuring certificate server instance
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Your system may be partly configured.
> >>>>>>>>>>>>> Run /usr/sbin/ipa-server-install --uninstall to clean up.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Unexpected error - see /var/log/ipareplica-ca-install.log for
> >>>>>>>>>>>>> details:
> >>>>>>>>>>>>> IOError: [Errno 2] No such file or directory:
> >>>>>>>>>>>>> '/var/lib/pki/pki-tomcat/alias/ca_backup_keys.p12'
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Root cause:
> >>>>>>>>>>>>> ...
> >>>>>>>>>>>>>     File
> >>>>>>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
> >>>>>>>>>>>>> line
> >>>>>>>>>>>>> 625, in __spawn_instance
> >>>>>>>>>>>>>       "/root/cacert.p12")
> >>>>>>>>>>>>> ...
> >>>>>>>>>>>>>
> >>>>>>>>>>>> I need to look into this.  I had fixed ipa-replica-install,
> >>>>>>>>>>>> rather than
> >>>>>>>>>>>> ipa-ca-install to create replicas.  I didn't know ipa-ca-install
> >>>>>>>>>>>> existed!  Should not be too bad to fix though - most likely
> >>>>>>>>>>>> just need to
> >>>>>>>>>>>> move files to the right place.
> >>>>>>>>>>>
> >>>>>>>>>>> Ok, thanks! Btw. CA on replica can be either installed during
> >>>>>>>>>>> ipa-replica-install time (when --setup-ca option is passed, you
> >>>>>>>>>>> probably used
> >>>>>>>>>>> that one) and the aforementioned ipa-ca-install run after
> >>>>>>>>>>> ipa-replica-install.
> >>>>>>>>>>>
> >>>>>>>>>> I will be testing this out again.  As ipa-ca-install uses the
> >>>>>>>>>> same code
> >>>>>>>>>> as ipa-replica-install, I would have expected it to work.  Did
> >>>>>>>>>> you try
> >>>>>>>>>> it with selinux in permissive mode?
> >>>>>>>>>>
> >>>>>>>>> OK -- I tested this out and realized that between the time I
> >>>>>>>>> initially
> >>>>>>>>> tested this and your tests, we had changed a URL
> >>>>>>>>> from /ca/pki/installer/installToken to
> >>>>>>>>> /ca/rest/installer/installToken,
> >>>>>>>>> but I forgot to update ipa-pki-proxy.conf.
> >>>>>>>>>
> >>>>>>>>> The new patch has been rebased and changes the relevant patch.  With
> >>>>>>>>> this, the CA replica installs correctly.
> >>>>>>>>
> >>>>>>>> Umh, where can I find the new rebased patch? :-)
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>> There was one issue that I encountered.  I have seen this once
> >>>>>>>>> before
> >>>>>>>>> and will need to investigate further.  IPA sometimes restarts the
> >>>>>>>>> dogtag
> >>>>>>>>> instance by doing systemctl stop pki-tomcatd.target.  and then
> >>>>>>>>> systemctl
> >>>>>>>>> start pki-tomcatd.target.   I have noticed that occasionally the
> >>>>>>>>> start
> >>>>>>>>> invocation fails, while the stop and restart invocations succeed
> >>>>>>>>> just
> >>>>>>>>> fine.  This makes absolutely no sense because restart is essentially
> >>>>>>>>> just stop and then start.
> >>>>>>>>>
> >>>>>>>>> I think this is actually a bug in systemd.  If I reboot the
> >>>>>>>>> machine, it
> >>>>>>>>> all works perfectly.  If we can't figure out whats going on with
> >>>>>>>>> systemd, we can always replace this start/stop with
> >>>>>>>>> starting/stopping
> >>>>>>>>> the service directly.  (pki-tomcatd at pki-tomcat.service).  That
> >>>>>>>>> seems to
> >>>>>>>>> work all the time with no problems.
> >>>>>>>>>
> >>>>>>>>> I suggest we leave this fix (if needed) for a separate patch.
> >>>>>>>>
> >>>>>>>> Ok, I will see if I hit this issue again. Btw. is it recommended
> >>>>>>>> in systemd to
> >>>>>>>> use target units this way? I thought that only service files are
> >>>>>>>> meant to be
> >>>>>>>> used for manipulation with a service. As you said yourself, using
> >>>>>>>> service unit
> >>>>>>>> file for restart worked every time.
> >>>>>>>>
> >>>>>>>> Martin
> >>>>>>>
> >>>>>>
> >>>>>> Now, I tested the rebased patch with VMs with permissive SELinux.
> >>>>>> IPA server
> >>>>>> install was OK, as always, but replica installation ended up with
> >>>>>> error.
> >>>>>> Although it got much further than before:
> >>>>>>
> >>>>>> # ipa-ca-install
> >>>>>> /home/mkosek/replica-info-vm-114.idm.lab.bos.redhat.com.gpg
> >>>>>> ...
> >>>>>>     [3/3]: restarting directory server
> >>>>>> done configuring pkids.
> >>>>>> Configuring certificate server: Estimated time 3 minutes 30 seconds
> >>>>>>     [1/14]: creating certificate server user
> >>>>>>     [2/14]: configuring certificate server instance
> >>>>>>     [3/14]: disabling nonces
> >>>>>>     [4/14]: importing CA chain to RA certificate database
> >>>>>>     [5/14]: fixing RA database permissions
> >>>>>>     [6/14]: setting up signing cert profile
> >>>>>>     [7/14]: set up CRL publishing
> >>>>>>     [8/14]: set certificate subject base
> >>>>>>     [9/14]: enabling Subject Key Identifier
> >>>>>>     [10/14]: configuring certificate server to start on boot
> >>>>>>     [11/14]: configure certmonger for renewals
> >>>>>>     [12/14]: configure clone certificate renewals
> >>>>>>     [13/14]: configure Server-Cert certificate renewal
> >>>>>>     [14/14]: Configure HTTP to proxy connections
> >>>>>> done configuring pki-tomcatd.
> >>>>>> Restarting the directory and certificate servers
> >>>>>>
> >>>>>> Your system may be partly configured.
> >>>>>> Run /usr/sbin/ipa-server-install --uninstall to clean up.
> >>>>>>
> >>>>>> It seems like that CA on a replica did not start and so a check for
> >>>>>> port 8080
> >>>>>> failed. Attached logs (pki-ca-logs.tgz) may provide more information
> >>>>>> to this issue.
> >>>>>>
> >>>>> This is the systemd issue I mentioned before.  I will submit a patch
> >>>>> which will change the restart mechanism to restart the service and not
> >>>>> the target.
> >>>>>>
> >>>>
> >>>> Patch attached.  This patch should be applied on top of the large patch
> >>>> (being reviewed).
> >>>>
> >>>>>> Now I am also testing upgrade from IPA+dogtag9 to PatchedIPA+dogtag10
> >>>>>> (permissive SELinux). Now, the service was upgraded smoothly, CA
> >>>>>> service
> >>>>>> could be restarted without failure. However, certificate operations now
> >>>>>> did not work:
> >>>>>>
> >>>>>> # ipactl restart
> >>>>>> Restarting Directory Service
> >>>>>> Restarting KDC Service
> >>>>>> Restarting KPASSWD Service
> >>>>>> Restarting MEMCACHE Service
> >>>>>> Restarting HTTP Service
> >>>>>> Restarting CA Service
> >>>>>> # ipactl status
> >>>>>> Directory Service: RUNNING
> >>>>>> KDC Service: RUNNING
> >>>>>> KPASSWD Service: RUNNING
> >>>>>> MEMCACHE Service: RUNNING
> >>>>>> HTTP Service: RUNNING
> >>>>>> CA Service: RUNNING
> >>>>>> # ipa cert-show 1
> >>>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to
> >>>>>> communicate with CMS (Internal Server Error)
> >>>>>>
> >>>>>> Attaching logs for this case as well (ipa-ca-logs-upgrade.tgz).
> >>>>>>
> >>>>> The reason for this is that the old dogtag instances are missing:
> >>>>> 1. a new parameter in CS.cfg
> >>>>> 2. a couple of symlinks
> >>>>>
> >>>>> I plan to fix this in the dogtag code.  Specifically,
> >>>>> 1. I will modify the dogtag code to provide a default value in the case
> >>>>> where the parameter does not exist.
> >>>>> 2. I plan to add a function to the dogtag startup scripts to check and
> >>>>> remake any required symlinks.
> >>>>>
> >>>>> Both of these changes are in the dogtag code, and do not affect this
> >>>>> patch.  As a workaround, to confirm that the upgrade actually works, do
> >>>>> the following manual steps on your upgraded instance:
> >>>>>
> >>>>> 1. Add the following parameter to CS.cfg
> >>>>>      pidDir=/var/run
> >>>>>
> >>>>> 2. Add the following links;
> >>>>>      cd /var/lib/pki-ca/common/lib
> >>>>>      ln -s /usr/share/java/apache-commons-codec.jar
> >>>>> apache-commons-codec.jar
> >>>>>      ln -s /usr/share/java/log4j.jar log4j.jar
> >>>>>
> >>>>> 3 Restart the dogtag instance
> >>>>>
> >>>>>> HTH,
> >>>>>> Martin
> >>>>>
> >>>>>
> >>>
> >>> The attached patch conditionalizes the changes, so that IPA supports
> >>> both Dogtag versions.
> >>>
> >>> I didn't put it in platform code for two reasons
> >>> - One platform (f17) can have either Dogtag version installed
> >>> - I didn't want to conflict with Timo Aaltonen's "platformizing" work.
> >>> According to his WIP patches, he plans to move the whole dogtag module
> >>> into platform code.
> >>>
> >>> I used the existence of /usr/bin/pkispawn as a test for Dogtag 10. If
> >>> you know a better way, please comment.
> >>>
> >>> The dogtag_version option is now always added to
> >>>
> >>> I've reverted the changes to install/ui/test/data/ipa_init.json, so
> >>> you'll have to change the ports manually when testing the UI with Dogtag
> >>> 10.
> >>>
> >>>
> >>>
> >>
> >> Attaching a patch with fixed ipa-pki-proxy.conf, as discussed on IRC.
> >>
> >
> > This approach looks good. I did not hit any regression on F17 with dogtag9, or
> > clean installs of IPA+dogtag10. I think we are getting close to pushing this code.
> >
> > The only issues I hit so far are for upgraded dogtag9 instances (on F17):
> >
> > 1) The service did not start for me. There were some SELinux AVCs and even when
> > I disabled SELinux the instance still did not start (logs attached).
> >
> > 2) Uninstall was also not clean, we leave some dogtag installation states for
> > upgraded dogtag9 instance:
> >
> > # ipa-server-install --uninstall --unattended
> > Shutting down all IPA services
> > Removing IPA client configuration
> > Unconfiguring ntpd
> > Unconfiguring CA directory server
> > Unconfiguring web server
> > Unconfiguring krb5kdc
> > Unconfiguring kadmin
> > Unconfiguring directory server
> > Unconfiguring ipa_memcached
> > ipa         : ERROR    Some installation state for pki-cad has not been
> > restored, see /var/lib/ipa/sysrestore/sysrestore.state
> >
> > /var/lib/ipa/sysrestore/sysrestore.state:
> > [pki-cad]
> > enabled = False
> 
> Ade is working on a new build to address the dogtag upgrade issues.
> 
> The left over state should be removed when we drop the separate instance 
> in the dogtag 9 -> 10 upgrade. I'm not completely sure reading the patch 
> who actually does this part, is this handled by dogtag?
> 
> rob
> 

The build is available on the developer repo.  Just do a yum update. 

As for this leftover state, this is an ipa file and such is never
handled by dogtag.  It must be handled somewhere within the ipa code.

Ade