[Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18
Rob Crittenden
rcritten at redhat.com
Wed Sep 5 20:20:08 UTC 2012
Martin Kosek wrote:
> On 08/31/2012 04:53 PM, Petr Viktorin wrote:
>> On 08/28/2012 03:40 PM, Petr Viktorin wrote:
>>> On 08/17/2012 06:04 PM, Ade Lee wrote:
>>>> On Fri, 2012-08-17 at 09:34 -0400, Ade Lee wrote:
>>>>> On Thu, 2012-08-16 at 18:45 +0200, Martin Kosek wrote:
>>>>>> On 08/16/2012 01:28 PM, Ade Lee wrote:
>>>>>>> Patch attached this time. I should know better than to do this in the
>>>>>>> middle of the night ..
>>>>>>>
>>>>>>> On Thu, 2012-08-16 at 09:12 +0200, Martin Kosek wrote:
>>>>>>>> On 08/16/2012 07:53 AM, Ade Lee wrote:
>>>>>>>>> On Wed, 2012-08-15 at 23:41 -0400, Ade Lee wrote:
>>>>>>>>>> On Wed, 2012-08-15 at 16:34 +0200, Martin Kosek wrote:
>>>>>>>>>>> On 08/15/2012 03:54 PM, Ade Lee wrote:
>>>>>>>>>>>> On Wed, 2012-08-15 at 13:24 +0200, Martin Kosek wrote:
>>>>>>>>>>>>> On 08/08/2012 10:05 PM, Ade Lee wrote:
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Dogtag 10 is being released on f18, and has a number of
>>>>>>>>>>>>>> changes that
>>>>>>>>>>>>>> will affect IPA. In particular, the following changes will
>>>>>>>>>>>>>> affect
>>>>>>>>>>>>>> current IPA code.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * The directory layout of the dogtag instance has changed.
>>>>>>>>>>>>>> Instead of
>>>>>>>>>>>>>> using separate tomcat instances to host different
>>>>>>>>>>>>>> subsystems, the
>>>>>>>>>>>>>> standard dogtag installation will allow one to install a CA.
>>>>>>>>>>>>>> KRA, OCSP
>>>>>>>>>>>>>> and TKS within the same instance. There have been
>>>>>>>>>>>>>> corresponding changes
>>>>>>>>>>>>>> in the directory layout, as well as the default instance name
>>>>>>>>>>>>>> (pki-tomcat instead of pki-ca), and startup daemon
>>>>>>>>>>>>>> (pki-tomcatd, instead
>>>>>>>>>>>>>> of pki-cad, pki-krad etc.)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * The default instance will use only four ports (HTTPS,
>>>>>>>>>>>>>> HTTP, AJP and
>>>>>>>>>>>>>> tomcat shutdown port) rather than the 6 previously used.
>>>>>>>>>>>>>> The default
>>>>>>>>>>>>>> ports will be changed to the standard tomcat ports. As
>>>>>>>>>>>>>> these ports are
>>>>>>>>>>>>>> local to the ipa server machine, this should not cause too much
>>>>>>>>>>>>>> disruption.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * There is a new single step installer written in python.
>>>>>>>>>>>>>> (pkispawn/destroy) vs. pkicreate/pkisilent/pkiremove.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * Dogtag 10 runs on tomcat7 - with a new corresponding
>>>>>>>>>>>>>> version of
>>>>>>>>>>>>>> tomcatjss.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The attached patch integrates all the above changes in IPA
>>>>>>>>>>>>>> installation
>>>>>>>>>>>>>> and maintenance code. Once the patch is applied, users will
>>>>>>>>>>>>>> be able to:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1. run ipa-server-install to completion on f18 with dogtag 10.
>>>>>>>>>>>>>> 2. install a new replica on f18 on dogtag 10.
>>>>>>>>>>>>>> 3. upgrade an f17 machine with an existing IPA instance to
>>>>>>>>>>>>>> f18/ dogtag
>>>>>>>>>>>>>> 10 - and have that old-style dogtag instance continue to run
>>>>>>>>>>>>>> correctly.
>>>>>>>>>>>>>> This will require the installation of the latest version of
>>>>>>>>>>>>>> tomcatjss as
>>>>>>>>>>>>>> well as the installation of tomcat6. The old-style instance
>>>>>>>>>>>>>> will
>>>>>>>>>>>>>> continue to use tomcat6.
>>>>>>>>>>>>>> 4. in addition, the new cert renewal code has been patched
>>>>>>>>>>>>>> and should
>>>>>>>>>>>>>> continue to work.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What is not yet completed / supported:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1. Installation with an external CA is not yet completed in
>>>>>>>>>>>>>> the new
>>>>>>>>>>>>>> installer. We plan to complete this soon.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2. There is some IPA upgrade code that has not yet been touched
>>>>>>>>>>>>>> (install/tools/ipa-upgradeconfig).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 3. A script needs to be written to allow admins to convert
>>>>>>>>>>>>>> their
>>>>>>>>>>>>>> old-style dogtag instances to new style instances, as well
>>>>>>>>>>>>>> as code to
>>>>>>>>>>>>>> periodically prompt admins to do this.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 4. Installation of old-style instances using
>>>>>>>>>>>>>> pkicreate/pkisilent on
>>>>>>>>>>>>>> dogtag 10 will no longer be supported, and will be disabled
>>>>>>>>>>>>>> soon.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 5. The pki-selinux policy has been updated to reflect these
>>>>>>>>>>>>>> changes,
>>>>>>>>>>>>>> but is still in flux. In fact, it is our intention to place
>>>>>>>>>>>>>> the dogtag
>>>>>>>>>>>>>> selinux policy in the base selinux policy for f18. In the
>>>>>>>>>>>>>> meantime, it
>>>>>>>>>>>>>> may be necessary to run installs in permissive mode.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The dogtag 10 code will be released shortly into f18. Prior
>>>>>>>>>>>>>> to that
>>>>>>>>>>>>>> though, we have placed the new dogtag 10 and tomcatjss code
>>>>>>>>>>>>>> in a
>>>>>>>>>>>>>> developer repo that is located at
>>>>>>>>>>>>>> http://nkinder.fedorapeople.org/dogtag-devel/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Testing can be done on both f18 and f17 - although the
>>>>>>>>>>>>>> target platform -
>>>>>>>>>>>>>> and the only platform for which official builds will be
>>>>>>>>>>>>>> created is f18.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Ade
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Ade,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the patch, I started with review and integration
>>>>>>>>>>>>> tests (currently
>>>>>>>>>>>>> running on Fedora 17 with Nathan's repo).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Installation on single master was smooth, it worked just
>>>>>>>>>>>>> fine, even with
>>>>>>>>>>>>> enforced SELinux, without any error - kudos to you and the
>>>>>>>>>>>>> whole dogtag team.
>>>>>>>>>>>>> The resulting logs and the structure of your log directory
>>>>>>>>>>>>> seems improved. I
>>>>>>>>>>>>> believe that the brand new Python installers will make it
>>>>>>>>>>>>> easier to debug
>>>>>>>>>>>>> issues with dogtag installation when they come. When I tried
>>>>>>>>>>>>> our unit tests or
>>>>>>>>>>>>> some simple cert operation, it worked fine as well.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Now the bad news, or rather few issues and suggestions I found:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1) As we already discussed on IRC, tomcat 7 was not pulled
>>>>>>>>>>>>> automatically on
>>>>>>>>>>>>> Fedora 17 when I updated pki-ca, you somewhere miss a Requires.
>>>>>>>>>>>>>
>>>>>>>>>>>> We have a dogtag patch that is currently in review that will
>>>>>>>>>>>> address
>>>>>>>>>>>> this. Once this is in, tomcatjss >=7.0.0 will be required for
>>>>>>>>>>>> f17+,
>>>>>>>>>>>> rather than f18+
>>>>>>>>>>>>
>>>>>>>>>>>>> 2) I had installed IPA with dogtag10 on master. However, CA
>>>>>>>>>>>>> installation on a
>>>>>>>>>>>>> replica (ipa-ca-install) with dogtag9 failed - is this
>>>>>>>>>>>>> expectable?
>>>>>>>>>>>>>
>>>>>>>>>>>> Yes. The current IPA patch is designed to work with dogtag 10
>>>>>>>>>>>> only,
>>>>>>>>>>>> which will be officially available on f18+. So if you update
>>>>>>>>>>>> to dogtag
>>>>>>>>>>>> 10, you must have this patch and visa versa. We probably need
>>>>>>>>>>>> to add
>>>>>>>>>>>> the relevant requires to IPA to enforce this.
>>>>>>>>>>>>
>>>>>>>>>>>> If you have an existing dogtag 9 instance, and you upgrade to
>>>>>>>>>>>> the new
>>>>>>>>>>>> dogtag 10 and patched IPA, then that instance will continue to
>>>>>>>>>>>> work.
>>>>>>>>>>>> But any new instances would be created using dogtag 10.
>>>>>>>>>>>>
>>>>>>>>>>>>> 3) I had installed IPA with dogtag10 on master. Replica had
>>>>>>>>>>>>> dogtag10 as well
>>>>>>>>>>>>> and I got the following error:
>>>>>>>>>>>>>
>>>>>>>>>>>>> # ipa-ca-install
>>>>>>>>>>>>> /home/mkosek/replica-info-vm-114.idm.lab.bos.redhat.com.gpg
>>>>>>>>>>>>> ...
>>>>>>>>>>>>> Configuring certificate server: Estimated time 3 minutes 30
>>>>>>>>>>>>> seconds
>>>>>>>>>>>>> [1/14]: creating certificate server user
>>>>>>>>>>>>> [2/14]: configuring certificate server instance
>>>>>>>>>>>>>
>>>>>>>>>>>>> Your system may be partly configured.
>>>>>>>>>>>>> Run /usr/sbin/ipa-server-install --uninstall to clean up.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Unexpected error - see /var/log/ipareplica-ca-install.log for
>>>>>>>>>>>>> details:
>>>>>>>>>>>>> IOError: [Errno 2] No such file or directory:
>>>>>>>>>>>>> '/var/lib/pki/pki-tomcat/alias/ca_backup_keys.p12'
>>>>>>>>>>>>>
>>>>>>>>>>>>> Root cause:
>>>>>>>>>>>>> ...
>>>>>>>>>>>>> File
>>>>>>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
>>>>>>>>>>>>> line
>>>>>>>>>>>>> 625, in __spawn_instance
>>>>>>>>>>>>> "/root/cacert.p12")
>>>>>>>>>>>>> ...
>>>>>>>>>>>>>
>>>>>>>>>>>> I need to look into this. I had fixed ipa-replica-install,
>>>>>>>>>>>> rather than
>>>>>>>>>>>> ipa-ca-install to create replicas. I didn't know ipa-ca-install
>>>>>>>>>>>> existed! Should not be too bad to fix though - most likely
>>>>>>>>>>>> just need to
>>>>>>>>>>>> move files to the right place.
>>>>>>>>>>>
>>>>>>>>>>> Ok, thanks! Btw. CA on replica can be either installed during
>>>>>>>>>>> ipa-replica-install time (when --setup-ca option is passed, you
>>>>>>>>>>> probably used
>>>>>>>>>>> that one) and the aforementioned ipa-ca-install run after
>>>>>>>>>>> ipa-replica-install.
>>>>>>>>>>>
>>>>>>>>>> I will be testing this out again. As ipa-ca-install uses the
>>>>>>>>>> same code
>>>>>>>>>> as ipa-replica-install, I would have expected it to work. Did
>>>>>>>>>> you try
>>>>>>>>>> it with selinux in permissive mode?
>>>>>>>>>>
>>>>>>>>> OK -- I tested this out and realized that between the time I
>>>>>>>>> initially
>>>>>>>>> tested this and your tests, we had changed a URL
>>>>>>>>> from /ca/pki/installer/installToken to
>>>>>>>>> /ca/rest/installer/installToken,
>>>>>>>>> but I forgot to update ipa-pki-proxy.conf.
>>>>>>>>>
>>>>>>>>> The new patch has been rebased and changes the relevant patch. With
>>>>>>>>> this, the CA replica installs correctly.
>>>>>>>>
>>>>>>>> Umh, where can I find the new rebased patch? :-)
>>>>>>>>
>>>>>>>>>
>>>>>>>>> There was one issue that I encountered. I have seen this once
>>>>>>>>> before
>>>>>>>>> and will need to investigate further. IPA sometimes restarts the
>>>>>>>>> dogtag
>>>>>>>>> instance by doing systemctl stop pki-tomcatd.target. and then
>>>>>>>>> systemctl
>>>>>>>>> start pki-tomcatd.target. I have noticed that occasionally the
>>>>>>>>> start
>>>>>>>>> invocation fails, while the stop and restart invocations succeed
>>>>>>>>> just
>>>>>>>>> fine. This makes absolutely no sense because restart is essentially
>>>>>>>>> just stop and then start.
>>>>>>>>>
>>>>>>>>> I think this is actually a bug in systemd. If I reboot the
>>>>>>>>> machine, it
>>>>>>>>> all works perfectly. If we can't figure out whats going on with
>>>>>>>>> systemd, we can always replace this start/stop with
>>>>>>>>> starting/stopping
>>>>>>>>> the service directly. (pki-tomcatd at pki-tomcat.service). That
>>>>>>>>> seems to
>>>>>>>>> work all the time with no problems.
>>>>>>>>>
>>>>>>>>> I suggest we leave this fix (if needed) for a separate patch.
>>>>>>>>
>>>>>>>> Ok, I will see if I hit this issue again. Btw. is it recommended
>>>>>>>> in systemd to
>>>>>>>> use target units this way? I thought that only service files are
>>>>>>>> meant to be
>>>>>>>> used for manipulation with a service. As you said yourself, using
>>>>>>>> service unit
>>>>>>>> file for restart worked every time.
>>>>>>>>
>>>>>>>> Martin
>>>>>>>
>>>>>>
>>>>>> Now, I tested the rebased patch with VMs with permissive SELinux.
>>>>>> IPA server
>>>>>> install was OK, as always, but replica installation ended up with
>>>>>> error.
>>>>>> Although it got much further than before:
>>>>>>
>>>>>> # ipa-ca-install
>>>>>> /home/mkosek/replica-info-vm-114.idm.lab.bos.redhat.com.gpg
>>>>>> ...
>>>>>> [3/3]: restarting directory server
>>>>>> done configuring pkids.
>>>>>> Configuring certificate server: Estimated time 3 minutes 30 seconds
>>>>>> [1/14]: creating certificate server user
>>>>>> [2/14]: configuring certificate server instance
>>>>>> [3/14]: disabling nonces
>>>>>> [4/14]: importing CA chain to RA certificate database
>>>>>> [5/14]: fixing RA database permissions
>>>>>> [6/14]: setting up signing cert profile
>>>>>> [7/14]: set up CRL publishing
>>>>>> [8/14]: set certificate subject base
>>>>>> [9/14]: enabling Subject Key Identifier
>>>>>> [10/14]: configuring certificate server to start on boot
>>>>>> [11/14]: configure certmonger for renewals
>>>>>> [12/14]: configure clone certificate renewals
>>>>>> [13/14]: configure Server-Cert certificate renewal
>>>>>> [14/14]: Configure HTTP to proxy connections
>>>>>> done configuring pki-tomcatd.
>>>>>> Restarting the directory and certificate servers
>>>>>>
>>>>>> Your system may be partly configured.
>>>>>> Run /usr/sbin/ipa-server-install --uninstall to clean up.
>>>>>>
>>>>>> It seems like that CA on a replica did not start and so a check for
>>>>>> port 8080
>>>>>> failed. Attached logs (pki-ca-logs.tgz) may provide more information
>>>>>> to this issue.
>>>>>>
>>>>> This is the systemd issue I mentioned before. I will submit a patch
>>>>> which will change the restart mechanism to restart the service and not
>>>>> the target.
>>>>>>
>>>>
>>>> Patch attached. This patch should be applied on top of the large patch
>>>> (being reviewed).
>>>>
>>>>>> Now I am also testing upgrade from IPA+dogtag9 to PatchedIPA+dogtag10
>>>>>> (permissive SELinux). Now, the service was upgraded smoothly, CA
>>>>>> service
>>>>>> could be restarted without failure. However, certificate operations now
>>>>>> did not work:
>>>>>>
>>>>>> # ipactl restart
>>>>>> Restarting Directory Service
>>>>>> Restarting KDC Service
>>>>>> Restarting KPASSWD Service
>>>>>> Restarting MEMCACHE Service
>>>>>> Restarting HTTP Service
>>>>>> Restarting CA Service
>>>>>> # ipactl status
>>>>>> Directory Service: RUNNING
>>>>>> KDC Service: RUNNING
>>>>>> KPASSWD Service: RUNNING
>>>>>> MEMCACHE Service: RUNNING
>>>>>> HTTP Service: RUNNING
>>>>>> CA Service: RUNNING
>>>>>> # ipa cert-show 1
>>>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to
>>>>>> communicate with CMS (Internal Server Error)
>>>>>>
>>>>>> Attaching logs for this case as well (ipa-ca-logs-upgrade.tgz).
>>>>>>
>>>>> The reason for this is that the old dogtag instances are missing:
>>>>> 1. a new parameter in CS.cfg
>>>>> 2. a couple of symlinks
>>>>>
>>>>> I plan to fix this in the dogtag code. Specifically,
>>>>> 1. I will modify the dogtag code to provide a default value in the case
>>>>> where the parameter does not exist.
>>>>> 2. I plan to add a function to the dogtag startup scripts to check and
>>>>> remake any required symlinks.
>>>>>
>>>>> Both of these changes are in the dogtag code, and do not affect this
>>>>> patch. As a workaround, to confirm that the upgrade actually works, do
>>>>> the following manual steps on your upgraded instance:
>>>>>
>>>>> 1. Add the following parameter to CS.cfg
>>>>> pidDir=/var/run
>>>>>
>>>>> 2. Add the following links;
>>>>> cd /var/lib/pki-ca/common/lib
>>>>> ln -s /usr/share/java/apache-commons-codec.jar
>>>>> apache-commons-codec.jar
>>>>> ln -s /usr/share/java/log4j.jar log4j.jar
>>>>>
>>>>> 3 Restart the dogtag instance
>>>>>
>>>>>> HTH,
>>>>>> Martin
>>>>>
>>>>>
>>>
>>> The attached patch conditionalizes the changes, so that IPA supports
>>> both Dogtag versions.
>>>
>>> I didn't put it in platform code for two reasons
>>> - One platform (f17) can have either Dogtag version installed
>>> - I didn't want to conflict with Timo Aaltonen's "platformizing" work.
>>> According to his WIP patches, he plans to move the whole dogtag module
>>> into platform code.
>>>
>>> I used the existence of /usr/bin/pkispawn as a test for Dogtag 10. If
>>> you know a better way, please comment.
>>>
>>> The dogtag_version option is now always added to
>>>
>>> I've reverted the changes to install/ui/test/data/ipa_init.json, so
>>> you'll have to change the ports manually when testing the UI with Dogtag
>>> 10.
>>>
>>>
>>>
>>
>> Attaching a patch with fixed ipa-pki-proxy.conf, as discussed on IRC.
>>
>
> This approach looks good. I did not hit any regression on F17 with dogtag9, or
> clean installs of IPA+dogtag10. I think we are getting close to pushing this code.
>
> The only issues I hit so far are for upgraded dogtag9 instances (on F17):
>
> 1) The service did not start for me. There were some SELinux AVCs and even when
> I disabled SELinux the instance still did not start (logs attached).
>
> 2) Uninstall was also not clean, we leave some dogtag installation states for
> upgraded dogtag9 instance:
>
> # ipa-server-install --uninstall --unattended
> Shutting down all IPA services
> Removing IPA client configuration
> Unconfiguring ntpd
> Unconfiguring CA directory server
> Unconfiguring web server
> Unconfiguring krb5kdc
> Unconfiguring kadmin
> Unconfiguring directory server
> Unconfiguring ipa_memcached
> ipa : ERROR Some installation state for pki-cad has not been
> restored, see /var/lib/ipa/sysrestore/sysrestore.state
>
> /var/lib/ipa/sysrestore/sysrestore.state:
> [pki-cad]
> enabled = False
Ade is working on a new build to address the dogtag upgrade issues.
The left over state should be removed when we drop the separate instance
in the dogtag 9 -> 10 upgrade. I'm not completely sure reading the patch
who actually does this part, is this handled by dogtag?
rob
More information about the Freeipa-devel
mailing list