[Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18

Fri Aug 17 16:06:21 UTC 2012

Ade Lee wrote:
> On Fri, 2012-08-17 at 09:34 -0400, Ade Lee wrote:
>> On Thu, 2012-08-16 at 18:45 +0200, Martin Kosek wrote:
>>> On 08/16/2012 01:28 PM, Ade Lee wrote:
>>>> Patch attached this time.  I should know better than to do this in the
>>>> middle of the night ..
>>>>
>>>> On Thu, 2012-08-16 at 09:12 +0200, Martin Kosek wrote:
>>>>> On 08/16/2012 07:53 AM, Ade Lee wrote:
>>>>>> On Wed, 2012-08-15 at 23:41 -0400, Ade Lee wrote:
>>>>>>> On Wed, 2012-08-15 at 16:34 +0200, Martin Kosek wrote:
>>>>>>>> On 08/15/2012 03:54 PM, Ade Lee wrote:
>>>>>>>>> On Wed, 2012-08-15 at 13:24 +0200, Martin Kosek wrote:
>>>>>>>>>> On 08/08/2012 10:05 PM, Ade Lee wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> Dogtag 10 is being released on f18, and has a number of changes that
>>>>>>>>>>> will affect IPA.  In particular, the following changes will affect
>>>>>>>>>>> current IPA code.
>>>>>>>>>>>
>>>>>>>>>>> * The directory layout of the dogtag instance has changed.  Instead of
>>>>>>>>>>> using separate tomcat instances to host different subsystems, the
>>>>>>>>>>> standard dogtag installation will allow one to install a CA. KRA, OCSP
>>>>>>>>>>> and TKS within the same instance.  There have been corresponding changes
>>>>>>>>>>> in the directory layout, as well as the default instance name
>>>>>>>>>>> (pki-tomcat instead of pki-ca), and startup daemon (pki-tomcatd, instead
>>>>>>>>>>> of pki-cad, pki-krad etc.)
>>>>>>>>>>>
>>>>>>>>>>> * The default instance will use only four ports (HTTPS, HTTP, AJP and
>>>>>>>>>>> tomcat shutdown port) rather than the 6 previously used.  The default
>>>>>>>>>>> ports will be changed to the standard tomcat ports.  As these ports are
>>>>>>>>>>> local to the ipa server machine, this should not cause too much
>>>>>>>>>>> disruption.
>>>>>>>>>>>
>>>>>>>>>>> * There is a new single step installer written in python.
>>>>>>>>>>> (pkispawn/destroy) vs. pkicreate/pkisilent/pkiremove.
>>>>>>>>>>>
>>>>>>>>>>> * Dogtag 10 runs on tomcat7 - with a new corresponding version of
>>>>>>>>>>> tomcatjss.
>>>>>>>>>>>
>>>>>>>>>>> The attached patch integrates all the above changes in IPA installation
>>>>>>>>>>> and maintenance code.  Once the patch is applied, users will be able to:
>>>>>>>>>>>
>>>>>>>>>>> 1. run ipa-server-install to completion on f18 with dogtag 10.
>>>>>>>>>>> 2. install a new replica on f18 on dogtag 10.
>>>>>>>>>>> 3. upgrade an f17 machine with an existing IPA instance to f18/ dogtag
>>>>>>>>>>> 10 - and have that old-style dogtag instance continue to run correctly.
>>>>>>>>>>> This will require the installation of the latest version of tomcatjss as
>>>>>>>>>>> well as the installation of tomcat6.  The old-style instance will
>>>>>>>>>>> continue to use tomcat6.
>>>>>>>>>>> 4. in addition, the new cert renewal code has been patched and should
>>>>>>>>>>> continue to work.
>>>>>>>>>>>
>>>>>>>>>>> What is not yet completed / supported:
>>>>>>>>>>>
>>>>>>>>>>> 1. Installation with an external CA is not yet completed in the new
>>>>>>>>>>> installer.  We plan to complete this soon.
>>>>>>>>>>>
>>>>>>>>>>> 2. There is some IPA upgrade code that has not yet been touched
>>>>>>>>>>> (install/tools/ipa-upgradeconfig).
>>>>>>>>>>>
>>>>>>>>>>> 3. A script needs to be written to allow admins to convert their
>>>>>>>>>>> old-style dogtag instances to new style instances, as well as code to
>>>>>>>>>>> periodically prompt admins to do this.
>>>>>>>>>>>
>>>>>>>>>>> 4. Installation of old-style instances using pkicreate/pkisilent on
>>>>>>>>>>> dogtag 10 will no longer be supported, and will be disabled soon.
>>>>>>>>>>>
>>>>>>>>>>> 5.  The pki-selinux policy has been updated to reflect these changes,
>>>>>>>>>>> but is still in flux.  In fact, it is our intention to place the dogtag
>>>>>>>>>>> selinux policy in the base selinux policy for f18.  In the meantime, it
>>>>>>>>>>> may be necessary to run installs in permissive mode.
>>>>>>>>>>>
>>>>>>>>>>> The dogtag 10 code will be released shortly into f18.  Prior to that
>>>>>>>>>>> though, we have placed the new dogtag 10 and tomcatjss code in a
>>>>>>>>>>> developer repo that is located at
>>>>>>>>>>> http://nkinder.fedorapeople.org/dogtag-devel/
>>>>>>>>>>>
>>>>>>>>>>> Testing can be done on both f18 and f17 - although the target platform -
>>>>>>>>>>> and the only platform for which official builds will be created is f18.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Ade
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi Ade,
>>>>>>>>>>
>>>>>>>>>> Thanks for the patch, I started with review and integration tests (currently
>>>>>>>>>> running on Fedora 17 with Nathan's repo).
>>>>>>>>>>
>>>>>>>>>> Installation on single master was smooth, it worked just fine, even with
>>>>>>>>>> enforced SELinux, without any error - kudos to you and the whole dogtag team.
>>>>>>>>>> The resulting logs and the structure of your log directory seems improved. I
>>>>>>>>>> believe that the brand new Python installers will make it easier to debug
>>>>>>>>>> issues with dogtag installation when they come.  When I tried our unit tests or
>>>>>>>>>> some simple cert operation, it worked fine as well.
>>>>>>>>>>
>>>>>>>>>> Now the bad news, or rather few issues and suggestions I found:
>>>>>>>>>>
>>>>>>>>>> 1) As we already discussed on IRC, tomcat 7 was not pulled automatically on
>>>>>>>>>> Fedora 17 when I updated pki-ca, you somewhere miss a Requires.
>>>>>>>>>>
>>>>>>>>> We have a dogtag patch that is currently in review that will address
>>>>>>>>> this.  Once this is in, tomcatjss >=7.0.0 will be required for f17+,
>>>>>>>>> rather than f18+
>>>>>>>>>
>>>>>>>>>> 2) I had installed IPA with dogtag10 on master. However, CA installation on a
>>>>>>>>>> replica (ipa-ca-install) with dogtag9 failed - is this expectable?
>>>>>>>>>>
>>>>>>>>> Yes.  The current IPA patch is designed to work with dogtag 10 only,
>>>>>>>>> which will be officially available on f18+.  So if you update to dogtag
>>>>>>>>> 10, you must have this patch and visa versa.  We probably need to add
>>>>>>>>> the relevant requires to IPA to enforce this.
>>>>>>>>>
>>>>>>>>> If you have an existing dogtag 9 instance, and you upgrade to the new
>>>>>>>>> dogtag 10 and patched IPA, then that instance will continue to work.
>>>>>>>>> But any new instances would be created using dogtag 10.
>>>>>>>>>
>>>>>>>>>> 3) I had installed IPA with dogtag10 on master. Replica had dogtag10 as well
>>>>>>>>>> and I got the following error:
>>>>>>>>>>
>>>>>>>>>> # ipa-ca-install /home/mkosek/replica-info-vm-114.idm.lab.bos.redhat.com.gpg
>>>>>>>>>> ...
>>>>>>>>>> Configuring certificate server: Estimated time 3 minutes 30 seconds
>>>>>>>>>>    [1/14]: creating certificate server user
>>>>>>>>>>    [2/14]: configuring certificate server instance
>>>>>>>>>>
>>>>>>>>>> Your system may be partly configured.
>>>>>>>>>> Run /usr/sbin/ipa-server-install --uninstall to clean up.
>>>>>>>>>>
>>>>>>>>>> Unexpected error - see /var/log/ipareplica-ca-install.log for details:
>>>>>>>>>> IOError: [Errno 2] No such file or directory:
>>>>>>>>>> '/var/lib/pki/pki-tomcat/alias/ca_backup_keys.p12'
>>>>>>>>>>
>>>>>>>>>> Root cause:
>>>>>>>>>> ...
>>>>>>>>>>    File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line
>>>>>>>>>> 625, in __spawn_instance
>>>>>>>>>>      "/root/cacert.p12")
>>>>>>>>>> ...
>>>>>>>>>>
>>>>>>>>> I need to look into this.  I had fixed ipa-replica-install, rather than
>>>>>>>>> ipa-ca-install to create replicas.  I didn't know ipa-ca-install
>>>>>>>>> existed!  Should not be too bad to fix though - most likely just need to
>>>>>>>>> move files to the right place.
>>>>>>>>
>>>>>>>> Ok, thanks! Btw. CA on replica can be either installed during
>>>>>>>> ipa-replica-install time (when --setup-ca option is passed, you probably used
>>>>>>>> that one) and the aforementioned ipa-ca-install run after ipa-replica-install.
>>>>>>>>
>>>>>>> I will be testing this out again.  As ipa-ca-install uses the same code
>>>>>>> as ipa-replica-install, I would have expected it to work.  Did you try
>>>>>>> it with selinux in permissive mode?
>>>>>>>
>>>>>> OK -- I tested this out and realized that between the time I initially
>>>>>> tested this and your tests, we had changed a URL
>>>>>> from /ca/pki/installer/installToken to /ca/rest/installer/installToken,
>>>>>> but I forgot to update ipa-pki-proxy.conf.
>>>>>>
>>>>>> The new patch has been rebased and changes the relevant patch.  With
>>>>>> this, the CA replica installs correctly.
>>>>>
>>>>> Umh, where can I find the new rebased patch? :-)
>>>>>
>>>>>>
>>>>>> There was one issue that I encountered.  I have seen this once before
>>>>>> and will need to investigate further.  IPA sometimes restarts the dogtag
>>>>>> instance by doing systemctl stop pki-tomcatd.target.  and then systemctl
>>>>>> start pki-tomcatd.target.   I have noticed that occasionally the start
>>>>>> invocation fails, while the stop and restart invocations succeed just
>>>>>> fine.  This makes absolutely no sense because restart is essentially
>>>>>> just stop and then start.
>>>>>>
>>>>>> I think this is actually a bug in systemd.  If I reboot the machine, it
>>>>>> all works perfectly.  If we can't figure out whats going on with
>>>>>> systemd, we can always replace this start/stop with starting/stopping
>>>>>> the service directly.  (pki-tomcatd at pki-tomcat.service).  That seems to
>>>>>> work all the time with no problems.
>>>>>>
>>>>>> I suggest we leave this fix (if needed) for a separate patch.
>>>>>
>>>>> Ok, I will see if I hit this issue again. Btw. is it recommended in systemd to
>>>>> use target units this way? I thought that only service files are meant to be
>>>>> used for manipulation with a service. As you said yourself, using service unit
>>>>> file for restart worked every time.
>>>>>
>>>>> Martin
>>>>
>>>
>>> Now, I tested the rebased patch with VMs with permissive SELinux. IPA server
>>> install was OK, as always, but replica installation ended up with error.
>>> Although it got much further than before:
>>>
>>> # ipa-ca-install /home/mkosek/replica-info-vm-114.idm.lab.bos.redhat.com.gpg
>>> ...
>>>    [3/3]: restarting directory server
>>> done configuring pkids.
>>> Configuring certificate server: Estimated time 3 minutes 30 seconds
>>>    [1/14]: creating certificate server user
>>>    [2/14]: configuring certificate server instance
>>>    [3/14]: disabling nonces
>>>    [4/14]: importing CA chain to RA certificate database
>>>    [5/14]: fixing RA database permissions
>>>    [6/14]: setting up signing cert profile
>>>    [7/14]: set up CRL publishing
>>>    [8/14]: set certificate subject base
>>>    [9/14]: enabling Subject Key Identifier
>>>    [10/14]: configuring certificate server to start on boot
>>>    [11/14]: configure certmonger for renewals
>>>    [12/14]: configure clone certificate renewals
>>>    [13/14]: configure Server-Cert certificate renewal
>>>    [14/14]: Configure HTTP to proxy connections
>>> done configuring pki-tomcatd.
>>> Restarting the directory and certificate servers
>>>
>>> Your system may be partly configured.
>>> Run /usr/sbin/ipa-server-install --uninstall to clean up.
>>>
>>> It seems like that CA on a replica did not start and so a check for port 8080
>>> failed. Attached logs (pki-ca-logs.tgz) may provide more information to this issue.
>>>
>> This is the systemd issue I mentioned before.  I will submit a patch
>> which will change the restart mechanism to restart the service and not
>> the target.
>>>
>
> Patch attached.  This patch should be applied on top of the large patch
> (being reviewed).
>
>>> Now I am also testing upgrade from IPA+dogtag9 to PatchedIPA+dogtag10
>>> (permissive SELinux). Now, the service was upgraded smoothly, CA service
>>> could be restarted without failure. However, certificate operations now
>>> did not work:
>>>
>>> # ipactl restart
>>> Restarting Directory Service
>>> Restarting KDC Service
>>> Restarting KPASSWD Service
>>> Restarting MEMCACHE Service
>>> Restarting HTTP Service
>>> Restarting CA Service
>>> # ipactl status
>>> Directory Service: RUNNING
>>> KDC Service: RUNNING
>>> KPASSWD Service: RUNNING
>>> MEMCACHE Service: RUNNING
>>> HTTP Service: RUNNING
>>> CA Service: RUNNING
>>> # ipa cert-show 1
>>> ipa: ERROR: Certificate operation cannot be completed: Unable to
>>> communicate with CMS (Internal Server Error)
>>>
>>> Attaching logs for this case as well (ipa-ca-logs-upgrade.tgz).
>>>
>> The reason for this is that the old dogtag instances are missing:
>> 1. a new parameter in CS.cfg
>> 2. a couple of symlinks
>>
>> I plan to fix this in the dogtag code.  Specifically,
>> 1. I will modify the dogtag code to provide a default value in the case
>> where the parameter does not exist.
>> 2. I plan to add a function to the dogtag startup scripts to check and
>> remake any required symlinks.
>>
>> Both of these changes are in the dogtag code, and do not affect this
>> patch.  As a workaround, to confirm that the upgrade actually works, do
>> the following manual steps on your upgraded instance:
>>
>> 1. Add the following parameter to CS.cfg
>>     pidDir=/var/run
>>
>> 2. Add the following links;
>>     cd /var/lib/pki-ca/common/lib
>>     ln -s /usr/share/java/apache-commons-codec.jar apache-commons-codec.jar
>>     ln -s /usr/share/java/log4j.jar log4j.jar
>>
>> 3 Restart the dogtag instance
>

To throw a little twist in here, we should make sure that the certmonger 
restart scripts still work as well.

rob