[Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18

Wed Sep 12 16:40:52 UTC 2012

A new Dogtag build with changed pkispawn/pkidestroy locations should be 
out later today. The attached patch should work with that build.

On 09/11/2012 08:45 PM, Rob Crittenden wrote:
> Petr Viktorin wrote:
>> On 09/11/2012 04:38 PM, Rob Crittenden wrote:
>>> Ade Lee wrote:
>>>> On Tue, 2012-09-11 at 08:59 -0400, Rob Crittenden wrote:
>>>>> Petr Viktorin wrote:
>>>>>> On 09/11/2012 04:04 AM, Ade Lee wrote:
>>>>>>> On Mon, 2012-09-10 at 16:58 -0400, Rob Crittenden wrote:
>>>>>>>> Petr Viktorin wrote:
>>>>>>>>> Attaching rebased and squashed patches. I've done some testing
>>>>>>>>> with
>>>>>>>>> them
>>>>>>>>> but please test some more.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Most of these aren't IPA issues, but dogtag issues. I'll try to
>>>>>>>> split
>>>>>>>> them out.
>>>>>>>>
>>>>>>>> IPA:
>>>>>>>>
>>>>>>>> For the configuration files in install/conf to be updated at rpm
>>>>>>>> update
>>>>>>>> time the VERSION needs to be incremented.
>>>>>>
>>>>>> These files should stay the same since on upgrade we're still using a
>>>>>> Dogtag 9 style instance. The Dogtag 10 ports are only used in new
>>>>>> installs.
>>>>>>
>>>>>>>> The ipa package lacks any updated dogtag dependencies, so I abused
>>>>>>>> it.
>>>>>>
>>>>>> What should the updated dependencies be? Since it should work with
>>>>>> both
>>>>>> dogtag 9 and 10, I don't see how they should change.
>>>>>
>>>>> I don't know either, but we need to prevent people from installing
>>>>> incompatible package combinations.
>>>>>
>>>> Would'nt the Conflicts: ipa < 3.0 in pki-ca mentioned below satisfy
>>>> this
>>>> requirement?  The main concern is that you must have ipa 3.0 if you
>>>> have
>>>> dogtag 10.
>>>>
>>>> Given that dogtag is consumed by IPA though, it makes more sense to put
>>>> the relevant conflicts in IPA rather than in dogtag.  So in this case,
>>>> that would mean putting Conflicts: pki-ca >= 10.0 in IPA 2.x.
>>>> Recall that dogtag 10 will only be officially available in f18+.
>>>
>>> That isn't enough. If a F-17 user with IPA 2.2 installed upgrades to
>>> F-18 they would be able to install dogtag 10 and blow up their IPA
>>> server.
>>>
>>>>
>>>>>>>> I installed IPA with dogtag 9 and created a replica.
>>>>>>>>
>>>>>>>> I updated the IPA bits, that worked fine.
>>>>>>>>
>>>>>>>> I updated to dogtag 10 and now the CA doesn't work on the master,
>>>>>>>> including starting the dogtag instance. Note that the rpm update
>>>>>>>> process
>>>>>>>> worked, no notice that the CA service didn't restart.
>>>>>>>>
>>>>>>> Did you try to restart the CA with selinux in permissive mode?
>>>>>>> This is
>>>>>>> still required right now until I get the selinux policy all
>>>>>>> straightened
>>>>>>> out.
>>>>>>>
>>>>>>> There is also a separate dogtag ticket (which is currently being
>>>>>>> worked
>>>>>>> on) to restart existing dogtag instances when dogtag is upgraded
>>>>>>> from
>>>>>>> 9->10.
>>>>>>
>>>>>> In permissive mode, this upgrade works for me.
>>>>>
>>>>> I was in enforcing mode but saw no AVCs. What is the ETA on fixing
>>>>> this?
>>>>>
>>>>
>>>> Within the next week or two, I need to finish the IPA merge database
>>>> patch first, and then co-ordinate changes with the selinux guys.
>>>>
>>>>>>
>>>>>>
>>>>>> Sometimes I do get this error intermittently:
>>>>>>
>>>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to
>>>>>> communicate with CMS (Service Temporarily Unavailable)
>>>>>>
>>>>>> Usually, waiting a couple of minutes clears this up. Perhaps we are
>>>>>> not
>>>>>> doing startup detection correctly. Ade mentioned that waiting for
>>>>>> ports
>>>>>> may not be ideal. How do we know if Dogtag is initialized?
>>>>>
>>>>> Years ago we had discussed with Andrew and Matt creating a URI that
>>>>> can
>>>>> be queried to determine dogtag status. I don't think that ever went
>>>>> anywhere.
>>>>>
>>>> Petr, this happens only on reboot, right?  And not on regular "service
>>>> ipa restart"?
>>
>> I've now seen it happen right after a 9 → 10 upgrade.
>>
>>>> Yeah, I remember this conversation - and even created a bug for it at
>>>> some point.  This went away because the mechanism you were using seemed
>>>> to be working.  The timing may be very different now with tomcat 7 and
>>>> systemd.  I'll open a dogtag trac ticket for this.
>>>
>>> Ok.
>>>
>>>>
>>>>>>
>>>>>>>> Uninstalling failed because it tried to run pkidestroy and not
>>>>>>>> pkiremove.
>>>>>>
>>>>>> I was under the impression that pkidestroy was the correct command to
>>>>>> remove an upgraded instance. I'll check with Ade.
>>>>>>
>>>>>>> I'll test this too.
>>>>>>>
>>>>>>>> The contents of the file passed to pkispawn should be logged so we
>>>>>>>> can
>>>>>>>> see exactly what was passed in.
>>>>>>>>
>>>>>>> Its a pretty big file.  You might want to only log the
>>>>>>> modifications.
>>>>>>> Or save the file somewhere.
>>>>>>
>>>>>> Our logs are pretty verbose, so that shouldn't be a problem. I'll
>>>>>> put it
>>>>>> in the next version of the patch.
>>>>>
>>>>> The question to ask is: would you need the contents of this file if
>>>>> all
>>>>> you got were logs and needed to evaluate why installation failed? In
>>>>> most cases this is yes.
>>>>>
>>>> Up to you guys.  There is a patch I am working on in which I will be
>>>> logging the object that is being passed to the server from pkispawn.
>>>> That - and the diffs to the standard config file as I mentioned above -
>>>> will likely be sufficient to debug almost all cases.
>>>>
>>>> Make sure not to log any passwords.
>>>>
>>
>> Thanks for the catch. Attaching updated patch that sanitizes the
>> passwords.
>>
>>>>>>>> DOGTAG:
>>>>>>>>
>>>>>>>> When upgrading using the dogtag-devel repo I had to specify
>>>>>>>> pki-tools.x86_64 otherwise it tried to install both 32 and 64-bit
>>>>>>>> versions (and failed).
>>>>>>>>
>>>>>>>> I ended up running: yum update pki-ca tomcatjss pki-tools.x86_64
>>>>>>>> --enablerepo=dogtag-devel --enablerepo=updates-testing
>>>>>>>>
>>>>>>> We'll look into this.  I think I know why this happens.
>>>>>>>
>>>>>>>> What happens if someone manually upgrades pki-ca without first
>>>>>>>> updating
>>>>>>>> ipa? I think that pki-ca is going to need a Conflicts ipa < 3.0 in
>>>>>>>> it.
>>>>>>>
>>>>>>> We can add that.
>>>>>>>
>>>>>>>> certificate renewal failed. I spent far too long trying to figure
>>>>>>>> out
>>>>>>>> why tomcat wasn't listening on port 9180 but failed. I think
>>>>>>>> 9180 is
>>>>>>>> actually the old server, right? So another missing dependency on a
>>>>>>>> fixed
>>>>>>>> certmonger?
>>>>>>>>
>>>>>>>> The best I could find was the certmonger error:
>>>>>>>>
>>>>>>>> ca-error: Error 7 connecting to
>>>>>>>> http://edsel.example.com:9180/ca/ee/ca/profileSubmit: Couldn't
>>>>>>>> connect
>>>>>>>> to server.
>>>>>>>>
>>>>>>
>>>>>> I'll set the certmonger min version to 0.60 as per Nalin's mail.
>>>>>
>>>>> Ok.
>>>>>
>>>>>>> Is this cert renewal on a dogtag 10 instance?  Or the upgraded
>>>>>>> dogtag 9?
>>>>>>> If its dogtag 10, perhaps you do not have the certmonger version
>>>>>>> that
>>>>>>> has the relevant fix?  If its dogtag 9, then we need to figure out
>>>>>>> whats
>>>>>>> going on.  That reminds me - I need to file a bug to allow
>>>>>>> certmonger to
>>>>>>> talk to the newly defined dogtag ports.  Do you have selinux
>>>>>>> permissive?
>>>>>>>
>>>>>>>> There is no man page for pkispawn/pkidestroy :-( According to the
>>>>>>>> FHS
>>>>>>>> these should not be in /bin but in /usr/sbin (not end-user
>>>>>>>> commands).
>>>>>>>>
>>>>>>> There is a trac ticket open for man pages for pkispawn and
>>>>>>> pkidestroy.
>>>>>>> We plan to complete this ticket by the time f18 is released.
>>>>>>>
>>>>>>> We'll look into the location of pkispawn/pkicreate.
>>>>>>>
>>>>>>>> The output of pkicreate/pkisilent was really terrible and not
>>>>>>>> usable at
>>>>>>>> all so we didn't display it when failures occurred. It looks like
>>>>>>>> that
>>>>>>>> has been addressed, at least for the case where a CA is already
>>>>>>>> configured and you try to install IPA. Perhaps we should capture
>>>>>>>> stderr
>>>>>>>> and display that instead of the command-line of pkispawn? Again, a
>>>>>>>> man
>>>>>>>> page would help with the integration.
>>>>>>>>
>>>>>>>> 2012-09-10T20:51:45Z DEBUG   [2/18]: configuring certificate server
>>>>>>>> instance
>>>>>>>> 2012-09-10T20:51:45Z DEBUG args=/bin/pkispawn -s CA -f
>>>>>>>> /tmp/tmp_Urraq
>>>>>>>> 2012-09-10T20:51:45Z DEBUG stdout=
>>>>>>>> 2012-09-10T20:51:45Z DEBUG stderr=pkispawn    : ERROR    .......
>>>>>>>> PKI
>>>>>>>> subsystem 'CA' for instance 'pki-tomcat' already exists!
>>>>>>>>
>>>>>>>> 2012-09-10T20:51:45Z CRITICAL failed to configure ca instance
>>>>>>>> Command
>>>>>>>> '/bin/pkispawn -s CA -f /tmp/tmp_Urraq' returned non-zero exit
>>>>>>>> status 1
>>>>>>>>
>>>>>>> That may be a good idea.  Of course. thats an IPA thing, right?
>>>>>
>>>>> Right, not a show-stopper for this but a new ticket should be
>>>>> opened to
>>>>> track it.
>>
>> https://fedorahosted.org/freeipa/ticket/3072
>
> Thanks.
>
> I tested a 2.2 -> 3.0 upgrade with dogtag 9 and it seems to have failed
> to restart something:
>
> [ post rpm -Uvh dist/rpms/freeipa*.rpm ]
>
> [root at edsel freeipa]# ipa cert-show 1
> ipa: ERROR: Certificate operation cannot be completed: Unable to
> communicate with CMS (Service Temporarily Unavailable)

I'm getting this error as well. systemctl shows that 
pki-cad at pki-ca.service as failed. But, after a ipactl restart, the 
service starts correctly and `ipa cert-show` works again (in permissive 
mode).

I didn't have time to finish investigating this today but I'll work on 
it more tomorrow.

> [root at edsel freeipa]# ipactl restart
> Restarting Directory Service
> Restarting KDC Service
> Restarting KPASSWD Service
> Restarting MEMCACHE Service
> Restarting HTTP Service
> Restarting CA Service
> [root at edsel freeipa]# ipa cert-show 1
>    Certificate: MIIDjTCCAnWgAwIBAgIBATANBgk
>    ...
>
> The apache log had this:
>
> [Tue Sep 11 14:35:42 2012] [error] (111)Connection refused: proxy: AJP:
> attempt to connect to 127.0.0.1:9447 (localhost) failed
> [Tue Sep 11 14:35:42 2012] [error] ap_proxy_connect_backend disabling
> worker for (localhost)
> [Tue Sep 11 14:35:42 2012] [error] proxy: AJP: failed to make connection
> to backend: localhost
>
> So I'm gathering that dogtag didn't restart properly, but I'm just
> guessing. It could have been the PKI-IPA ds instance too, I'm not sure
> where to check.
>
> I also noticed this in /var/log/ipaupgrade.log:
>
> 2012-09-11T18:28:22Z DEBUG args=/bin/systemctl start certmonger.service
> 2012-09-11T18:28:22Z DEBUG stdout=
> 2012-09-11T18:28:22Z DEBUG stderr=
> 2012-09-11T18:28:22Z DEBUG args=/bin/systemctl is-active certmonger.service
> 2012-09-11T18:28:22Z DEBUG stdout=active
> ...
> ... [ snip certutil output ]
> ...
> 2012-09-11T18:28:52Z DEBUG args=/usr/bin/getcert start-tracking -d
> /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c
> dogtag-ipa-renew-agent -C /usr/lib64/ipa/certmonger/renew_ca_cert
> "auditSigningCert cert-pki-ca" -P XXXXXXXX
> 2012-09-11T18:28:52Z DEBUG stdout=Please verify that the certmonger
> service is still running.
>
> 2012-09-11T18:28:52Z DEBUG stderr=
> 2012-09-11T18:28:52Z INFO   File
> "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py",
> line 614, in run_script
>      return_value = main_function()
>
>    File "/usr/sbin/ipa-upgradeconfig", line 540, in main
>      enable_certificate_renewal(api.env.realm)
>
>    File "/usr/sbin/ipa-upgradeconfig", line 455, in
> enable_certificate_renewal
>      ca.configure_renewal()
>
>    File
> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line
> 1298, in configure_renewal
>      self.dogtag_constants.ALIAS_DIR, 'renew_ca_cert "%s"' % nickname)
>
>    File "/usr/lib/python2.7/site-packages/ipapython/certmonger.py", line
> 394, in dogtag_start_tracking
>      (stdout, stderr, returncode) = ipautil.run(args, nolog=[pin])
>
>    File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line
> 309, in run
>      raise CalledProcessError(p.returncode, args)
>
> 2012-09-11T18:28:52Z INFO The ipa-upgradeconfig command failed,
> exception: CalledProcessError: Command '/usr/bin/getcert start-tracking
> -d /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c
> dogtag-ipa-renew-agent -C /usr/lib64/ipa/certmonger/renew_ca_cert
> "auditSigningCert cert-pki-ca" -P XXXXXXXX' returned non-zero exit status 1
>
> I'm not sure if this is related to this patch or not. If it isn't can
> you file a new ticket on it, or add it the 2.2 -> 3.0 upgrade ticket?

I've opened a ticket.

-- 
Petr³
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pviktori-0074-07-Use-Dogtag-10-only-when-it-is-available.patch
Type: text/x-patch
Size: 59913 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20120912/4dddcb1e/attachment.bin>