[Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18
Rob Crittenden
rcritten at redhat.com
Tue Sep 11 19:20:24 UTC 2012
Rob Crittenden wrote:
> Petr Viktorin wrote:
>> On 09/11/2012 04:38 PM, Rob Crittenden wrote:
>>> Ade Lee wrote:
>>>> On Tue, 2012-09-11 at 08:59 -0400, Rob Crittenden wrote:
>>>>> Petr Viktorin wrote:
>>>>>> On 09/11/2012 04:04 AM, Ade Lee wrote:
>>>>>>> On Mon, 2012-09-10 at 16:58 -0400, Rob Crittenden wrote:
>>>>>>>> Petr Viktorin wrote:
>>>>>>>>> Attaching rebased and squashed patches. I've done some testing
>>>>>>>>> with
>>>>>>>>> them
>>>>>>>>> but please test some more.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Most of these aren't IPA issues, but dogtag issues. I'll try to
>>>>>>>> split
>>>>>>>> them out.
>>>>>>>>
>>>>>>>> IPA:
>>>>>>>>
>>>>>>>> For the configuration files in install/conf to be updated at rpm
>>>>>>>> update
>>>>>>>> time the VERSION needs to be incremented.
>>>>>>
>>>>>> These files should stay the same since on upgrade we're still using a
>>>>>> Dogtag 9 style instance. The Dogtag 10 ports are only used in new
>>>>>> installs.
>>>>>>
>>>>>>>> The ipa package lacks any updated dogtag dependencies, so I abused
>>>>>>>> it.
>>>>>>
>>>>>> What should the updated dependencies be? Since it should work with
>>>>>> both
>>>>>> dogtag 9 and 10, I don't see how they should change.
>>>>>
>>>>> I don't know either, but we need to prevent people from installing
>>>>> incompatible package combinations.
>>>>>
>>>> Would'nt the Conflicts: ipa < 3.0 in pki-ca mentioned below satisfy
>>>> this
>>>> requirement? The main concern is that you must have ipa 3.0 if you
>>>> have
>>>> dogtag 10.
>>>>
>>>> Given that dogtag is consumed by IPA though, it makes more sense to put
>>>> the relevant conflicts in IPA rather than in dogtag. So in this case,
>>>> that would mean putting Conflicts: pki-ca >= 10.0 in IPA 2.x.
>>>> Recall that dogtag 10 will only be officially available in f18+.
>>>
>>> That isn't enough. If a F-17 user with IPA 2.2 installed upgrades to
>>> F-18 they would be able to install dogtag 10 and blow up their IPA
>>> server.
>>>
>>>>
>>>>>>>> I installed IPA with dogtag 9 and created a replica.
>>>>>>>>
>>>>>>>> I updated the IPA bits, that worked fine.
>>>>>>>>
>>>>>>>> I updated to dogtag 10 and now the CA doesn't work on the master,
>>>>>>>> including starting the dogtag instance. Note that the rpm update
>>>>>>>> process
>>>>>>>> worked, no notice that the CA service didn't restart.
>>>>>>>>
>>>>>>> Did you try to restart the CA with selinux in permissive mode?
>>>>>>> This is
>>>>>>> still required right now until I get the selinux policy all
>>>>>>> straightened
>>>>>>> out.
>>>>>>>
>>>>>>> There is also a separate dogtag ticket (which is currently being
>>>>>>> worked
>>>>>>> on) to restart existing dogtag instances when dogtag is upgraded
>>>>>>> from
>>>>>>> 9->10.
>>>>>>
>>>>>> In permissive mode, this upgrade works for me.
>>>>>
>>>>> I was in enforcing mode but saw no AVCs. What is the ETA on fixing
>>>>> this?
>>>>>
>>>>
>>>> Within the next week or two, I need to finish the IPA merge database
>>>> patch first, and then co-ordinate changes with the selinux guys.
>>>>
>>>>>>
>>>>>>
>>>>>> Sometimes I do get this error intermittently:
>>>>>>
>>>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to
>>>>>> communicate with CMS (Service Temporarily Unavailable)
>>>>>>
>>>>>> Usually, waiting a couple of minutes clears this up. Perhaps we are
>>>>>> not
>>>>>> doing startup detection correctly. Ade mentioned that waiting for
>>>>>> ports
>>>>>> may not be ideal. How do we know if Dogtag is initialized?
>>>>>
>>>>> Years ago we had discussed with Andrew and Matt creating a URI that
>>>>> can
>>>>> be queried to determine dogtag status. I don't think that ever went
>>>>> anywhere.
>>>>>
>>>> Petr, this happens only on reboot, right? And not on regular "service
>>>> ipa restart"?
>>
>> I've now seen it happen right after a 9 → 10 upgrade.
>>
>>>> Yeah, I remember this conversation - and even created a bug for it at
>>>> some point. This went away because the mechanism you were using seemed
>>>> to be working. The timing may be very different now with tomcat 7 and
>>>> systemd. I'll open a dogtag trac ticket for this.
>>>
>>> Ok.
>>>
>>>>
>>>>>>
>>>>>>>> Uninstalling failed because it tried to run pkidestroy and not
>>>>>>>> pkiremove.
>>>>>>
>>>>>> I was under the impression that pkidestroy was the correct command to
>>>>>> remove an upgraded instance. I'll check with Ade.
>>>>>>
>>>>>>> I'll test this too.
>>>>>>>
>>>>>>>> The contents of the file passed to pkispawn should be logged so we
>>>>>>>> can
>>>>>>>> see exactly what was passed in.
>>>>>>>>
>>>>>>> Its a pretty big file. You might want to only log the
>>>>>>> modifications.
>>>>>>> Or save the file somewhere.
>>>>>>
>>>>>> Our logs are pretty verbose, so that shouldn't be a problem. I'll
>>>>>> put it
>>>>>> in the next version of the patch.
>>>>>
>>>>> The question to ask is: would you need the contents of this file if
>>>>> all
>>>>> you got were logs and needed to evaluate why installation failed? In
>>>>> most cases this is yes.
>>>>>
>>>> Up to you guys. There is a patch I am working on in which I will be
>>>> logging the object that is being passed to the server from pkispawn.
>>>> That - and the diffs to the standard config file as I mentioned above -
>>>> will likely be sufficient to debug almost all cases.
>>>>
>>>> Make sure not to log any passwords.
>>>>
>>
>> Thanks for the catch. Attaching updated patch that sanitizes the
>> passwords.
>>
>>>>>>>> DOGTAG:
>>>>>>>>
>>>>>>>> When upgrading using the dogtag-devel repo I had to specify
>>>>>>>> pki-tools.x86_64 otherwise it tried to install both 32 and 64-bit
>>>>>>>> versions (and failed).
>>>>>>>>
>>>>>>>> I ended up running: yum update pki-ca tomcatjss pki-tools.x86_64
>>>>>>>> --enablerepo=dogtag-devel --enablerepo=updates-testing
>>>>>>>>
>>>>>>> We'll look into this. I think I know why this happens.
>>>>>>>
>>>>>>>> What happens if someone manually upgrades pki-ca without first
>>>>>>>> updating
>>>>>>>> ipa? I think that pki-ca is going to need a Conflicts ipa < 3.0 in
>>>>>>>> it.
>>>>>>>
>>>>>>> We can add that.
>>>>>>>
>>>>>>>> certificate renewal failed. I spent far too long trying to figure
>>>>>>>> out
>>>>>>>> why tomcat wasn't listening on port 9180 but failed. I think
>>>>>>>> 9180 is
>>>>>>>> actually the old server, right? So another missing dependency on a
>>>>>>>> fixed
>>>>>>>> certmonger?
>>>>>>>>
>>>>>>>> The best I could find was the certmonger error:
>>>>>>>>
>>>>>>>> ca-error: Error 7 connecting to
>>>>>>>> http://edsel.example.com:9180/ca/ee/ca/profileSubmit: Couldn't
>>>>>>>> connect
>>>>>>>> to server.
>>>>>>>>
>>>>>>
>>>>>> I'll set the certmonger min version to 0.60 as per Nalin's mail.
>>>>>
>>>>> Ok.
>>>>>
>>>>>>> Is this cert renewal on a dogtag 10 instance? Or the upgraded
>>>>>>> dogtag 9?
>>>>>>> If its dogtag 10, perhaps you do not have the certmonger version
>>>>>>> that
>>>>>>> has the relevant fix? If its dogtag 9, then we need to figure out
>>>>>>> whats
>>>>>>> going on. That reminds me - I need to file a bug to allow
>>>>>>> certmonger to
>>>>>>> talk to the newly defined dogtag ports. Do you have selinux
>>>>>>> permissive?
>>>>>>>
>>>>>>>> There is no man page for pkispawn/pkidestroy :-( According to the
>>>>>>>> FHS
>>>>>>>> these should not be in /bin but in /usr/sbin (not end-user
>>>>>>>> commands).
>>>>>>>>
>>>>>>> There is a trac ticket open for man pages for pkispawn and
>>>>>>> pkidestroy.
>>>>>>> We plan to complete this ticket by the time f18 is released.
>>>>>>>
>>>>>>> We'll look into the location of pkispawn/pkicreate.
>>>>>>>
>>>>>>>> The output of pkicreate/pkisilent was really terrible and not
>>>>>>>> usable at
>>>>>>>> all so we didn't display it when failures occurred. It looks like
>>>>>>>> that
>>>>>>>> has been addressed, at least for the case where a CA is already
>>>>>>>> configured and you try to install IPA. Perhaps we should capture
>>>>>>>> stderr
>>>>>>>> and display that instead of the command-line of pkispawn? Again, a
>>>>>>>> man
>>>>>>>> page would help with the integration.
>>>>>>>>
>>>>>>>> 2012-09-10T20:51:45Z DEBUG [2/18]: configuring certificate server
>>>>>>>> instance
>>>>>>>> 2012-09-10T20:51:45Z DEBUG args=/bin/pkispawn -s CA -f
>>>>>>>> /tmp/tmp_Urraq
>>>>>>>> 2012-09-10T20:51:45Z DEBUG stdout=
>>>>>>>> 2012-09-10T20:51:45Z DEBUG stderr=pkispawn : ERROR .......
>>>>>>>> PKI
>>>>>>>> subsystem 'CA' for instance 'pki-tomcat' already exists!
>>>>>>>>
>>>>>>>> 2012-09-10T20:51:45Z CRITICAL failed to configure ca instance
>>>>>>>> Command
>>>>>>>> '/bin/pkispawn -s CA -f /tmp/tmp_Urraq' returned non-zero exit
>>>>>>>> status 1
>>>>>>>>
>>>>>>> That may be a good idea. Of course. thats an IPA thing, right?
>>>>>
>>>>> Right, not a show-stopper for this but a new ticket should be
>>>>> opened to
>>>>> track it.
>>
>> https://fedorahosted.org/freeipa/ticket/3072
>
> Thanks.
>
> I tested a 2.2 -> 3.0 upgrade with dogtag 9 and it seems to have failed
> to restart something:
>
> [ post rpm -Uvh dist/rpms/freeipa*.rpm ]
>
> [root at edsel freeipa]# ipa cert-show 1
> ipa: ERROR: Certificate operation cannot be completed: Unable to
> communicate with CMS (Service Temporarily Unavailable)
> [root at edsel freeipa]# ipactl restart
> Restarting Directory Service
> Restarting KDC Service
> Restarting KPASSWD Service
> Restarting MEMCACHE Service
> Restarting HTTP Service
> Restarting CA Service
> [root at edsel freeipa]# ipa cert-show 1
> Certificate: MIIDjTCCAnWgAwIBAgIBATANBgk
> ...
>
> The apache log had this:
>
> [Tue Sep 11 14:35:42 2012] [error] (111)Connection refused: proxy: AJP:
> attempt to connect to 127.0.0.1:9447 (localhost) failed
> [Tue Sep 11 14:35:42 2012] [error] ap_proxy_connect_backend disabling
> worker for (localhost)
> [Tue Sep 11 14:35:42 2012] [error] proxy: AJP: failed to make connection
> to backend: localhost
>
> So I'm gathering that dogtag didn't restart properly, but I'm just
> guessing. It could have been the PKI-IPA ds instance too, I'm not sure
> where to check.
>
> I also noticed this in /var/log/ipaupgrade.log:
>
> 2012-09-11T18:28:22Z DEBUG args=/bin/systemctl start certmonger.service
> 2012-09-11T18:28:22Z DEBUG stdout=
> 2012-09-11T18:28:22Z DEBUG stderr=
> 2012-09-11T18:28:22Z DEBUG args=/bin/systemctl is-active certmonger.service
> 2012-09-11T18:28:22Z DEBUG stdout=active
> ...
> ... [ snip certutil output ]
> ...
> 2012-09-11T18:28:52Z DEBUG args=/usr/bin/getcert start-tracking -d
> /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c
> dogtag-ipa-renew-agent -C /usr/lib64/ipa/certmonger/renew_ca_cert
> "auditSigningCert cert-pki-ca" -P XXXXXXXX
> 2012-09-11T18:28:52Z DEBUG stdout=Please verify that the certmonger
> service is still running.
>
> 2012-09-11T18:28:52Z DEBUG stderr=
> 2012-09-11T18:28:52Z INFO File
> "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py",
> line 614, in run_script
> return_value = main_function()
>
> File "/usr/sbin/ipa-upgradeconfig", line 540, in main
> enable_certificate_renewal(api.env.realm)
>
> File "/usr/sbin/ipa-upgradeconfig", line 455, in
> enable_certificate_renewal
> ca.configure_renewal()
>
> File
> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line
> 1298, in configure_renewal
> self.dogtag_constants.ALIAS_DIR, 'renew_ca_cert "%s"' % nickname)
>
> File "/usr/lib/python2.7/site-packages/ipapython/certmonger.py", line
> 394, in dogtag_start_tracking
> (stdout, stderr, returncode) = ipautil.run(args, nolog=[pin])
>
> File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line
> 309, in run
> raise CalledProcessError(p.returncode, args)
>
> 2012-09-11T18:28:52Z INFO The ipa-upgradeconfig command failed,
> exception: CalledProcessError: Command '/usr/bin/getcert start-tracking
> -d /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c
> dogtag-ipa-renew-agent -C /usr/lib64/ipa/certmonger/renew_ca_cert
> "auditSigningCert cert-pki-ca" -P XXXXXXXX' returned non-zero exit status 1
>
> I'm not sure if this is related to this patch or not. If it isn't can
> you file a new ticket on it, or add it the 2.2 -> 3.0 upgrade ticket?
These are both easily reproducable. Rather than restarting the world
this time I looked to see what was running:
# ipactl status
Directory Service: RUNNING
KDC Service: RUNNING
KPASSWD Service: RUNNING
MEMCACHE Service: RUNNING
HTTP Service: RUNNING
CA Service: RUNNING
# ps -ef|grep java
<nada>
# /bin/systemctl -a status pki-cad at pki-ca.service
pki-cad at pki-ca.service - PKI Certificate Authority Server pki-ca
Loaded: loaded (/usr/lib/systemd/system/pki-cad at .service;
enabled)
Active: failed (Result: exit-code) since Tue, 11 Sep 2012
15:10:35 -0400; 8min ago
Process: 5917 ExecStop=/usr/bin/pkicontrol stop ca %i
(code=exited, status=1/FAILURE)
Process: 5763 ExecStart=/usr/bin/pkicontrol start ca %i
(code=exited, status=0/SUCCESS)
Main PID: 5830 (code=exited, status=0/SUCCESS)
CGroup: name=systemd:/system/pki-cad at .service/pki-ca
Sep 11 15:10:27 edsel.greyoak.com runuser[5792]:
pam_unix(runuser-l:session): session opened for user pkiuser by (uid=0)
Sep 11 15:10:27 edsel.greyoak.com runuser[5792]:
pam_unix(runuser-l:session): session closed for user pkiuser
Sep 11 15:10:35 edsel.greyoak.com runuser[5936]:
pam_unix(runuser-l:session): session opened for user pkiuser by (uid=0)
Sep 11 15:10:35 edsel.greyoak.com runuser[5936]:
pam_unix(runuser-l:session): session closed for user pkiuser
So I'm thinking that IPA believes that the CA is running, and systemd
reported it started, but something went wrong.
The catalina.out log doesn't really show a failure, just the stupid "I
can't stop because I'm already stopped" traceback. I wonder if this is
an issue with pkicontrol where restart doesn't handle a stop failure,
but I'm just guessing.
I suspect that the start-tracking issue is not related.
rob
More information about the Freeipa-devel
mailing list