[Freeipa-users] Dogtag not working?

Tue Dec 3 17:45:24 UTC 2013

Erinn Looney-Triggs wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 12/02/2013 10:18 AM, Rob Crittenden wrote:
>> Erinn Looney-Triggs wrote:
>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
>>>
>>> On 12/02/2013 08:03 AM, Rob Crittenden wrote:
>>>> Erinn Looney-Triggs wrote:
>>>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
>>>>>
>>>>> On 12/02/2013 07:40 AM, Rob Crittenden wrote:
>>>>>> Erinn Looney-Triggs wrote:
>>>>>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
>>>>>>>
>>>>>>> On 11/28/2013 03:50 PM, Erinn Looney-Triggs wrote:
>>>>>>>> In the process of prepping a replication host for
>>>>>>>> changing over the CA I had to use certmonger to
>>>>>>>> generate another certificate on my secondary IPA
>>>>>>>> server. Unfortunately it seems to fail every single
>>>>>>>> time. Here is what I am running and here is what I am
>>>>>>>> getting:
>>>>>>>>
>>>>>>>> ipa-getcert request -k private/ipa2.abaqis.com.key -f
>>>>>>>> certs/ipa2.abaqis.com.crt -g 2048
>>>>>>>>
>>>>>>>> The request appears to work, however when checking the
>>>>>>>> list I receive the following:
>>>>>>>>
>>>>>>>> ipa-getcert list -r Number of certificates and
>>>>>>>> requests being tracked: 9. Request ID '20131128202128':
>>>>>>>> status: CA_UNREACHABLE ca-error: Server failed request,
>>>>>>>> will retry: 4301 (RPC failed at server.  Certificate
>>>>>>>> operation cannot be completed: FAILURE (Authentication
>>>>>>>> Error)). stuck: yes key pair storage:
>>>>>>>> type=FILE,location='/etc/pki/tls/private/ipa2.abaqis.com.key'
>>>>>>>>
>>>>>>>>
>>>
>>>>>>>>
> certificate:
>>>>>>>> type=FILE,location='/etc/pki/tls/certs/ipa2.abaqis.com.crt'
>>>>>>>>
>>>>>>>>
>>>
>>>>>>>>
> CA: IPA issuer: subject: expires: unknown pre-save command:
>>>>>>>> post-save command: track: yes auto-renew: yes
>>>>>>>>
>>>>>>>> Fine, I check the http logs and get about the same:
>>>>>>>> [Thu Nov 28 22:03:06 2013] [error] ipa: ERROR:
>>>>>>>> ipaserver.plugins.dogtag.ra.request_certificate():
>>>>>>>> FAILURE (Authentication Error)
>>>>>>>>
>>>>>>>> Now as I understand it ipa-getcert is going to
>>>>>>>> theserver listed in /etc/ipa/default.conf, which in
>>>>>>>> this case is ipa2.abaqis.com (the request is coming
>>>>>>>> from the same host). The host principle in
>>>>>>>> /etc/krb5.keytab is used for authentication.
>>>>>>>>
>>>>>>>> I have tested against the primary ipa server and
>>>>>>>> everything works as it should. However, any requests
>>>>>>>> going against ipa2 for certificates are failing.
>>>>>>>>
>>>>>>>> At this point I am stuck, so any suggestions are
>>>>>>>> welcome.
>>>>>>>>
>>>>>>>> -Erinn
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> Replying to myself here, and narrowing this down a bit
>>>>>>> further this seems to be a straight auth problem against
>>>>>>> my secondary ipa server. All command work against the
>>>>>>> primary, all certificate commands against the secondary
>>>>>>> fail.
>>>>>>>
>>>>>>> It appears to be confined to dogtag (other commands like
>>>>>>> ipa user-show work), but how exactly dogtag handles auth
>>>>>>> I am not clear on. It appears as though mod_auth_kerb
>>>>>>> handles most things and that is definitely working.
>>>>>>> However any access against dogtag components is failing,
>>>>>>> so dogtag must/should/may be handling auth internally in
>>>>>>> a way that is failing.
>>>>>>>
>>>>>>> Anyway, suggestions are still welcome,
>>>>>>
>>>>>> Run this on the replica and see if it is being tracked by
>>>>>> certmonger
>>>>>>
>>>>>> # getcert list -d /etc/httpd/alias -n ipaCert
>>>>>>
>>>>>> If not, see if the a cert with the nickname ipaCert is in
>>>>>> /etc/httpd/alias:
>>>>>>
>>>>>> # certutil -L -d /etc/httpd/alias -n ipaCert
>>>>>>
>>>>>> If so, see if you have the key:
>>>>>>
>>>>>> # certutil -K -d /etc/httpd/alias -n ipaCert -f
>>>>>> /etc/httpd/alias/pwdfile.txt
>>>>>>
>>>>>> This is the RA agent certificate that IPA uses to
>>>>>> authenticate to dogtag. If it doesn't exist, or is expired,
>>>>>> or is the wrong one, then authentication will fail.
>>>>>>
>>>>>> The cert is shared amongst all the IPA masters, so if it
>>>>>> is working on one master then fixing the replica should be
>>>>>> straightforward assuming it already has the key.
>>>>>>
>>>>>> rob
>>>>>
>>>>> getcert list -d /etc/httpd/alias -n ipaCert Number of
>>>>> certificates and requests being tracked: 9. Request ID
>>>>> '20130221171049': status: MONITORING stuck: no key pair
>>>>> storage:
>>>>> type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
>>>>>
>>>>>
>>>
>>>>>
> Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
>>>>> certificate:
>>>>> type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
>>>>>
>>>>>
>>>
>>>>>
> Certificate DB'
>>>>> CA: dogtag-ipa-retrieve-agent-submit issuer: CN=Certificate
>>>>> Authority,O=ABAQIS.COM subject: CN=IPA RA,O=ABAQIS.COM
>>>>> expires: 2013-12-10 03:23:26 UTC eku:
>>>>> id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save
>>>>> command: /usr/lib64/ipa/certmonger/restart_httpd track: yes
>>>>> auto-renew: yes
>>>>>
>>>>> All the components appear to be there, the certificate is
>>>>> valid until the 10th as you can see about. The other two
>>>>> commands worked fine as well and everything appears to me to
>>>>> be valid.
>>>>>
>>>>> However, I am still getting the auth errors, and I note in
>>>>> the log, what I assume to be the first ipa server attempting
>>>>> to connect and getting auth errors as well.
>>>>
>>>> Ok, I'm a little unclear on something. Can any of your IPA
>>>> masters communicate with dogtag? I thought that one master
>>>> could and one couldn't.
>>>>
>>>> A simple way to find out is:
>>>>
>>>> # ipa cert-show 1
>>>>
>>>> This shows some random cert. If you get cert output then
>>>> things are working. If not the logs will probably confirm the
>>>> auth failure.
>>>>
>>>> Next step is to run this on each master:
>>>>
>>>> # certutil -L -d /etc/httpd/alias -n ipaCert | grep Serial
>>>>
>>>> The serial numbers should be the same. I wonder if you'll find
>>>> the to be different.
>>>>
>>>> The serial number should match this value:
>>>>
>>>> # ldapsearch -x -h localhost -p 7389 -b
>>>> uid=ipara,ou=People,o=ipaca description
>>>>
>>>> The second integer in description is the serial number.
>>>>
>>>> rob
>>>
>>> Yep that is correct, on one instance I can communicate on the
>>> other I can't. Let's call them primary and secondary, the primary
>>> was the original install from whence all the others came.
>>>
>>> ipa cert-show 1 on primary works, fails on secondary.
>>>
>>> Running certutil -L -d /etc/httpd/alias -n ipaCert | grep Serial
>>> on primary renders this:
>>>
>>> Serial Number: 136 (0x88) Serial Number: 92 (0x5c)
>>>
>>> Running against secondary gives only this: Serial Number: 92
>>> (0x5c)
>>>
>>> So it looks like that is probably the cause of all this mess,
>>> though how the primary got two certs? I dunno.
>>>
>>> Finally running the ldap query on both hosts confirms that 136 is
>>> the serial for the cert in question, so it appears that this
>>> somehow got messed up on the secondary.
>>>
>>> Sooo, with all that known (and thank you for the help, I figured
>>> it was something like this but reading the dogtag docs was taking
>>> me a good long while to suss out the issue), how does one fix
>>> it?
>>
>> NSS uses a database to store its certificates so it's possible
>> (and legal) to have multiple certs for the same private key. NSS
>> picks the best one (e.g. the one that's valid).
>>
>> So what you need to do on the primary is:
>>
>> # certutil -L -d /etc/httpd/alias -n ipaCert -a > /tmp/ra.crt
>>
>> Edit that file. there will be two certs in BEGIN/END blocks. If
>> memory serves the last one is the one you want, so remove the
>> first. You can double-check this after updating the file with:
>>
>> # openssl x509 -text -in /tmp/ra.crt
>>
>> It should have only the cert with the latest serial #.
>>
>> Now add it to your cert database:
>>
>> # certutil -A -n ipaCert -d /etc/httpd/alias -t u,u,u -a -i
>> /tmp/ra.crt # service httpd restart
>>
>> Things should work now.
>>
>> rob
>>
>
> Rob,
> Thanks so much for the help. It was the first certificate but other
> than that you were spot on, we can't all be perfect ;). That fixed the
> issue and I am now able to do cert operations on both hosts.
>
> Any idea how this situation could have come about?

The way it is supposed to work is this:

certmonger on primary should be tracking ipaCert and handled renewal. 
This involves renewing the cert, stuffing it into /etc/httpd/alias, 
updating an entry in the dogtag CA server and then stuffing the cert 
into the IPA LDAP database under cn=ca_renewal,cn=ipa,cn=etc,$SUFFIX.

certmonger on secondary should also be tracking ipaCert but it instead 
watches the cn=ca_renewal for an updated cert and if one appears, it 
will update it locally.

So either the cert never got stuffed into cn=ca_renewal or secondary 
isn't tracking the cert, or something I haven't thought of.

rob