[Freeipa-users] Dogtag not working?

Tue Dec 3 23:07:12 UTC 2013

On 12/3/2013 9:45 AM, Rob Crittenden wrote:
> Erinn Looney-Triggs wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> On 12/02/2013 10:18 AM, Rob Crittenden wrote:
>>> Erinn Looney-Triggs wrote:
>>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
>>>>
>>>> On 12/02/2013 08:03 AM, Rob Crittenden wrote:
>>>>> Erinn Looney-Triggs wrote:
>>>>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
>>>>>>
>>>>>> On 12/02/2013 07:40 AM, Rob Crittenden wrote:
>>>>>>> Erinn Looney-Triggs wrote:
>>>>>>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
>>>>>>>>
>>>>>>>> On 11/28/2013 03:50 PM, Erinn Looney-Triggs wrote:
>>>>>>>>> In the process of prepping a replication host for
>>>>>>>>> changing over the CA I had to use certmonger to
>>>>>>>>> generate another certificate on my secondary IPA
>>>>>>>>> server. Unfortunately it seems to fail every single
>>>>>>>>> time. Here is what I am running and here is what I am
>>>>>>>>> getting:
>>>>>>>>>
>>>>>>>>> ipa-getcert request -k private/ipa2.abaqis.com.key -f
>>>>>>>>> certs/ipa2.abaqis.com.crt -g 2048
>>>>>>>>>
>>>>>>>>> The request appears to work, however when checking the
>>>>>>>>> list I receive the following:
>>>>>>>>>
>>>>>>>>> ipa-getcert list -r Number of certificates and
>>>>>>>>> requests being tracked: 9. Request ID '20131128202128':
>>>>>>>>> status: CA_UNREACHABLE ca-error: Server failed request,
>>>>>>>>> will retry: 4301 (RPC failed at server. Certificate
>>>>>>>>> operation cannot be completed: FAILURE (Authentication
>>>>>>>>> Error)). stuck: yes key pair storage:
>>>>>>>>> type=FILE,location='/etc/pki/tls/private/ipa2.abaqis.com.key'
>>>>>>>>>
>>>>>>>>>
>>>>
>>>>>>>>>
>> certificate:
>>>>>>>>> type=FILE,location='/etc/pki/tls/certs/ipa2.abaqis.com.crt'
>>>>>>>>>
>>>>>>>>>
>>>>
>>>>>>>>>
>> CA: IPA issuer: subject: expires: unknown pre-save command:
>>>>>>>>> post-save command: track: yes auto-renew: yes
>>>>>>>>>
>>>>>>>>> Fine, I check the http logs and get about the same:
>>>>>>>>> [Thu Nov 28 22:03:06 2013] [error] ipa: ERROR:
>>>>>>>>> ipaserver.plugins.dogtag.ra.request_certificate():
>>>>>>>>> FAILURE (Authentication Error)
>>>>>>>>>
>>>>>>>>> Now as I understand it ipa-getcert is going to
>>>>>>>>> theserver listed in /etc/ipa/default.conf, which in
>>>>>>>>> this case is ipa2.abaqis.com (the request is coming
>>>>>>>>> from the same host). The host principle in
>>>>>>>>> /etc/krb5.keytab is used for authentication.
>>>>>>>>>
>>>>>>>>> I have tested against the primary ipa server and
>>>>>>>>> everything works as it should. However, any requests
>>>>>>>>> going against ipa2 for certificates are failing.
>>>>>>>>>
>>>>>>>>> At this point I am stuck, so any suggestions are
>>>>>>>>> welcome.
>>>>>>>>>
>>>>>>>>> -Erinn
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> Replying to myself here, and narrowing this down a bit
>>>>>>>> further this seems to be a straight auth problem against
>>>>>>>> my secondary ipa server. All command work against the
>>>>>>>> primary, all certificate commands against the secondary
>>>>>>>> fail.
>>>>>>>>
>>>>>>>> It appears to be confined to dogtag (other commands like
>>>>>>>> ipa user-show work), but how exactly dogtag handles auth
>>>>>>>> I am not clear on. It appears as though mod_auth_kerb
>>>>>>>> handles most things and that is definitely working.
>>>>>>>> However any access against dogtag components is failing,
>>>>>>>> so dogtag must/should/may be handling auth internally in
>>>>>>>> a way that is failing.
>>>>>>>>
>>>>>>>> Anyway, suggestions are still welcome,
>>>>>>>
>>>>>>> Run this on the replica and see if it is being tracked by
>>>>>>> certmonger
>>>>>>>
>>>>>>> # getcert list -d /etc/httpd/alias -n ipaCert
>>>>>>>
>>>>>>> If not, see if the a cert with the nickname ipaCert is in
>>>>>>> /etc/httpd/alias:
>>>>>>>
>>>>>>> # certutil -L -d /etc/httpd/alias -n ipaCert
>>>>>>>
>>>>>>> If so, see if you have the key:
>>>>>>>
>>>>>>> # certutil -K -d /etc/httpd/alias -n ipaCert -f
>>>>>>> /etc/httpd/alias/pwdfile.txt
>>>>>>>
>>>>>>> This is the RA agent certificate that IPA uses to
>>>>>>> authenticate to dogtag. If it doesn't exist, or is expired,
>>>>>>> or is the wrong one, then authentication will fail.
>>>>>>>
>>>>>>> The cert is shared amongst all the IPA masters, so if it
>>>>>>> is working on one master then fixing the replica should be
>>>>>>> straightforward assuming it already has the key.
>>>>>>>
>>>>>>> rob
>>>>>>
>>>>>> getcert list -d /etc/httpd/alias -n ipaCert Number of
>>>>>> certificates and requests being tracked: 9. Request ID
>>>>>> '20130221171049': status: MONITORING stuck: no key pair
>>>>>> storage:
>>>>>> type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
>>>>>>
>>>>>>
>>>>
>>>>>>
>> Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
>>>>>> certificate:
>>>>>> type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
>>>>>>
>>>>>>
>>>>
>>>>>>
>> Certificate DB'
>>>>>> CA: dogtag-ipa-retrieve-agent-submit issuer: CN=Certificate
>>>>>> Authority,O=ABAQIS.COM subject: CN=IPA RA,O=ABAQIS.COM
>>>>>> expires: 2013-12-10 03:23:26 UTC eku:
>>>>>> id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save
>>>>>> command: /usr/lib64/ipa/certmonger/restart_httpd track: yes
>>>>>> auto-renew: yes
>>>>>>
>>>>>> All the components appear to be there, the certificate is
>>>>>> valid until the 10th as you can see about. The other two
>>>>>> commands worked fine as well and everything appears to me to
>>>>>> be valid.
>>>>>>
>>>>>> However, I am still getting the auth errors, and I note in
>>>>>> the log, what I assume to be the first ipa server attempting
>>>>>> to connect and getting auth errors as well.
>>>>>
>>>>> Ok, I'm a little unclear on something. Can any of your IPA
>>>>> masters communicate with dogtag? I thought that one master
>>>>> could and one couldn't.
>>>>>
>>>>> A simple way to find out is:
>>>>>
>>>>> # ipa cert-show 1
>>>>>
>>>>> This shows some random cert. If you get cert output then
>>>>> things are working. If not the logs will probably confirm the
>>>>> auth failure.
>>>>>
>>>>> Next step is to run this on each master:
>>>>>
>>>>> # certutil -L -d /etc/httpd/alias -n ipaCert | grep Serial
>>>>>
>>>>> The serial numbers should be the same. I wonder if you'll find
>>>>> the to be different.
>>>>>
>>>>> The serial number should match this value:
>>>>>
>>>>> # ldapsearch -x -h localhost -p 7389 -b
>>>>> uid=ipara,ou=People,o=ipaca description
>>>>>
>>>>> The second integer in description is the serial number.
>>>>>
>>>>> rob
>>>>
>>>> Yep that is correct, on one instance I can communicate on the
>>>> other I can't. Let's call them primary and secondary, the primary
>>>> was the original install from whence all the others came.
>>>>
>>>> ipa cert-show 1 on primary works, fails on secondary.
>>>>
>>>> Running certutil -L -d /etc/httpd/alias -n ipaCert | grep Serial
>>>> on primary renders this:
>>>>
>>>> Serial Number: 136 (0x88) Serial Number: 92 (0x5c)
>>>>
>>>> Running against secondary gives only this: Serial Number: 92
>>>> (0x5c)
>>>>
>>>> So it looks like that is probably the cause of all this mess,
>>>> though how the primary got two certs? I dunno.
>>>>
>>>> Finally running the ldap query on both hosts confirms that 136 is
>>>> the serial for the cert in question, so it appears that this
>>>> somehow got messed up on the secondary.
>>>>
>>>> Sooo, with all that known (and thank you for the help, I figured
>>>> it was something like this but reading the dogtag docs was taking
>>>> me a good long while to suss out the issue), how does one fix
>>>> it?
>>>
>>> NSS uses a database to store its certificates so it's possible
>>> (and legal) to have multiple certs for the same private key. NSS
>>> picks the best one (e.g. the one that's valid).
>>>
>>> So what you need to do on the primary is:
>>>
>>> # certutil -L -d /etc/httpd/alias -n ipaCert -a > /tmp/ra.crt
>>>
>>> Edit that file. there will be two certs in BEGIN/END blocks. If
>>> memory serves the last one is the one you want, so remove the
>>> first. You can double-check this after updating the file with:
>>>
>>> # openssl x509 -text -in /tmp/ra.crt
>>>
>>> It should have only the cert with the latest serial #.
>>>
>>> Now add it to your cert database:
>>>
>>> # certutil -A -n ipaCert -d /etc/httpd/alias -t u,u,u -a -i
>>> /tmp/ra.crt # service httpd restart
>>>
>>> Things should work now.
>>>
>>> rob
>>>
>>
>> Rob,
>> Thanks so much for the help. It was the first certificate but other
>> than that you were spot on, we can't all be perfect ;). That fixed the
>> issue and I am now able to do cert operations on both hosts.
>>
>> Any idea how this situation could have come about?
>
> The way it is supposed to work is this:
>
> certmonger on primary should be tracking ipaCert and handled renewal. 
> This involves renewing the cert, stuffing it into /etc/httpd/alias, 
> updating an entry in the dogtag CA server and then stuffing the cert 
> into the IPA LDAP database under cn=ca_renewal,cn=ipa,cn=etc,$SUFFIX.
>
> certmonger on secondary should also be tracking ipaCert but it instead 
> watches the cn=ca_renewal for an updated cert and if one appears, it 
> will update it locally.
>
> So either the cert never got stuffed into cn=ca_renewal or secondary 
> isn't tracking the cert, or something I haven't thought of.
>
> rob

Hmm ok, well I can confirm that the ipaCert value is correct and the 
latest certificate. certmonger on the primary did another renewal last 
night, which of course landed me in the same place on the secondary, so 
I manually imported again.

I will look into certmonger and it's config on the secondary, as it 
doesn't seem to be pulling down the most recent cert from LDAP.

-Erinn