[Freeipa-users] Certificate system unavailable

[Rob Crittenden]

Sigbjorn Lie wrote:
> On 20/02/14 21:19, Rob Crittenden wrote:
>> Sigbjorn Lie wrote:
>>>
>>>
>>>
>>> On Wed, February 19, 2014 13:45, Sigbjorn Lie wrote:
>>>>
>>>
>>>>
>>>> On Tue, February 18, 2014 20:45, Rob Crittenden wrote:
>>>>
>>>>> Sigbjorn Lie wrote:
>>>>>
>>>>>
>>>>>>> On what machine are you trying to use the ipa tool? Is it one of the
>>>>>>> masters, all of them, enrolled clients?
>>>>>>>
>>>>>>
>>>>>> It's the same error message when the "ipa" command is run directly
>>>>>> on any of the masters.
>>>>>>
>>>>>>
>>>>>>
>>>>>> And it's the same error message if I run the "ipa" command on any
>>>>>> of the clients.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I do not have a working "ipa" command anywhere anymore.
>>>>>>
>>>>>>
>>>>>
>>>>> Ok, let's test out the cert that ipa is using. Try this on any one of
>>>>> the masters:
>>>>>
>>>>> $ curl https://`hostname`/ipa/xml
>>>>> Should fail with Peer certificate cannot be authenticated with
>>>>> known CA
>>>>> certificates
>>>>
>>>> Yes, this did not work:
>>>>
>>>>
>>>> curl: (60) Peer certificate cannot be authenticated with known CA
>>>> certificates
>>>> More details here: http://curl.haxx.se/docs/sslcerts.html
>>>>
>>>>
>>>>
>>>>>
>>>>> $ curl --cacert /etc/ipa/ca.crt https://`hostname`/ipa/xml
>>>>> Should succeed in that you get the "you are not logged in" HTML page
>>>>>
>>>>>
>>>>
>>>> Indeed - this works.
>>>>
>>>>
>>>>>
>>>>> Ok, now unfortunately curl only handles the sql-style NSS databases so
>>>>> we can't fully reproduce it the same way that the IPA client is
>>>>> doing things, but here is an
>>>>> approximation:
>>>>>
>>>>>
>>>>>
>>>>> # certutil -A -d sql:/etc/pki/nssdb -n 'IPA CA' -t CT,C,C -a -i
>>>>> /etc/ipa/ca.crt
>>>>> $ curl https://`hostname`/ipa/xml
>>>>> You should see the login page HTML
>>>>>
>>>>>
>>>>
>>>> Yes, I can now see the login page HTML.
>>>>
>>>>
>>>>
>>>>>
>>>>> If you stick a -v in there it'll give you more verbose output which
>>>>> would be useful if any of these fail in an unexpected way.
>>>>>
>>>>>>> Whatever is going on isn't likely related to the web server Apache
>>>>>>> database as you get the same error out of each one. The client
>>>>>>> log you sent confirmed that
>>>>>>> it tried to contact each master. The SSL error we're getting is
>>>>>>> that the client doesn't
>>>>>>> trust the CA that
>>>>>>> signed the server certificate so this appears to be a problem on
>>>>>>> the client, which begs the
>>>>>>> question: all clients or just this one?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> All clients.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> NSS is smart enough to handle multiple certificates, it should
>>>>>>> pick the
>>>>>>> newest one on startup.
>>>>>>>
>>>>>>
>>>>>> Ok.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Where do you suggest I continue troubleshooting this issue?
>>>>>>
>>>>>>
>>>>>
>>>>> We can also tackle this on the server side. Let's verify the server
>>>>> cert:
>>>>>
>>>>>
>>>>>
>>>>> # certutil -V -u V -n Server-Cert -d /etc/httpd/alias
>>>>>
>>>>>
>>>> This produces the same output for all 3 servers:
>>>>
>>>>
>>>> certutil: certificate is valid
>>>>
>>>>
>>>>>
>>>>> This is verified on server startup so I expect it to be valid, but
>>>>> doesn't hurt to try.
>>>>>
>>>>> Restarting the Apache process might be something to try as changes to
>>>>> the NSS database aren't picked up until a restart.
>>>>>
>>>>
>>>> I ran "# certutil -A -d sql:/etc/pki/nssdb -n 'IPA CA' -t CT,C,C -a
>>>> -i /etc/ipa/ca.crt" on ipa03,
>>>>   and restarted httpd.
>>>>
>>>> The "ipa" command is now *working* on ipa03. :)
>>>>
>>>>
>>>> However it's *not working* a client who's /etc/ipa/default.conf
>>>> points at ipa03.
>>>>
>>>>
>>>> Do I have to refresh the PKI CA in /etc/pki/nssdb on every client?
>>
>> You shouldn't need to. Frankly I'm surprised this worked. What is the
>> distro and what version of nss is installed?
>>
>> rob
>>
>
> This is RHEL 6.5 with nss-3.15.3-3.el6_5.x86_64.
>
> I am surprised too. I dumped the PKI CA certificate from /etc/pki/nssdb
> before and after I updated it into text files, and diff'ed them. No
> differences was reported.

I can't think of a reason it would be using the sqlite database at all. 
You don't have NSS_DEFAULT_DB_TYPE set somewhere do you? I'd find it 
hard to believe that this would be set EVERYWHERE.

If we want to brute force things, trying strace against a client that 
isn't working to confirm that it is trying to open cert9 might give us a 
data point at least.

rob