[Freeipa-users] Replica created with expired certs
Jim Richard
jrichard at placeiq.com
Sat Oct 1 17:23:04 UTC 2016
Hi Rob:
First I wanted to thank you for all of your valuable input/tips. As you well know, everything about certs, certmonger, dogtag and FreeIPA can get very complicated - there’s no easy answer, so many things can go wrong :)
But, your answers to my questions got me thinking, gave me some clues, pointed me in the right direction.
I wanted to take the time to specifically thank you because these concepts have mystified me for quite a while, our FreeIPA system has been running for more than a year with everything regarding certs kinda wacky and with me just praying that that fact didn’t crash everything and make the most important function for us (ssh, sssd, authentication, sso) stop working.
With your help I have certainly not become an expert but have gone from pretty much clueless to having somewhat of a clue :) That’s progress !!
My issue with the CA certs themselves is solved thanks to you pointing out the issue with creating replicas in 3.0 which has been fixed in 3.3 - the issue that can be solved by manually exporting a new cacert.p12 file and boom, new replicas created with expired certs issue solved.
And then there was the issue of “sec error legacy database” which would manifest itself in various forms and can be caused by many things - it is temporarily solved by restarting httpd but then just comes right back.
Based on your input I started looking at the certs/certmonger/getcert list - on all my nodes/hosts and noticed that many of them had bogus certs with principal names pointed at hosts that no longer existed. No other way to describe them other than WTF !!.
My theory now is that all the nodes calling in to the CA with all those bogus certs were just overloading the CA and so after restarting httpd, it would temporarily clear up until all the nodes starting calling in to the CA again - or something like that.
Anyways, Ansible to the rescue….
I exported a list of hosts from my IPA system, that became my Ansible inventory file.
Now, throw together a quick playbook to look at every host, identify the bogus cert or certs and tell certmonger to stop tracking them.
The simple Ansible playbook follows here.
Run that against all hosts and bingo !!! - my httpd logs on the CA are no longer getting spammed with bogus cert requests, “sec error legacy database” errors are not happening, etc , etc.
In short, my FreeIPA CA situation is now, I hope and pray, fairly stable.
So HUGE shout out to you Rob !!!
---
- hosts: ipa-hosts
gather_facts: False
tasks:
- name: get request id
shell: ipa-getcert list -r | gawk -F\' '/Request/ {print $2}'
register: my_id
#- debug: var=my_id
- name: kill bad certs
shell: ipa-getcert stop-tracking -i {{ item }}
with_items: "{{ my_id.stdout_lines }}"
<http://www.placeiq.com/> <http://www.placeiq.com/> <http://www.placeiq.com/> Jim Richard <https://twitter.com/placeiq> <https://twitter.com/placeiq> <https://twitter.com/placeiq> <https://www.facebook.com/PlaceIQ> <https://www.facebook.com/PlaceIQ> <https://www.linkedin.com/company/placeiq> <https://www.linkedin.com/company/placeiq>
SYSTEM ADMINISTRATOR III
(646) 338-8905
<http://www.placeiq.com/2015/05/26/placeiq-named-winner-of-prestigious-2015-oracle-data-cloud-activate-award/> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/> <http://placeiq.com/2016/03/08/measuring-addressable-tv-campaigns-is-now-possible/> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/> <http://pages.placeiq.com/Location-Data-Accuracy-Whitepaper-Download.html?utm_source=Signature&utm_medium=Email&utm_campaign=AccuracyWP>
> On Sep 30, 2016, at 4:53 AM, Rob Crittenden <rcritten at redhat.com> wrote:
>
> Jim Richard wrote:
>> Can I and how…
>>
>> delete all certs for all hosts
>>
>> I mean, we only use FreeIPA for user login/sssd
>>
>> That said, do we even need those certs?
>
> There is no simple answer, really.
>
> Yes, you can deleted all certs for all hosts (not recommended as some of those are for IPA services). I doubt it would do anything positive and if the certificate is tracked by certmonger on the client it would eventually renew.
>
> Do you need the certs? Only you would know that, but chances are the vast majority aren't being used.
>
> In 3.0 when a client is registered a host certificate is obtained for it. This certificate was never used and in 4.something it isn't requested at all unless an option is passed to ipa-client-install.
>
> rob
>
>>
>>
>>
>> <http://www.placeiq.com/><http://www.placeiq.com/><http://www.placeiq.com/>
>> Jim Richard
>> <https://twitter.com/placeiq><https://twitter.com/placeiq><https://twitter.com/placeiq>
>> <https://www.facebook.com/PlaceIQ><https://www.facebook.com/PlaceIQ>
>> <https://www.linkedin.com/company/placeiq><https://www.linkedin.com/company/placeiq>
>> SYSTEM ADMINISTRATOR III
>> /(646) 338-8905 /
>>
>>
>> <http://www.placeiq.com/2015/05/26/placeiq-named-winner-of-prestigious-2015-oracle-data-cloud-activate-award/><http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/><http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/><http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/><http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/><http://placeiq.com/2016/03/08/measuring-addressable-tv-campaigns-is-now-possible/><http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/><http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/><http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/><http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/><http://placeiq.com/2016/04/13/placeiq-joins-the-network-a!
> dvertising
> -initiative-nai-as-100th-member/>PlaceIQ:Location
>> Data Accuracy
>> <http://pages.placeiq.com/Location-Data-Accuracy-Whitepaper-Download.html?utm_source=Signature&utm_medium=Email&utm_campaign=AccuracyWP>
>>
>>
>>
>>> On Sep 29, 2016, at 8:53 PM, Jim Richard <jrichard at placeiq.com
>>> <mailto:jrichard at placeiq.com>> wrote:
>>>
>>> another interesting thing, my httpd/error_logs are constantly getting
>>> spammed with: (I removed the stuff between the single quotes)
>>>
>>> Notice those names don’t match, should they?
>>>
>>> Me thinks not since those “principal=“ items are ALMOST all hosts that
>>> no longer exist in the FreeIPA system. I rare few do exist.
>>>
>>> So, that’s weird :)
>>>
>>> [Thu Sep 29 20:44:59 2016] [error] ipa: INFO:
>>> host/aerospike-cl1-203.nym1.placeiq.net at PLACEIQ.NET
>>> <mailto:host/aerospike-cl1-203.nym1.placeiq.net at placeiq.net>:
>>> cert_request(u’…………………..',
>>> principal=u'host/sbtt-nyc1-028.thum01.nym1.placeiq.net at PLACEIQ.NET
>>> <mailto:principal=u'host/sbtt-nyc1-028.thum01.nym1.placeiq.net at placeiq.net>',
>>> add=True): CertificateOperationError
>>>
>>> [Thu Sep 29 20:45:06 2016] [error] ipa: INFO:
>>> host/aerospike-cl2-210.nym1.placeiq.net at PLACEIQ.NET
>>> <mailto:host/aerospike-cl2-210.nym1.placeiq.net at placeiq.net>:
>>> cert_request(u’…………………..',
>>> principal=u'host/017.prod07.nym1.placeiq.net at PLACEIQ.NET
>>> <mailto:principal=u'host/017.prod07.nym1.placeiq.net at placeiq.net>',
>>> add=True): CertificateOperationError
>>>
>>> [Thu Sep 29 20:45:09 2016] [error] ipa: INFO:
>>> host/adsgateway-14.nym1.placeiq.net at PLACEIQ.NET
>>> <mailto:host/adsgateway-14.nym1.placeiq.net at placeiq.net>:
>>> cert_request(u’……………………...',
>>> principal=u'host/025.prod07.nym1.placeiq.net at PLACEIQ.NET
>>> <mailto:principal=u'host/025.prod07.nym1.placeiq.net at placeiq.net>',
>>> add=True): CertificateOperationError
>>>
>>> [Thu Sep 29 20:45:29 2016] [error] ipa: INFO:
>>> host/ttsandbox-022.nym1.placeiq.net at PLACEIQ.NET
>>> <mailto:host/ttsandbox-022.nym1.placeiq.net at placeiq.net>:
>>> cert_request(u’……………………….',
>>> principal=u'host/sbtt-nyc1-022.thum01.nym1.placeiq.net at PLACEIQ.NET
>>> <mailto:principal=u'host/sbtt-nyc1-022.thum01.nym1.placeiq.net at placeiq.net>',
>>> add=True): CertificateOperationError
>>>
>>>
>>>
>>>
>>>
>>>
>>> <http://www.placeiq.com/><http://www.placeiq.com/><http://www.placeiq.com/>
>>> Jim Richard
>>> <https://twitter.com/placeiq><https://twitter.com/placeiq><https://twitter.com/placeiq>
>>> <https://www.facebook.com/PlaceIQ><https://www.facebook.com/PlaceIQ>
>>> <https://www.linkedin.com/company/placeiq><https://www.linkedin.com/company/placeiq>
>>> SYSTEM ADMINISTRATOR III
>>> /(646) 338-8905 /
>>>
>>>
>>> <http://www.placeiq.com/2015/05/26/placeiq-named-winner-of-prestigious-2015-oracle-data-cloud-activate-award/><http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/><http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/><http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/><http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/><http://placeiq.com/2016/03/08/measuring-addressable-tv-campaigns-is-now-possible/><http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/><http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/><http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/><http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/><http://placeiq.com/2016/04/13/placeiq-joins-the-network-!
> advertisin
> g-initiative-nai-as-100th-member/>PlaceIQ:Location
>>> Data Accuracy
>>> <http://pages.placeiq.com/Location-Data-Accuracy-Whitepaper-Download.html?utm_source=Signature&utm_medium=Email&utm_campaign=AccuracyWP>
>>>
>>>
>>>
>>>> On Sep 29, 2016, at 8:11 AM, Rob Crittenden <rcritten at redhat.com
>>>> <mailto:rcritten at redhat.com>> wrote:
>>>>
>>>> Natxo Asenjo wrote:
>>>>> hi Jim,
>>>>>
>>>>> On Thu, Sep 29, 2016 at 7:37 AM, Jim Richard <jrichard at placeiq.com
>>>>> <mailto:jrichard at placeiq.com>
>>>>> <mailto:jrichard at placeiq.com>> wrote:
>>>>>
>>>>> Thanks Rob, that worked.
>>>>>
>>>>> Still on the subject of certs, any idea how to solve this error:
>>>>>
>>>>> Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The
>>>>> certificate/key database is in an old, unsupported format.
>>>>>
>>>>> I see that in the gui when querying hosts as well as from cli when I
>>>>> ipa-show or ipa-find
>>>>>
>>>>>
>>>>> I have had this too, and we did not find a solution (search my recent
>>>>> posts on the archives). As a workaround I have created replicas and
>>>>> decommissioned the older replicas.
>>>>
>>>> On the one hand I'm glad this fixed it for you. On the other it is a
>>>> rather unsatisfying answer. Unfortunately NSS doesn't always provide
>>>> the most context with its error messages. This error is usually seen
>>>> when one tries to open a non-existent database, which in this case is
>>>> a very strange thing, especially since it goes from working to
>>>> non-working in the same apache process over a few minutes.
>>>>
>>>> I'm not sure how I'd troubleshoot this if it were easily
>>>> reproducible. I suspect we'd need to figure out which database cannot
>>>> be found (most likely /etc/httpd/alias) and go from there. An strace
>>>> is a brute-force way to see the file open but finding the right
>>>> process to attach to is a bit of an art.
>>>>
>>>> rob
>>>>
>>>> --
>>>> Manage your subscription for the Freeipa-users mailing list:
>>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>>> Go to http://freeipa.org for more info on the project
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20161001/2302c6c4/attachment.htm>
More information about the Freeipa-users
mailing list