[Freeipa-users] strange replica install error (another one)

Thu Dec 4 17:30:48 UTC 2014

On 12/04/2014 11:01 AM, Janelle wrote:
> To help understand the environment a bit - perhaps this will help.
>
>  1. Approx 7500 clients across 3 datacenters- all manor of *nix,
>     ranging from AIX, Linux, HP-UX and Solaris - hence the reason why
>     they all can't use ipa-client configs. Although that is in the
>     plan at least for Linux systems.
>  2. Servers/masters/replicas = 12 (4 each in 3 datacenters). Each DC
>     has a primary CA so the replication agreements are between the 3
>     servers in each DC plus 1 server is a hot-spare and used strictly
>     for replication to the other datacenters.
>  3. running out of FDs because there are 100s of jobs running across
>     all the servers that do a lot of sudo and group lookups and more
>     have to happen. Also, approx 1100 users accessing servers in vary
>     random ways - but just using ssh/pssh/other-tools.
>

This is what nscd/sssd/others are used for - to cache those client 
connections/loads to keep the load off of the server.

Note that openldap uses epoll instead of poll() which is why it is 
better able to scale to 65k connections.  389 works well in environments 
where client caching is used.  The use of 65k connections in 389 could 
explain the high CPU load, but we would need stacktraces to determine 
that (as in the aforementioned Debugging_Hangs).

This still doesn't explain the crashing.

> Not sure if this helps - but perhaps?
>
> ~Janelle
>
> On 12/4/14 8:41 AM, Ludwig Krispenz wrote:
>>
>> On 12/04/2014 04:56 PM, Janelle wrote:
>>> Hi all,
>>>
>>> just (pam)auth and nslcd
>>>
>>> It was ported from a running OpenLDAP environment to IPA. Just 
>>> trying to do conversions in stages so as not to change too much all 
>>> at once. Thought I could go from OpenLDAP to IPA and just use the 
>>> backend of 389ds. Functionally it does work, but the load kills it. 
>>> Seems like FDs are a huge problem.  But all the settings documented 
>>> don't see to resolve the magic:
>>>
>>> / Netscape Portable Runtime error -5971 (Process open FD table is 
>>> full.)/
>>>
>>> error.
>>>
>>> Shouldn't this increase file descriptors in conjunction with 
>>> /etc/sysconfig/dirsrv.systemd change? FS-limits across the OS are 
>>> set to 65535 - /etc/security/limits.conf, /proc, sysctl.conf -- 
>>> everything but 389-ds itself. But I still can't get this to work, 
>>> although it does not give an error.
>>>
>>> ldapmodify -x -D "cn=directory manager" -W <<EOF
>>> dn: cn=config,cn=ldbm database,cn=plugins,cn=config
>>> changetype: modify
>>> replace: nsslapd-maxdescriptors
>>> nsslapd-maxdescriptors: 65535
>>> -
>>> replace: nsslapd-dtablesize
>>> nsslapd-dtablesize: 65535
>>> -
>>> replace: nsslapd-reservedescriptors
>>> nsslapd-reservedescriptors: 100
>>> EOF
>> there are two problems
>> - how to increase the connetction table size in DS
>> I would suggest to replace nsslapd-conntablesize and restart the 
>> server, the change is not dynamic
>> - why you are running out of file descriptors
>> you should query "cn=monitor" to check the effective tablesize and 
>> the number of established connections
>> it could be a problem of clients not well behaving and not closing 
>> connections. you could set a low value of nsslapd-idletimeout to get 
>> rid of these connections
>>
>>>
>>> Thanks
>>> ~J
>>>
>>> On 12/4/14 7:40 AM, Rob Crittenden wrote:
>>>> Dmitri Pal wrote:
>>>>> On 12/04/2014 09:41 AM, Rich Megginson wrote:
>>>>>> On 12/04/2014 08:39 AM, Rich Megginson wrote:
>>>>>>> On 12/04/2014 01:45 AM, Petr Spacek wrote:
>>>>>>>> On 4.12.2014 05:02, Janelle wrote:
>>>>>>>>> Thanks -- still a bit strange that it did not show up on some
>>>>>>>>> servers - vary
>>>>>>>>> random and intermittent.
>>>>>>>>>
>>>>>>>>> BTW - a bit of information others might find useful.  If you try to
>>>>>>>>> use the
>>>>>>>>> "LDAP" portion of IPA for authentication - rather than fulling
>>>>>>>>> installing the
>>>>>>>>> IPA client and using Kerberos - the servers running ds-389 do not
>>>>>>>>> do well in
>>>>>>>>> handling the load. In other words - a few hundred hosts trying to
>>>>>>>>> authenticate
>>>>>>>>> via LDAP only will send CPU through the roof and crashes the slapd
>>>>>>>>> process
>>>>>>>>> often.
>>>>>>> That should not happen.
>>>>>>> For crashes, we would need to look at some stack traces:
>>>>>>> http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes
>>>>>>> For situations when the CPU is through the roof, that is very similar
>>>>>>> to debugging hangs:
>>>>>>> http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-hangs
>>>>>> Sorry, forgot to mention that since this is IPA you'll also need to
>>>>>> install the ipa-debuginfo and slapi-nis-debuginfo packages.
>>>>>>
>>>>> I would also add a question about your client configuration.
>>>>> For example if you use SSSD with enumerate=true for your clients then
>>>>> yes you will get your environment to the knees pretty quickly.
>>>> I assumed SSSD wasn't being used at all which begs the question: what
>>>> is? nss_ldap? Is nslcd being used?
>>>>
>>>> What is hitting LDAP, only auth or something else (e.g. sudo, automount).
>>>>
>>>> rob
>>>>
>>>>>>>>> Since IPA is supposed to handle all options, I guess I am
>>>>>>>>> disappointed.
>>>>>>>>>
>>>>>>>>> regards
>>>>>>>>> ~J
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 12/3/14 2:56 PM, Dmitri Pal wrote:
>>>>>>>>>> On 12/03/2014 04:40 PM, Janelle wrote:
>>>>>>>>>>> Here is a bit of baffling one on 4.0.5:
>>>>>>>>>>>
>>>>>>>>>>> Replica install p11-kit???
>>>>>>>>>> This is a part of the DNSSEC set of packages.
>>>>>>>>>>
>>>>>>>>>>> Connection from master to replica is OK.
>>>>>>>>>>>
>>>>>>>>>>> Connection check OK
>>>>>>>>>>> p11-kit: ipa.p11-kit: x-public-key-info: invalid or unsupported
>>>>>>>>>>> attribute
>>>>>>>>>>> Configuring NTP daemon (ntpd)
>>>>>>>>>>>     [1/4]: stopping ntpd
>>>>>>>>>>>     [2/4]: writing configuration
>>>>>>>>>>> ...
>>>>>>>>>>>
>>>>>>>>>>> Your system may be partly configured.
>>>>>>>>>>> Run /usr/sbin/ipa-server-install --uninstall to clean up.
>>>>>>>>>>>
>>>>>>>>>>> LDAP error: UNWILLING_TO_PERFORM
>>>>>>>>>>> database is read-only
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thoughts?
>>>>>>>> We need more information about your problem.
>>>>>>>>
>>>>>>>> As always, please start with information requested on
>>>>>>>> http://www.freeipa.org/page/Troubleshooting#Reporting_bugs
>>>>>>>>
>>>>>>>> /var/log/ipa*.log from affected replica will be invaluable (along
>>>>>>>> with exact
>>>>>>>> package version numbers [including p11-kit] and repo configuration).
>>>>>>>>
>>>
>>>
>>>
>>
>>
>>
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20141204/bc62e3ca/attachment.htm>