<html> <head> <meta content="text/html; charset=windows-1252" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> <div class="moz-cite-prefix">On 12/04/2014 11:01 AM, Janelle wrote: </div> <blockquote cite="mid:5480936F.1040500@gmail.com" type="cite"> <meta content="text/html; charset=windows-1252" http-equiv="Content-Type"> To help understand the environment a bit - perhaps this will help. <ol> <li>Approx 7500 clients across 3 datacenters- all manor of *nix, ranging from AIX, Linux, HP-UX and Solaris - hence the reason why they all can't use ipa-client configs. Although that is in the plan at least for Linux systems. </li> <li>Servers/masters/replicas = 12 (4 each in 3 datacenters). Each DC has a primary CA so the replication agreements are between the 3 servers in each DC plus 1 server is a hot-spare and used strictly for replication to the other datacenters. </li> <li>running out of FDs because there are 100s of jobs running across all the servers that do a lot of sudo and group lookups and more have to happen. Also, approx 1100 users accessing servers in vary random ways - but just using ssh/pssh/other-tools.</li> </ol> </blockquote> This is what nscd/sssd/others are used for - to cache those client connections/loads to keep the load off of the server. Note that openldap uses epoll instead of poll() which is why it is better able to scale to 65k connections. 389 works well in environments where client caching is used. The use of 65k connections in 389 could explain the high CPU load, but we would need stacktraces to determine that (as in the aforementioned Debugging_Hangs). This still doesn't explain the crashing. <blockquote cite="mid:5480936F.1040500@gmail.com" type="cite"> <ol> </ol> Not sure if this helps - but perhaps? ~Janelle <div class="moz-cite-prefix">On 12/4/14 8:41 AM, Ludwig Krispenz wrote: </div> <blockquote cite="mid:54808ECC.5010406@redhat.com" type="cite"> <meta content="text/html; charset=windows-1252" http-equiv="Content-Type"> <div class="moz-cite-prefix">On 12/04/2014 04:56 PM, Janelle wrote: </div> <blockquote cite="mid:5480844A.6040405@gmail.com" type="cite"> <meta content="text/html; charset=windows-1252" http-equiv="Content-Type"> Hi all, just (pam)auth and nslcd It was ported from a running OpenLDAP environment to IPA. Just trying to do conversions in stages so as not to change too much all at once. Thought I could go from OpenLDAP to IPA and just use the backend of 389ds. Functionally it does work, but the load kills it. Seems like FDs are a huge problem. But all the settings documented don't see to resolve the magic: Netscape Portable Runtime error -5971 (Process open FD table is full.) error. Shouldn't this increase file descriptors in conjunction with /etc/sysconfig/dirsrv.systemd change? FS-limits across the OS are set to 65535 - /etc/security/limits.conf, /proc, sysctl.conf -- everything but 389-ds itself. But I still can't get this to work, although it does not give an error. ldapmodify -x -D "cn=directory manager" -W <<EOF dn: cn=config,cn=ldbm database,cn=plugins,cn=config changetype: modify replace: nsslapd-maxdescriptors nsslapd-maxdescriptors: 65535 - replace: nsslapd-dtablesize nsslapd-dtablesize: 65535 - replace: nsslapd-reservedescriptors nsslapd-reservedescriptors: 100 EOF </blockquote> there are two problems - how to increase the connetction table size in DS I would suggest to replace nsslapd-conntablesize and restart the server, the change is not dynamic - why you are running out of file descriptors you should query "cn=monitor" to check the effective tablesize and the number of established connections it could be a problem of clients not well behaving and not closing connections. you could set a low value of nsslapd-idletimeout to get rid of these connections <blockquote cite="mid:5480844A.6040405@gmail.com" type="cite"> Thanks ~J <div class="moz-cite-prefix">On 12/4/14 7:40 AM, Rob Crittenden wrote: </div> <blockquote cite="mid:5480807B.8040805@redhat.com" type="cite"> <pre wrap="">Dmitri Pal wrote: </pre> <blockquote type="cite"> <pre wrap="">On 12/04/2014 09:41 AM, Rich Megginson wrote: </pre> <blockquote type="cite"> <pre wrap="">On 12/04/2014 08:39 AM, Rich Megginson wrote: </pre> <blockquote type="cite"> <pre wrap="">On 12/04/2014 01:45 AM, Petr Spacek wrote: </pre> <blockquote type="cite"> <pre wrap="">On 4.12.2014 05:02, Janelle wrote: </pre> <blockquote type="cite"> <pre wrap="">Thanks -- still a bit strange that it did not show up on some servers - vary random and intermittent. BTW - a bit of information others might find useful. If you try to use the "LDAP" portion of IPA for authentication - rather than fulling installing the IPA client and using Kerberos - the servers running ds-389 do not do well in handling the load. In other words - a few hundred hosts trying to authenticate via LDAP only will send CPU through the roof and crashes the slapd process often. </pre> </blockquote> </blockquote> <pre wrap="">That should not happen. For crashes, we would need to look at some stack traces: <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes">http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes</a> For situations when the CPU is through the roof, that is very similar to debugging hangs: <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-hangs">http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-hangs</a> </pre> </blockquote> <pre wrap="">Sorry, forgot to mention that since this is IPA you'll also need to install the ipa-debuginfo and slapi-nis-debuginfo packages. </pre> </blockquote> <pre wrap="">I would also add a question about your client configuration. For example if you use SSSD with enumerate=true for your clients then yes you will get your environment to the knees pretty quickly. </pre> </blockquote> <pre wrap="">I assumed SSSD wasn't being used at all which begs the question: what is? nss_ldap? Is nslcd being used? What is hitting LDAP, only auth or something else (e.g. sudo, automount). rob </pre> <blockquote type="cite"> <blockquote type="cite"> <blockquote type="cite"> <blockquote type="cite"> <blockquote type="cite"> <pre wrap="">Since IPA is supposed to handle all options, I guess I am disappointed. regards ~J On 12/3/14 2:56 PM, Dmitri Pal wrote: </pre> <blockquote type="cite"> <pre wrap="">On 12/03/2014 04:40 PM, Janelle wrote: </pre> <blockquote type="cite"> <pre wrap="">Here is a bit of baffling one on 4.0.5: Replica install p11-kit??? </pre> </blockquote> <pre wrap="">This is a part of the DNSSEC set of packages. </pre> <blockquote type="cite"> <pre wrap="">Connection from master to replica is OK. Connection check OK p11-kit: ipa.p11-kit: x-public-key-info: invalid or unsupported attribute Configuring NTP daemon (ntpd) [1/4]: stopping ntpd [2/4]: writing configuration ... Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up. LDAP error: UNWILLING_TO_PERFORM database is read-only Thoughts? </pre> </blockquote> </blockquote> </blockquote> <pre wrap="">We need more information about your problem. As always, please start with information requested on <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.freeipa.org/page/Troubleshooting#Reporting_bugs">http://www.freeipa.org/page/Troubleshooting#Reporting_bugs</a> /var/log/ipa*.log from affected replica will be invaluable (along with exact package version numbers [including p11-kit] and repo configuration). </pre> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <fieldset class="mimeAttachmentHeader"></fieldset> </blockquote> <fieldset class="mimeAttachmentHeader"></fieldset> </blockquote> <fieldset class="mimeAttachmentHeader"></fieldset> </blockquote> </body> </html>