<html> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi Daryl, In fact the slow DS startup is due to slapi-nis priming: <tt> #0 0x00007f189a2689fc in strcmpi_fast #1 oc_find_nolock #2 0x00007f189a2699bd in va_expand_one_oc #3 0x00007f189a269d70 in schema_expand_objectclasses_ext #4 0x00007f189a26cbea in slapi_schema_expand_objectclasses #5 0x00007f189a2124a5 in slapi_str2entry #6 0x00007f188c38037d in backend_set_entry_from #7 0x00007f188c383316 in backend_shr_set_entry_cb #8 0x00007f189a26358d in send_ldap_search_entry_ext #9 0x00007f189a263dcc in send_ldap_search_entry #10 0x00007f189a240ad3 in iterate #11 0x00007f189a240c7a in send_results_ext #12 0x00007f189a24265e in op_shared_search #13 0x00007f189a2528de in search_internal_callback_pb #14 0x00007f188c387628 in backend_shr_set_config_entry_add #15 0x00007f188c3827ad in backend_set_config_entry_add_cb #16 0x00007f189a26358d in send_ldap_search_entry_ext #17 0x00007f189a263dcc in send_ldap_search_entry #18 0x00007f189a240ad3 in iterate #19 0x00007f189a240c7a in send_results_ext #20 0x00007f189a24265e in op_shared_search #21 0x00007f189a2528de in search_internal_callback_pb #22 0x00007f188c387cbb in backend_shr_startup #23 0x00007f188c394135 in plugin_startup #24 0x00007f189a24d847 in plugin_call_func #25 0x00007f189a24df78 in plugin_call_one #26 plugin_dependency_startall #27 0x00007f189a24e381 in plugin_startall #28 0x00007f189a716bc2 in main </tt> It lasts from Mon Mar 14 08:50:21 -> Mon Mar 14 08:51:17 CDT kadmin.service failed to start but the console log does not contain the exact time of the failure. Would you check if the failure occurred while DS was starting up ? If that is the case, like Alexander mentioned, it is already fixed in slapi-nis 0.55. thanks thierry <div class="moz-cite-prefix">On 03/14/2016 03:06 PM, Daryl Fonseca-Holt wrote: </div> <blockquote cite="mid:56E6C57D.3010203@umanitoba.ca" type="cite"> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> Hi Thierry, I moved the old logs into a subdirectory called try1. I did the recommended ipa-server-install --uninstall. Tried the replica install again. Failed during kadmind start like the previous time. The log from ipa-replica-install (with -d) is at <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://home.cc.umanitoba.ca/%7Efonsecah/ipa/ipareplica-install.log"><a class="moz-txt-link-freetext" href="http://home.cc.umanitoba.ca/~fonsecah/ipa/ipareplica-install.log">http://home.cc.umanitoba.ca/~fonsecah/ipa/ipareplica-install.log</a></a> The console script (mostly the same as the log but with my entries) is at <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://home.cc.umanitoba.ca/%7Efonsecah/ipa/ipa-replica-install.console">http://home.cc.umanitoba.ca/~fonsecah/ipa/ipa-replica-install.console</a> The 5 second pstacks are at <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://home.cc.umanitoba.ca/%7Efonsecah/ipa/ipa-replica-install.console">http://home.cc.umanitoba.ca/~fonsecah/ipa/slapd-pstacks.console</a> Thanks, Daryl <div class="moz-cite-prefix">On 03/11/16 02:40, thierry bordaz wrote: </div> <blockquote cite="mid:56E28490.6030406@redhat.com" type="cite"> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> Hello Deryl, <blockquote>My understanding is that ns-slapd is first slow to startup. Then when krb5kdc is starting it may load ns-slapd. We identified krb5kdc may be impacted by the number of users accounts. From the ns-slapd errors log it is not clear why it is so slow to start. Would you provide the ns-slapd access logs from that period. Also in order to know where ns-slapd is spending time, it would really help if you can get regular (each 5s) pstacks (with 389-ds-debuginfo), during DS startup and then later during krb5kdc startup. best regards thierry </blockquote> <div class="moz-cite-prefix">On 03/10/2016 11:10 PM, Daryl Fonseca-Holt wrote: </div> <blockquote cite="mid:56E1F0E4.9080605@umanitoba.ca" type="cite">Environment: RHEL 7.2 IPA 4.2.0-15 nss 3.19.1-19 389-ds-base 1.3.4.0-26 sssd 1.13.0-40 I've encountered this problem in IPA 3.0.0 but hoped it was addressed in 4.2.0. Trying to set up a replica of a master with 150,000+ user accounts, NIS and Schema Compatability enabled on the master. During ipa-replica-install it attempts to start IPA. dirsrv starts, krb5kdc starts, but then kadmind fails because krb5kdc has gone missing. This happens during restart of IPA in version 3.0.0 too. There it can be overcome by manually starting each component of IPA _but_ waiting until ns-slapd-<instance> has settled down (as seen from top) before starting krb5kdc. I also think that the startup of krb5kdc loads the LDAP instance quite a bit. There is a problem in the startup logic where dirsrv is so busy that even though krb5kdc successfully starts and allows the kadmin to begin kdb5kdc is not really able to do its duties. I'm reporting this since there must be some way to delay the start of krb5kdc and then kadmind until ns-slapd-<instance> is really open for business. # systemctl status krb5kdc.service ● krb5kdc.service - Kerberos 5 KDC Loaded: loaded (/usr/lib/systemd/system/krb5kdc.service; disabled; vendor preset: disabled) Active: inactive (dead) Mar 10 14:19:13 jutta.cc.umanitoba.ca systemd[1]: Stopped Kerberos 5 KDC. Mar 10 14:20:36 jutta.cc.umanitoba.ca systemd[1]: Starting Kerberos 5 KDC... Mar 10 14:20:39 jutta.cc.umanitoba.ca systemd[1]: Started Kerberos 5 KDC. # systemctl status krb5kdc.service ● krb5kdc.service - Kerberos 5 KDC Loaded: loaded (/usr/lib/systemd/system/krb5kdc.service; disabled; vendor preset: disabled) Active: inactive (dead) Mar 10 14:19:13 jutta.cc.umanitoba.ca systemd[1]: Stopped Kerberos 5 KDC. Mar 10 14:20:36 jutta.cc.umanitoba.ca systemd[1]: Starting Kerberos 5 KDC... Mar 10 14:20:39 jutta.cc.umanitoba.ca systemd[1]: Started Kerberos 5 KDC. journalctl -xe was stale by the time I got to it so I've attached /var/log/messages instead. The log from ipa-replica-install (with -d) is at <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://home.cc.umanitoba.ca/%7Efonsecah/ipa/ipareplica-install.log"><a class="moz-txt-link-freetext" href="http://home.cc.umanitoba.ca/~fonsecah/ipa/ipareplica-install.log">http://home.cc.umanitoba.ca/~fonsecah/ipa/ipareplica-install.log</a></a> The console script (mostly the same as the log but with my entries) is at <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://home.cc.umanitoba.ca/%7Efonsecah/ipa/ipa-replica-install.console">http://home.cc.umanitoba.ca/~fonsecah/ipa/ipa-replica-install.console</a> The /var/log/dirsrv/ns-slapd-<instance> access log is at <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://home.cc.umanitoba.ca/%7Efonsecah/ipa/access">http://home.cc.umanitoba.ca/~fonsecah/ipa/access</a> Regards, Daryl <fieldset class="mimeAttachmentHeader"></fieldset> </blockquote> </blockquote> <pre class="moz-signature" cols="72">-- -- Daryl Fonseca-Holt IST/CNS/Unix Server Team University of Manitoba 204.480.1079 </pre> </blockquote> </body> </html>