<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hi Daryl,<br>
<br>
Thanks for all the data. I will look at the pstacks. A first look
shows that you capture import, bind... so may be a complete
ipa-replica-install session.<br>
I will try to retrieve the specific startup time to see what was
going on at that time.<br>
If you have the time to monitor only startup, it will help me
shrinking the set of pstacks.<br>
Startup of DS last > 1min. If you may start DS and as soon as the
ns-slapd process is launched, do regular pstacks. Then when you are
able to send a simple ldapsearch (ldapsearch -x -b "" -s base), you
may stop taking pstacks.<br>
<br>
thanks<br>
thierry<br>
<br>
<div class="moz-cite-prefix">On 03/14/2016 03:06 PM, Daryl
Fonseca-Holt wrote:<br>
</div>
<blockquote cite="mid:56E6C57D.3010203@umanitoba.ca" type="cite">
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
Hi Thierry,<br>
<br>
I moved the old logs into a subdirectory called try1. I did the
recommended ipa-server-install --uninstall. Tried the replica
install again. Failed during kadmind start like the previous time.<br>
<br>
The log from ipa-replica-install (with -d) is at <a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://home.cc.umanitoba.ca/%7Efonsecah/ipa/ipareplica-install.log"><a class="moz-txt-link-freetext" href="http://home.cc.umanitoba.ca/~fonsecah/ipa/ipareplica-install.log">http://home.cc.umanitoba.ca/~fonsecah/ipa/ipareplica-install.log</a></a>
<br>
The console script (mostly the same as the log but with my
entries) is at <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="http://home.cc.umanitoba.ca/%7Efonsecah/ipa/ipa-replica-install.console">http://home.cc.umanitoba.ca/~fonsecah/ipa/ipa-replica-install.console</a>
<br>
The 5 second pstacks are at <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="http://home.cc.umanitoba.ca/%7Efonsecah/ipa/ipa-replica-install.console">http://home.cc.umanitoba.ca/~fonsecah/ipa/slapd-pstacks.console</a><br>
<br>
Thanks, Daryl<br>
<br>
<br>
<div class="moz-cite-prefix">On 03/11/16 02:40, thierry bordaz
wrote:<br>
</div>
<blockquote cite="mid:56E28490.6030406@redhat.com" type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
Hello Deryl,<br>
<br>
<blockquote>My understanding is that ns-slapd is first slow to
startup. Then when krb5kdc is starting it may load ns-slapd.<br>
<br>
We identified krb5kdc may be impacted by the number of users
accounts.<br>
From the ns-slapd errors log it is not clear why it is so slow
to start.<br>
<br>
Would you provide the ns-slapd access logs from that period.<br>
Also in order to know where ns-slapd is spending time, it
would really help if you can get regular (each 5s) pstacks
(with 389-ds-debuginfo), during DS startup and then later
during krb5kdc startup.<br>
<br>
best regards<br>
thierry<br>
</blockquote>
<br>
<div class="moz-cite-prefix">On 03/10/2016 11:10 PM, Daryl
Fonseca-Holt wrote:<br>
</div>
<blockquote cite="mid:56E1F0E4.9080605@umanitoba.ca" type="cite">Environment:
<br>
RHEL 7.2 <br>
IPA 4.2.0-15 <br>
nss 3.19.1-19 <br>
389-ds-base 1.3.4.0-26 <br>
sssd 1.13.0-40 <br>
<br>
<br>
I've encountered this problem in IPA 3.0.0 but hoped it was
addressed in 4.2.0. <br>
<br>
Trying to set up a replica of a master with 150,000+ user
accounts, NIS and Schema Compatability enabled on the master.
<br>
<br>
During ipa-replica-install it attempts to start IPA. dirsrv
starts, krb5kdc starts, but then kadmind fails because krb5kdc
has gone missing. <br>
<br>
This happens during restart of IPA in version 3.0.0 too. There
it can be overcome by manually starting each component of IPA
_but_ waiting until ns-slapd-<instance> has settled down
(as seen from top) before starting krb5kdc. I also think that
the startup of krb5kdc loads the LDAP instance quite a bit. <br>
<br>
There is a problem in the startup logic where dirsrv is so
busy that even though krb5kdc successfully starts and allows
the kadmin to begin kdb5kdc is not really able to do its
duties. <br>
<br>
I'm reporting this since there must be some way to delay the
start of krb5kdc and then kadmind until
ns-slapd-<instance> is really open for business. <br>
<br>
# systemctl status krb5kdc.service <br>
● krb5kdc.service - Kerberos 5 KDC <br>
Loaded: loaded (/usr/lib/systemd/system/krb5kdc.service;
disabled; vendor preset: disabled) <br>
Active: inactive (dead) <br>
<br>
Mar 10 14:19:13 jutta.cc.umanitoba.ca systemd[1]: Stopped
Kerberos 5 KDC. <br>
Mar 10 14:20:36 jutta.cc.umanitoba.ca systemd[1]: Starting
Kerberos 5 KDC... <br>
Mar 10 14:20:39 jutta.cc.umanitoba.ca systemd[1]: Started
Kerberos 5 KDC. <br>
<br>
# systemctl status krb5kdc.service <br>
● krb5kdc.service - Kerberos 5 KDC <br>
Loaded: loaded (/usr/lib/systemd/system/krb5kdc.service;
disabled; vendor preset: disabled) <br>
Active: inactive (dead) <br>
<br>
Mar 10 14:19:13 jutta.cc.umanitoba.ca systemd[1]: Stopped
Kerberos 5 KDC. <br>
Mar 10 14:20:36 jutta.cc.umanitoba.ca systemd[1]: Starting
Kerberos 5 KDC... <br>
Mar 10 14:20:39 jutta.cc.umanitoba.ca systemd[1]: Started
Kerberos 5 KDC. <br>
<br>
journalctl -xe was stale by the time I got to it so I've
attached /var/log/messages instead. <br>
<br>
The log from ipa-replica-install (with -d) is at <a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://home.cc.umanitoba.ca/%7Efonsecah/ipa/ipareplica-install.log"><a class="moz-txt-link-freetext" href="http://home.cc.umanitoba.ca/~fonsecah/ipa/ipareplica-install.log">http://home.cc.umanitoba.ca/~fonsecah/ipa/ipareplica-install.log</a></a>
<br>
The console script (mostly the same as the log but with my
entries) is at <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="http://home.cc.umanitoba.ca/%7Efonsecah/ipa/ipa-replica-install.console">http://home.cc.umanitoba.ca/~fonsecah/ipa/ipa-replica-install.console</a>
<br>
The /var/log/dirsrv/ns-slapd-<instance> access log is at
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://home.cc.umanitoba.ca/%7Efonsecah/ipa/access">http://home.cc.umanitoba.ca/~fonsecah/ipa/access</a>
<br>
<br>
Regards, Daryl <br>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
</blockquote>
<br>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
--
Daryl Fonseca-Holt
IST/CNS/Unix Server Team
University of Manitoba
204.480.1079
</pre>
</blockquote>
<br>
</body>
</html>