<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 02/18/2015 11:05 PM, Jatin Nansi
wrote:<br>
</div>
<blockquote cite="mid:54E57D28.103@redhat.com" type="cite">
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
Check the ns-slapd access and error logs of the DS instance
hosting the IPA instance. The strace output indicates that
ns-slapd spent most of its time waiting for network activity to
happen (poll), which is normal for ns-slapd.<br>
</blockquote>
<br>
The number of open connections correlates to the CPU usage. Do
this:<br>
<br>
# ls -al /proc/`cat /var/run/dirsrv/slapd-MY-DOMAIN.pid`/fd|grep
socket|wc -l<br>
<br>
How many socket connections do you have?<br>
<br>
Also, it will be very useful to get some stack traces of the running
server to see what the various threads are doing. See
<a class="moz-txt-link-freetext" href="http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-hangs">http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-hangs</a><br>
<br>
<blockquote cite="mid:54E57D28.103@redhat.com" type="cite"> <br>
Jatin<br>
<div class="moz-cite-prefix">On 19/02/15 15:52, Jim Richard wrote:<br>
</div>
<blockquote
cite="mid:30BE1A3D-7170-485D-9A91-C868682450D9@placeiq.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
I’ve got 4 Redhat IDM masters in a multi-master
config. 3.0.0-42.el6.centos is the IPA version, 389-ds-base
version 1.2.11.15-48.el6_6, Centos 6.6
<div class=""><br class="">
</div>
<div class="">Monitoring established connections on port 389 and
dsInOps over time shows a consistent/even level of activity
however 2 of the 4 IPA servers show ever increasing CPI usage
by ns-slapd. One ns-slapd process will start to show increased
CPU for a time, then drop off as another then increases. This
cycle continues with each switch seeing more and more total
SPU usage by ns-slapd.</div>
<div class=""><br class="">
</div>
<div class="">strace timing for the offending ns-slapd looks
like the following:</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">% time seconds usecs/call calls
errors syscall</div>
<div class="">------ ----------- ----------- ---------
--------- ----------------</div>
<div class=""> 96.12 9.342272 1133 8243
poll</div>
<div class=""> 3.86 0.375457 53 7066
41 futex</div>
<div class=""> 0.01 0.000668 0 8244
8244 getpeername</div>
<div class=""> 0.00 0.000374 0 929
close</div>
<div class=""> 0.00 0.000368 0 3201
read</div>
<div class=""> 0.00 0.000151 0 882
setsockopt</div>
<div class=""> 0.00 0.000095 2 42
access</div>
<div class=""> 0.00 0.000033 0 1365
fcntl</div>
<div class=""> 0.00 0.000000 0 42
open</div>
<div class=""> 0.00 0.000000 0 39
stat</div>
<div class=""> 0.00 0.000000 0 42
fstat</div>
<div class=""> 0.00 0.000000 0 1
madvise</div>
<div class=""> 0.00 0.000000 0 441
accept</div>
<div class=""> 0.00 0.000000 0 441
getsockname</div>
<div class=""> 0.00 0.000000 0 1
restart_syscall</div>
<div class="">------ ----------- ----------- ---------
--------- ----------------</div>
<div class="">100.00 9.719418 30979
8285 total</div>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">I have carefully reviewed cn=config settings on
all four master servers to confirm that they match.</div>
<div class=""><br class="">
</div>
<div class="">Based on this strace output can you perhaps point
me in the right direction, give me a clue on what I should be
looking at.</div>
<div class=""><br class="">
</div>
<div class="">Here’s a screen shot of my Zabbix reporting to
help describe the problem. Note the graph in the bottom right
corner.</div>
<div class=""><br class="">
</div>
<div class="">The problem is most certainly related to
replication but I just don’t know what specifically to look
at.</div>
<div class=""><br class="">
</div>
<div class=""><img apple-width="yes" apple-height="yes"
apple-inline="yes" id="1A427CEE-BFF2-4759-8335-AA62EF79AC68"
src="cid:part1.04040701.00090201@redhat.com" class=""
height="713" width="1212"></div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Thanks in advance for any clues you can provide.<br
class="">
<div class="">
<div style="color: rgb(0, 0, 0); font-family: Verdana;
font-size: 11px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height:
normal; orphans: auto; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows:
auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"
class=""><br class="Apple-interchange-newline">
<br class="">
</div>
<div style="color: rgb(0, 0, 0); font-family: Verdana;
font-size: 11px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height:
normal; orphans: auto; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows:
auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"
class="">
<table style="color: rgb(34, 34, 34); margin: 0px;
font-size: 13.63636302948px; font-family: Times;
background-color: rgb(255, 255, 255);" class=""
cellpadding="0" cellspacing="0">
<tbody class="">
<tr class="" height="3">
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top;" class=""><br>
</td>
</tr>
</tbody><tbody class="">
<tr class="" height="3">
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(68, 54, 124);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(99, 80, 178);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(148, 145, 145);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(203, 203, 203);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(148, 145, 145);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(99, 80, 178);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(68, 54, 124);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(99, 80, 178);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(148, 145, 145);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(203, 203, 203);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(148, 145, 145);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(99, 80, 178);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br>
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class=""><br>
</td>
</tr>
</tbody><tbody class="">
<tr class="" height="10">
<td colspan="36" style="font-family: Georgia, serif;
margin: 0px; vertical-align: top; padding-top:
5px; font-style: italic; font-size: 10px;
letter-spacing: 0.1em; line-height: 1.5em; color:
rgb(117, 107, 172);" class=""><span style="color:
rgb(70, 70, 70);" class="">Jim Richard | </span><a
moz-do-not-send="true"
href="http://www.google.com/url?q=http%3A%2F%2Fwww.placeiq.com%2F&sa=D&sntz=1&usg=AFrqEzcYjZpDPyqW7feNK9EgLq-c9JlHiw"
target="_blank" style="color: rgb(117, 107,
172); text-decoration: none;" class="">PlaceIQ</a> |
Systems Administrator | jrichard<a
moz-do-not-send="true"
href="mailto:name@placeiq.com" target="_blank"
style="color: rgb(117, 107, 172);
text-decoration: none;" class="">@placeiq.com</a> | <a
moz-do-not-send="true" style="color: rgb(17, 85,
204);" class="">+1 (646) 338-8905</a></td>
</tr>
</tbody>
</table>
<div class=""><br class="">
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
</blockquote>
<br>
</body>
</html>