<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 09/04/2013 07:58 AM, John Moyer
wrote:<br>
</div>
<blockquote
cite="mid:8EF44D2E-1F0D-481C-B189-7319248B67A7@digitalreasoning.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
It was our opinion that it wasn't an index issue. I cleared the
logs from the IPA server, and then just ran a JIRA sync with the
server. I gave Rich the log file from my IPA for that sync. I
can't find the exact conversation, but we determined that JIRA was
connecting to LDAP some 1000 times or so to do the sync.</blockquote>
<br>
Right. For every single entry in IPA (user and group), JIRA LDAP
sync does - connect/bind/search/unbind/disconnect. This is horribly
inefficient, but it is what it is, and apparently other apps work
the same way (nexus? svn?), so this would be a good avenue to
investigate performance.<br>
<br>
<blockquote
cite="mid:8EF44D2E-1F0D-481C-B189-7319248B67A7@digitalreasoning.com"
type="cite">The logs didn't show but one search done that didn't
have an index which is why we concluded it wasn't an index issue.
<br>
</blockquote>
<br>
Adding indexing did help, but not much, and not nearly enough to
make the performance acceptable.<br>
<br>
<blockquote
cite="mid:8EF44D2E-1F0D-481C-B189-7319248B67A7@digitalreasoning.com"
type="cite">
<div><br>
</div>
<div apple-content-edited="true">
<div style="color: rgb(0, 0, 0); font-family: Helvetica;
font-size: medium; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height:
normal; orphans: 2; text-align: -webkit-auto; text-indent:
0px; text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; word-wrap: break-word;
-webkit-nbsp-mode: space; -webkit-line-break:
after-white-space; ">
<div style="color: rgb(0, 0, 0); font-family: Helvetica;
font-size: medium; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height:
normal; orphans: 2; text-align: -webkit-auto; text-indent:
0px; text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; word-wrap: break-word;
-webkit-nbsp-mode: space; -webkit-line-break:
after-white-space; ">
<div style="color: rgb(0, 0, 0); font-family: Helvetica;
font-size: medium; font-style: normal; font-variant:
normal; font-weight: normal; letter-spacing: normal;
line-height: normal; orphans: 2; text-align: -webkit-auto;
text-indent: 0px; text-transform: none; white-space:
normal; widows: 2; word-spacing: 0px;
-webkit-text-size-adjust: auto; -webkit-text-stroke-width:
0px; word-wrap: break-word; -webkit-nbsp-mode: space;
-webkit-line-break: after-white-space; ">
<div style="color: rgb(0, 0, 0); font-family: Helvetica;
font-size: medium; font-style: normal; font-variant:
normal; font-weight: normal; letter-spacing: normal;
line-height: normal; orphans: 2; text-align:
-webkit-auto; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; word-wrap: break-word;
-webkit-nbsp-mode: space; -webkit-line-break:
after-white-space; ">
<div style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal;
letter-spacing: normal; line-height: normal; orphans:
2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; word-wrap: break-word;
-webkit-nbsp-mode: space; -webkit-line-break:
after-white-space; ">
<div style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal;
letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent:
0px; text-transform: none; white-space: normal;
widows: 2; word-spacing: 0px;
-webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; word-wrap:
break-word; -webkit-nbsp-mode: space;
-webkit-line-break: after-white-space; ">
<div style="font-family: Helvetica; font-size:
medium; font-style: normal; font-variant: normal;
letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent:
0px; text-transform: none; white-space: normal;
widows: 2; word-spacing: 0px;
-webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; word-wrap:
break-word; -webkit-nbsp-mode: space;
-webkit-line-break: after-white-space; ">
<div style="color: rgb(0, 0, 0); font-weight:
normal; font-family: Calibri, sans-serif;
font-size: 14px; ">Thanks, </div>
<div style="color: rgb(0, 0, 0); font-weight:
normal; font-family: Calibri, sans-serif;
font-size: 14px; ">_____________________________________________________</div>
<div style="color: rgb(0, 0, 0); font-weight:
normal; font-family: Calibri, sans-serif;
font-size: 14px; ">John Moyer<br>
Director, IT Operations<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<div>
<div>On Sep 4, 2013, at 9:51 AM, Martin Kosek <<a
moz-do-not-send="true" href="mailto:mkosek@redhat.com">mkosek@redhat.com</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">Ah, ok. One of the reasons why I was
poking to this thread is exactly this<br>
ticket. It does not contain much information _what exactly_ is
making IPA<br>
performance poor - whether it is missing indices (which ones?)
or some issue<br>
in IPA plugins during binds, etc.<br>
<br>
Without more information, we do not know what to fix, what to
improve.<br>
<br>
Martin<br>
<br>
On 09/04/2013 02:01 PM, John Moyer wrote:<br>
<blockquote type="cite">Martin,<br>
<br>
I apologize there was a large offline conversation between
Rich and<br>
myself. Rich was kind enough to help me through some of my
issues. We<br>
did a lot more tests and poking and prodding. We
discovered that IPA is<br>
not as efficient when dealing with large number of
connections. Most of<br>
my load inefficiently reconnect to IPA over and over and
over and though<br>
LDAP can deal with this fairly efficiently, IPA apparently
drops to it's<br>
knees.<br>
<br>
A ticket was opened to addressed this issue.<br>
<br>
<a moz-do-not-send="true"
href="https://fedorahosted.org/freeipa/ticket/3892">https://fedorahosted.org/freeipa/ticket/3892</a><br>
<br>
<br>
Thanks,
_____________________________________________________ John
Moyer <br>
Director, IT Operations Digital Reasoning Systems, Inc. <br>
<a class="moz-txt-link-abbreviated" href="mailto:John.Moyer@digitalreasoning.com">John.Moyer@digitalreasoning.com</a> Office:<span
class="Apple-tab-span" style="white-space:pre"> </span>703.678.2311
Mobile:<span class="Apple-tab-span" style="white-space:pre">
</span>240.460.0023 <br>
Fax:<span class="Apple-tab-span" style="white-space:pre"> </span><span
class="Apple-tab-span" style="white-space:pre"> </span>703.678.2312
<a class="moz-txt-link-abbreviated" href="http://www.digitalreasoning.com">www.digitalreasoning.com</a><br>
<br>
On Sep 4, 2013, at 3:44 AM, Martin Kosek
<a class="moz-txt-link-rfc2396E" href="mailto:mkosek@redhat.com"><mkosek@redhat.com></a> wrote:<br>
<br>
<blockquote type="cite">On 08/30/2013 11:08 PM, John Moyer
wrote:<br>
<blockquote type="cite">Well IPA has machine entries on
some test clusters that I'm rolling<br>
IPA out on (20 machines maybe) but the user base is the
same (about 80<br>
~ 100) accounts with maybe 40 to 50 groups?<br>
<br>
I've stood up a clone of the jira server along with IPA.
I cleared<br>
my logs and then did the sync and ran the log analyzer
on it. These<br>
stats are pretty much ONLY for that jira sync I don't
have any other<br>
connections pointed to it.<br>
<br>
<br>
Start of Log: 30/Aug/2013:15:57:13 End of Log: <br>
30/Aug/2013:16:01:14<br>
<br>
Processed Log Time: Hours, 4 Minutes, 1 Seconds<br>
<br>
Restarts: 1 Total Connections:
824 SSL <br>
Connections: 824 Peak Concurrent
Connections: 6 Total <br>
Operations: 1806 Total Results:
1805<br>
Overall Performance: 99.9%<br>
<br>
Searches: 968 (4.02/sec)
(241.00/min) <br>
Modifications: 5 (0.02/sec)
(1.24/min) Adds: <br>
0 (0.00/sec) (0.00/min) Deletes:
0 <br>
(0.00/sec) (0.00/min) Mod RDNs: 0<br>
(0.00/sec) (0.00/min) Compares: 0<br>
(0.00/sec) (0.00/min) Binds: 833<br>
(3.46/sec) (207.39/min)<br>
<br>
Proxied Auth Operations: 0 Persistent Searches:
1<br>
Internal Operations: 0 Entry Operations:
0<br>
Extended Operations: 0 Abandoned Requests:
0 Smart<br>
Referrals Received: 0<br>
<br>
VLV Operations: 0 VLV Unindexed Searches:
0 SORT <br>
Operations: 0<br>
<br>
Entire Search Base Queries: 0 Unindexed Searches:
1<br>
<br>
</blockquote>
<br>
This looks like a promising way to find out the reason,
thanks John.<br>
However, I see just one unindexed search. Is the access
log complete?<br>
Previously I see that the sync takes 900 seconds/15
minutes, but there<br>
is only 4 minutes the access log. Note that it it may take
some time<br>
until the log is dumped.<br>
<br>
I think it would be also useful to run the analyzer with
"-ula" flags as<br>
Rob suggested earlier to find out the unindexed searches
(if any).<br>
<br>
What I find interesting is that JIRA does a lot of LDAP
BINDs. Can the <br>
problem be in longer BINDs than with than expected
(compared to for<br>
example plain LDAP servers)? Performance-wise, it would be
I think<br>
better if JIRA does just one BIND and run all the LDAP
searches the<br>
established connection. But I do not know if it can be
configured this<br>
way.<br>
<br>
Rich, Rob, I am wondering if the slow up is not really
caused by the<br>
binds, we have several DS plugins tied to the BIND
operation, it may be<br>
useful to analyze if they do not take too long.<br>
<br>
Martin<br>
</blockquote>
<br>
<br>
</blockquote>
<br>
</blockquote>
</div>
<br>
</blockquote>
<br>
</body>
</html>