<html><head><meta http-equiv="Content-Type" content="text/html charset=iso-8859-1"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Sure, just let me know what needs to be run/applied. I've already rolled back to LDAP, so if the fix looks like it works I can then roll it out again.</div><br><div><div apple-content-edited="true">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div style="font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div style="font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div style="font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div style="color: rgb(0, 0, 0); font-weight: normal; font-family: Calibri, sans-serif; font-size: 14px; ">Thanks, </div><div style="color: rgb(0, 0, 0); font-weight: normal; font-family: Calibri, sans-serif; font-size: 14px; ">_____________________________________________________</div><div style="color: rgb(0, 0, 0); font-weight: normal; font-family: Calibri, sans-serif; font-size: 14px; ">John Moyer<br>Director, IT Operations</div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px; "><br></div></div></div></div></div></div></div></div></div><div><div>On Sep 4, 2013, at 9:12 AM, Dmitri Pal <<a href="mailto:dpal@redhat.com">dpal@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">
<meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type">
<div bgcolor="#FFFFFF" text="#000000">
On 09/04/2013 08:53 AM, John Moyer wrote:
<blockquote cite="mid:E525D771-F0FB-4604-832A-1BE8E874AFA4@digitalreasoning.com" type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
That summary is correct. The only thing I would add is that
other applications could easily bring the IPA server to it's knees
as well. <br>
</blockquote>
<br>
Yes this is what I meant. It is not only JIRA. Any client that
creates a lot of connections can cause problems.<br>
<br>
<blockquote cite="mid:E525D771-F0FB-4604-832A-1BE8E874AFA4@digitalreasoning.com" type="cite"> Our artifact server also did many connections per sec
when used, and one person doing a build could bring IPA to it's
knees as well. Also, not only would IPA be maxed at 100%, but
users would complain that their builds were taking longer than
normal (with or without the JIRA sync going, however, it was
obviously worse when JIRA was running).
<div><br>
</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>Also,
my IPA server was a larger/faster server than my LDAP server.
So my LDAP server would run circles around IPA even though it
was on a smaller machine. LDAP would run at about 10% maybe 15%
CPU when the JIRA sync ran. </div>
<div><br>
</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>IF
you need any other information let me know.</div>
</blockquote>
<br>
No this seems to be enough.<br>
Thank you.<br>
<br>
Would you be willing to test a fix if one is provided?<br>
<br>
Thanks<br>
Dmitri<br>
<br>
<blockquote cite="mid:E525D771-F0FB-4604-832A-1BE8E874AFA4@digitalreasoning.com" type="cite">
<div>
<div><br>
<div apple-content-edited="true">
<div style="font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div style="font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div style="font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div style="font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div style="font-family: Helvetica; font-size:
medium; font-style: normal; font-variant: normal;
letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent:
0px; text-transform: none; white-space: normal;
widows: 2; word-spacing: 0px;
-webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; word-wrap:
break-word; -webkit-nbsp-mode: space;
-webkit-line-break: after-white-space; ">
<div style="font-family: Helvetica; font-size:
medium; font-style: normal; font-variant:
normal; letter-spacing: normal; line-height:
normal; orphans: 2; text-align: -webkit-auto;
text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing:
0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; word-wrap:
break-word; -webkit-nbsp-mode: space;
-webkit-line-break: after-white-space; ">
<div style="font-family: Helvetica; font-size:
medium; font-style: normal; font-variant:
normal; letter-spacing: normal; line-height:
normal; orphans: 2; text-align: -webkit-auto;
text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing:
0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; word-wrap:
break-word; -webkit-nbsp-mode: space;
-webkit-line-break: after-white-space; ">
<div style="font-weight: normal; font-family: Calibri, sans-serif; font-size: 14px; ">Thanks, </div>
<div style="font-weight: normal; font-family: Calibri, sans-serif; font-size: 14px; ">_____________________________________________________</div>
<div style="font-weight: normal; font-family: Calibri, sans-serif; font-size: 14px; ">John Moyer<br>
Director, IT Operations</div>
<div style="font-weight: normal; font-family: Calibri, sans-serif; font-size: 14px; "><br>
</div>
<div style="font-weight: normal; font-family: Calibri, sans-serif; font-size: 14px; "><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>On Sep 4, 2013, at 8:32 AM, Dmitri Pal <<a moz-do-not-send="true" href="mailto:dpal@redhat.com">dpal@redhat.com</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type">
<div bgcolor="#FFFFFF" text="#000000"> On 09/04/2013 08:01
AM, John Moyer wrote:
<blockquote cite="mid:52822038-F299-46D2-8BD1-D45DA0D99EA6@digitalreasoning.com" type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
Martin,
<div><br>
</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>I apologize there
was a large offline conversation between Rich and
myself. Rich was kind enough to help me through
some of my issues. We did a lot more tests and
poking and prodding. We discovered that IPA is not
as efficient when dealing with large number of
connections. Most of my load inefficiently
reconnect to IPA over and over and over and though
LDAP can deal with this fairly efficiently, IPA
apparently drops to it's knees. </div>
<div><br>
</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>A ticket was
opened to addressed this issue.</div>
<div> </div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span><a moz-do-not-send="true" href="https://fedorahosted.org/freeipa/ticket/3892">https://fedorahosted.org/freeipa/ticket/3892</a></div>
<div><br>
</div>
<div><br>
</div>
</blockquote>
<br>
Thank you for reporting this ticket.<br>
Martin is investigating it and trying to see what is the
cause. The information mentioned above is missing from
the the ticket, thus the question.<br>
<br>
So to summarize: you identified that the cause of the
performance issue is that JIRA makes a lot of parallel
connections to LDAP server and IPA is slow processing
bind operations thus clients that do a lot of
connections can experience a low performance.<br>
<br>
Martin, I wonder if we can have a test that would just
do a lot of binds.<br>
There are a lot of plugins and one of the recent ones is
the OTP one. I wonder if we do too much during bind when
OTP is not enabled (by default).<br>
<br>
<blockquote cite="mid:52822038-F299-46D2-8BD1-D45DA0D99EA6@digitalreasoning.com" type="cite">
<div>
<div>
<div style="font-family: Helvetica; font-size:
medium; font-style: normal; font-variant:
normal; font-weight: normal; letter-spacing:
normal; line-height: normal; orphans: 2;
text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal;
widows: 2; word-spacing: 0px;
-webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; word-wrap:
break-word; -webkit-nbsp-mode: space;
-webkit-line-break: after-white-space; ">
<div style="font-family: Helvetica; font-size:
medium; font-style: normal; font-variant:
normal; font-weight: normal; letter-spacing:
normal; line-height: normal; orphans: 2;
text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal;
widows: 2; word-spacing: 0px;
-webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; word-wrap:
break-word; -webkit-nbsp-mode: space;
-webkit-line-break: after-white-space; ">
<div style="font-family: Helvetica; font-size:
medium; font-style: normal; font-variant:
normal; font-weight: normal; letter-spacing:
normal; line-height: normal; orphans: 2;
text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal;
widows: 2; word-spacing: 0px;
-webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; word-wrap:
break-word; -webkit-nbsp-mode: space;
-webkit-line-break: after-white-space; ">
<div style="font-family: Helvetica;
font-size: medium; font-style: normal;
font-variant: normal; font-weight: normal;
letter-spacing: normal; line-height:
normal; orphans: 2; text-align:
-webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal;
widows: 2; word-spacing: 0px;
-webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; word-wrap:
break-word; -webkit-nbsp-mode: space;
-webkit-line-break: after-white-space; ">
<div style="font-family: Helvetica;
font-size: medium; font-style: normal;
font-variant: normal; letter-spacing:
normal; line-height: normal; orphans: 2;
text-align: -webkit-auto; text-indent:
0px; text-transform: none; white-space:
normal; widows: 2; word-spacing: 0px;
-webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px;
word-wrap: break-word;
-webkit-nbsp-mode: space;
-webkit-line-break: after-white-space; ">
<div style="font-family: Helvetica;
font-size: medium; font-style: normal;
font-variant: normal; letter-spacing:
normal; line-height: normal; orphans:
2; text-align: -webkit-auto;
text-indent: 0px; text-transform:
none; white-space: normal; widows: 2;
word-spacing: 0px;
-webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px;
word-wrap: break-word;
-webkit-nbsp-mode: space;
-webkit-line-break: after-white-space;
">
<div style="font-family: Helvetica;
font-size: medium; font-style:
normal; font-variant: normal;
letter-spacing: normal; line-height:
normal; orphans: 2; text-align:
-webkit-auto; text-indent: 0px;
text-transform: none; white-space:
normal; widows: 2; word-spacing:
0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px;
word-wrap: break-word;
-webkit-nbsp-mode: space;
-webkit-line-break:
after-white-space; ">
<div style="font-weight: normal;
font-family: Calibri, sans-serif;
font-size: 14px; ">Thanks, </div>
<div style="font-weight: normal;
font-family: Calibri, sans-serif;
font-size: 14px; ">_____________________________________________________</div>
<div style="font-weight: normal;
font-family: Calibri, sans-serif;
font-size: 14px; ">John Moyer<br>
Director, IT Operations</div>
<div style="font-family: Calibri,
sans-serif; font-size: 14px; "><b>Digital
Reasoning Systems, Inc.</b></div>
<div style="font-family: Calibri,
sans-serif; font-size: 14px; "><a moz-do-not-send="true" href="mailto:john.moyer@digitalreasoning.com">John.Moyer@digitalreasoning.com</a></div>
<div style="font-weight: normal;
font-family: Calibri, sans-serif;
font-size: 14px; ">Office:<span class="Apple-tab-span" style="white-space: pre; "> </span>703.678.2311<br>
Mobile:<span class="Apple-tab-span" style="white-space: pre; "> </span>240.460.0023<br>
Fax:<span class="Apple-tab-span" style="white-space: pre; "> </span>703.678.2312<br>
</div>
<div style="font-weight: normal;
font-family: Calibri, sans-serif;
font-size: 14px; "><a moz-do-not-send="true" href="http://www.digitalreasoning.com/">www.digitalreasoning.com</a></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<div>
<div>On Sep 4, 2013, at 3:44 AM, Martin Kosek <<a moz-do-not-send="true" href="mailto:mkosek@redhat.com">mkosek@redhat.com</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">On 08/30/2013 11:08 PM,
John Moyer wrote:<br>
<blockquote type="cite">Well IPA has machine
entries on some test clusters that I'm rolling
IPA<br>
out on (20 machines maybe) but the user base
is the same (about 80 ~ 100)<br>
accounts with maybe 40 to 50 groups?<br>
<br>
I've stood up a clone of the jira server along
with IPA. I cleared my<br>
logs and then did the sync and ran the log
analyzer on it. These stats<br>
are pretty much ONLY for that jira sync I
don't have any other connections<br>
pointed to it.<br>
<br>
<br>
Start of Log: 30/Aug/2013:15:57:13 End of
Log:<br>
30/Aug/2013:16:01:14<br>
<br>
Processed Log Time: Hours, 4 Minutes, 1
Seconds<br>
<br>
Restarts: 1 Total
Connections: 824 SSL<br>
Connections: 824 Peak Concurrent
Connections: 6 Total<br>
Operations: 1806 Total Results:
1805 Overall<br>
Performance: 99.9%<br>
<br>
Searches: 968
(4.02/sec) (241.00/min) <br>
Modifications: 5
(0.02/sec) (1.24/min) Adds:<br>
0 (0.00/sec) (0.00/min) Deletes:
0<br>
(0.00/sec) (0.00/min) Mod RDNs:
0 (0.00/sec)<br>
(0.00/min) Compares: 0
(0.00/sec)<br>
(0.00/min) Binds: 833
(3.46/sec)<br>
(207.39/min)<br>
<br>
Proxied Auth Operations: 0 Persistent
Searches: 1 Internal<br>
Operations: 0 Entry Operations:
0 Extended<br>
Operations: 0 Abandoned Requests:
0 Smart Referrals<br>
Received: 0<br>
<br>
VLV Operations: 0 VLV Unindexed
Searches: 0 SORT<br>
Operations: 0<br>
<br>
Entire Search Base Queries: 0 Unindexed
Searches: 1<br>
<br>
</blockquote>
<br>
This looks like a promising way to find out the
reason, thanks John. However,<br>
I see just one unindexed search. Is the access
log complete? Previously I see<br>
that the sync takes 900 seconds/15 minutes, but
there is only 4 minutes the<br>
access log. Note that it it may take some time
until the log is dumped.<br>
<br>
I think it would be also useful to run the
analyzer with "-ula" flags as Rob<br>
suggested earlier to find out the unindexed
searches (if any).<br>
<br>
What I find interesting is that JIRA does a lot
of LDAP BINDs. Can the<br>
problem be in longer BINDs than with than
expected (compared to for example<br>
plain LDAP servers)? Performance-wise, it would
be I think better if JIRA<br>
does just one BIND and run all the LDAP searches
the established connection.<br>
But I do not know if it can be configured this
way.<br>
<br>
Rich, Rob, I am wondering if the slow up is not
really caused by the binds,<br>
we have several DS plugins tied to the BIND
operation, it may be useful to<br>
analyze if they do not take too long.<br>
<br>
Martin<br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Freeipa-users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Freeipa-users@redhat.com">Freeipa-users@redhat.com</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://www.redhat.com/mailman/listinfo/freeipa-users">https://www.redhat.com/mailman/listinfo/freeipa-users</a></pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Thank you,
Dmitri Pal
Sr. Engineering Manager for IdM portfolio
Red Hat Inc.
-------------------------------
Looking to carve out IT costs?
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="http://www.redhat.com/carveoutcosts/">www.redhat.com/carveoutcosts/</a>
</pre>
</div>
_______________________________________________<br>
Freeipa-users mailing list<br>
<a moz-do-not-send="true" href="mailto:Freeipa-users@redhat.com">Freeipa-users@redhat.com</a><br>
<a class="moz-txt-link-freetext" href="https://www.redhat.com/mailman/listinfo/freeipa-users">https://www.redhat.com/mailman/listinfo/freeipa-users</a></blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Thank you,
Dmitri Pal
Sr. Engineering Manager for IdM portfolio
Red Hat Inc.
-------------------------------
Looking to carve out IT costs?
<a class="moz-txt-link-abbreviated" href="http://www.redhat.com/carveoutcosts/">www.redhat.com/carveoutcosts/</a>
</pre>
</div>
</blockquote></div><br></div></body></html>