[Freeipa-users] IPA Load Problems?
John Moyer
john.moyer at digitalreasoning.com
Wed Sep 4 13:20:01 UTC 2013
Sure, just let me know what needs to be run/applied. I've already rolled back to LDAP, so if the fix looks like it works I can then roll it out again.
Thanks,
_____________________________________________________
John Moyer
Director, IT Operations
On Sep 4, 2013, at 9:12 AM, Dmitri Pal <dpal at redhat.com> wrote:
> On 09/04/2013 08:53 AM, John Moyer wrote:
>>
>> That summary is correct. The only thing I would add is that other applications could easily bring the IPA server to it's knees as well.
>
> Yes this is what I meant. It is not only JIRA. Any client that creates a lot of connections can cause problems.
>
>> Our artifact server also did many connections per sec when used, and one person doing a build could bring IPA to it's knees as well. Also, not only would IPA be maxed at 100%, but users would complain that their builds were taking longer than normal (with or without the JIRA sync going, however, it was obviously worse when JIRA was running).
>>
>> Also, my IPA server was a larger/faster server than my LDAP server. So my LDAP server would run circles around IPA even though it was on a smaller machine. LDAP would run at about 10% maybe 15% CPU when the JIRA sync ran.
>>
>> IF you need any other information let me know.
>
> No this seems to be enough.
> Thank you.
>
> Would you be willing to test a fix if one is provided?
>
> Thanks
> Dmitri
>
>>
>> Thanks,
>> _____________________________________________________
>> John Moyer
>> Director, IT Operations
>>
>>
>> On Sep 4, 2013, at 8:32 AM, Dmitri Pal <dpal at redhat.com> wrote:
>>
>>> On 09/04/2013 08:01 AM, John Moyer wrote:
>>>>
>>>> Martin,
>>>>
>>>> I apologize there was a large offline conversation between Rich and myself. Rich was kind enough to help me through some of my issues. We did a lot more tests and poking and prodding. We discovered that IPA is not as efficient when dealing with large number of connections. Most of my load inefficiently reconnect to IPA over and over and over and though LDAP can deal with this fairly efficiently, IPA apparently drops to it's knees.
>>>>
>>>> A ticket was opened to addressed this issue.
>>>>
>>>> https://fedorahosted.org/freeipa/ticket/3892
>>>>
>>>>
>>>
>>> Thank you for reporting this ticket.
>>> Martin is investigating it and trying to see what is the cause. The information mentioned above is missing from the the ticket, thus the question.
>>>
>>> So to summarize: you identified that the cause of the performance issue is that JIRA makes a lot of parallel connections to LDAP server and IPA is slow processing bind operations thus clients that do a lot of connections can experience a low performance.
>>>
>>> Martin, I wonder if we can have a test that would just do a lot of binds.
>>> There are a lot of plugins and one of the recent ones is the OTP one. I wonder if we do too much during bind when OTP is not enabled (by default).
>>>
>>>> Thanks,
>>>> _____________________________________________________
>>>> John Moyer
>>>> Director, IT Operations
>>>> Digital Reasoning Systems, Inc.
>>>> John.Moyer at digitalreasoning.com
>>>> Office: 703.678.2311
>>>> Mobile: 240.460.0023
>>>> Fax: 703.678.2312
>>>> www.digitalreasoning.com
>>>>
>>>> On Sep 4, 2013, at 3:44 AM, Martin Kosek <mkosek at redhat.com> wrote:
>>>>
>>>>> On 08/30/2013 11:08 PM, John Moyer wrote:
>>>>>> Well IPA has machine entries on some test clusters that I'm rolling IPA
>>>>>> out on (20 machines maybe) but the user base is the same (about 80 ~ 100)
>>>>>> accounts with maybe 40 to 50 groups?
>>>>>>
>>>>>> I've stood up a clone of the jira server along with IPA. I cleared my
>>>>>> logs and then did the sync and ran the log analyzer on it. These stats
>>>>>> are pretty much ONLY for that jira sync I don't have any other connections
>>>>>> pointed to it.
>>>>>>
>>>>>>
>>>>>> Start of Log: 30/Aug/2013:15:57:13 End of Log:
>>>>>> 30/Aug/2013:16:01:14
>>>>>>
>>>>>> Processed Log Time: Hours, 4 Minutes, 1 Seconds
>>>>>>
>>>>>> Restarts: 1 Total Connections: 824 SSL
>>>>>> Connections: 824 Peak Concurrent Connections: 6 Total
>>>>>> Operations: 1806 Total Results: 1805 Overall
>>>>>> Performance: 99.9%
>>>>>>
>>>>>> Searches: 968 (4.02/sec) (241.00/min)
>>>>>> Modifications: 5 (0.02/sec) (1.24/min) Adds:
>>>>>> 0 (0.00/sec) (0.00/min) Deletes: 0
>>>>>> (0.00/sec) (0.00/min) Mod RDNs: 0 (0.00/sec)
>>>>>> (0.00/min) Compares: 0 (0.00/sec)
>>>>>> (0.00/min) Binds: 833 (3.46/sec)
>>>>>> (207.39/min)
>>>>>>
>>>>>> Proxied Auth Operations: 0 Persistent Searches: 1 Internal
>>>>>> Operations: 0 Entry Operations: 0 Extended
>>>>>> Operations: 0 Abandoned Requests: 0 Smart Referrals
>>>>>> Received: 0
>>>>>>
>>>>>> VLV Operations: 0 VLV Unindexed Searches: 0 SORT
>>>>>> Operations: 0
>>>>>>
>>>>>> Entire Search Base Queries: 0 Unindexed Searches: 1
>>>>>>
>>>>>
>>>>> This looks like a promising way to find out the reason, thanks John. However,
>>>>> I see just one unindexed search. Is the access log complete? Previously I see
>>>>> that the sync takes 900 seconds/15 minutes, but there is only 4 minutes the
>>>>> access log. Note that it it may take some time until the log is dumped.
>>>>>
>>>>> I think it would be also useful to run the analyzer with "-ula" flags as Rob
>>>>> suggested earlier to find out the unindexed searches (if any).
>>>>>
>>>>> What I find interesting is that JIRA does a lot of LDAP BINDs. Can the
>>>>> problem be in longer BINDs than with than expected (compared to for example
>>>>> plain LDAP servers)? Performance-wise, it would be I think better if JIRA
>>>>> does just one BIND and run all the LDAP searches the established connection.
>>>>> But I do not know if it can be configured this way.
>>>>>
>>>>> Rich, Rob, I am wondering if the slow up is not really caused by the binds,
>>>>> we have several DS plugins tied to the BIND operation, it may be useful to
>>>>> analyze if they do not take too long.
>>>>>
>>>>> Martin
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Freeipa-users mailing list
>>>> Freeipa-users at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>>
>>>
>>> --
>>> Thank you,
>>> Dmitri Pal
>>>
>>> Sr. Engineering Manager for IdM portfolio
>>> Red Hat Inc.
>>>
>>>
>>> -------------------------------
>>> Looking to carve out IT costs?
>>> www.redhat.com/carveoutcosts/
>>>
>>>
>>> _______________________________________________
>>> Freeipa-users mailing list
>>> Freeipa-users at redhat.com
>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>
>
>
> --
> Thank you,
> Dmitri Pal
>
> Sr. Engineering Manager for IdM portfolio
> Red Hat Inc.
>
>
> -------------------------------
> Looking to carve out IT costs?
> www.redhat.com/carveoutcosts/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20130904/45fd4718/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20130904/45fd4718/attachment.sig>
More information about the Freeipa-users
mailing list