[Freeipa-users] IPA Load Problems?

Wed Sep 4 14:02:47 UTC 2013

On 09/04/2013 07:58 AM, John Moyer wrote:
> It was our opinion that it wasn't an index issue.  I cleared the logs 
> from the IPA server, and then just ran a JIRA sync with the server.  I 
> gave Rich the log file from my IPA for that sync.  I can't find the 
> exact conversation, but we determined that JIRA was connecting to LDAP 
> some 1000 times or so to do the sync.

Right.  For every single entry in IPA (user and group), JIRA LDAP sync 
does - connect/bind/search/unbind/disconnect.  This is horribly 
inefficient, but it is what it is, and apparently other apps work the 
same way (nexus?  svn?), so this would be a good avenue to investigate 
performance.

> The logs didn't show but one search done that didn't have an index 
> which is why we concluded it wasn't an index issue.

Adding indexing did help, but not much, and not nearly enough to make 
the performance acceptable.

>
> Thanks,
> _____________________________________________________
> John Moyer
> Director, IT Operations
>
> On Sep 4, 2013, at 9:51 AM, Martin Kosek <mkosek at redhat.com 
> <mailto:mkosek at redhat.com>> wrote:
>
>> Ah, ok. One of the reasons why I was poking to this thread is exactly 
>> this
>> ticket. It does not contain much information _what exactly_ is making IPA
>> performance poor - whether it is missing indices (which ones?) or 
>> some issue
>> in IPA plugins during binds, etc.
>>
>> Without more information, we do not know what to fix, what to improve.
>>
>> Martin
>>
>> On 09/04/2013 02:01 PM, John Moyer wrote:
>>> Martin,
>>>
>>> I apologize there was a large offline conversation between Rich and
>>> myself.   Rich was kind enough to help me through some of my issues.  We
>>> did a lot more tests and poking and prodding.   We discovered that 
>>> IPA is
>>> not as efficient when dealing with large number of connections.  Most of
>>> my load inefficiently reconnect to IPA over and over and over and though
>>> LDAP can deal with this fairly efficiently, IPA apparently drops to it's
>>> knees.
>>>
>>> A ticket was opened to addressed this issue.
>>>
>>> https://fedorahosted.org/freeipa/ticket/3892
>>>
>>>
>>> Thanks, _____________________________________________________ John 
>>> Moyer
>>> Director, IT Operations Digital Reasoning Systems, Inc.
>>> John.Moyer at digitalreasoning.com Office:703.678.2311 Mobile:240.460.0023
>>> Fax:703.678.2312 www.digitalreasoning.com
>>>
>>> On Sep 4, 2013, at 3:44 AM, Martin Kosek <mkosek at redhat.com> wrote:
>>>
>>>> On 08/30/2013 11:08 PM, John Moyer wrote:
>>>>> Well IPA has machine entries on some test clusters that I'm rolling
>>>>> IPA out on (20 machines maybe) but the user base is the same (about 80
>>>>> ~ 100) accounts with maybe 40 to 50 groups?
>>>>>
>>>>> I've stood up a clone of the jira server along with IPA.   I cleared
>>>>> my logs and then did the sync and ran the log analyzer on it.   These
>>>>> stats are pretty much ONLY for that jira sync I don't have any other
>>>>> connections pointed to it.
>>>>>
>>>>>
>>>>> Start of Log:    30/Aug/2013:15:57:13 End of Log:
>>>>> 30/Aug/2013:16:01:14
>>>>>
>>>>> Processed Log Time:   Hours, 4 Minutes, 1 Seconds
>>>>>
>>>>> Restarts:                     1 Total Connections:            824 SSL
>>>>> Connections:              824 Peak Concurrent Connections:  6 Total
>>>>> Operations:             1806 Total Results:                1805
>>>>> Overall Performance:          99.9%
>>>>>
>>>>> Searches:                     968        (4.02/sec)  (241.00/min)
>>>>> Modifications:                5          (0.02/sec)  (1.24/min) Adds:
>>>>> 0          (0.00/sec)  (0.00/min) Deletes:                      0
>>>>> (0.00/sec)  (0.00/min) Mod RDNs:                     0
>>>>> (0.00/sec) (0.00/min) Compares:                     0
>>>>> (0.00/sec) (0.00/min) Binds:                        833
>>>>> (3.46/sec) (207.39/min)
>>>>>
>>>>> Proxied Auth Operations:      0 Persistent Searches:          1
>>>>> Internal Operations:          0 Entry Operations:             0
>>>>> Extended Operations:          0 Abandoned Requests:           0 Smart
>>>>> Referrals Received:     0
>>>>>
>>>>> VLV Operations:               0 VLV Unindexed Searches:       0 SORT
>>>>> Operations:              0
>>>>>
>>>>> Entire Search Base Queries:   0 Unindexed Searches:           1
>>>>>
>>>>
>>>> This looks like a promising way to find out the reason, thanks John.
>>>> However, I see just one unindexed search. Is the access log complete?
>>>> Previously I see that the sync takes 900 seconds/15 minutes, but there
>>>> is only 4 minutes the access log. Note that it it may take some time
>>>> until the log is dumped.
>>>>
>>>> I think it would be also useful to run the analyzer with "-ula" 
>>>> flags as
>>>> Rob suggested earlier to find out the unindexed searches (if any).
>>>>
>>>> What I find interesting is that JIRA does a lot of LDAP BINDs. Can the
>>>> problem be in longer BINDs than with than expected (compared to for
>>>> example plain LDAP servers)? Performance-wise, it would be I think
>>>> better if JIRA does just one BIND and run all the LDAP searches the
>>>> established connection. But I do not know if it can be configured this
>>>> way.
>>>>
>>>> Rich, Rob, I am wondering if the slow up is not really caused by the
>>>> binds, we have several DS plugins tied to the BIND operation, it may be
>>>> useful to analyze if they do not take too long.
>>>>
>>>> Martin
>>>
>>>
>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20130904/3c5b65aa/attachment.htm>