[Freeipa-users] SOLVED Fwd: Re: ipa user-add slows down as more users are added

[Daryl Fonseca-Holt]

Hi all,

Splitting ipausers helped ipa user-add speed a lot. Two other things helped:

1) Setting nsslapd-cachememsize much larger than the size of the 
id2entry file
2) Increasing the size of the DN cache size, nsslapd-ndn-cache-max-size

I've got 60,000+ users now and user-add only takes slightly over three 
seconds.

Because we will continue to add large number of users at the start of 
the University's fall term and the new userids will be sequential I 
chose to use a prime number,  211, of ipauser groups and hash the 
userid. Thus the new ones will spread evenly over the groups and we 
should end up with less than 1,000 users per group.

Thanks again for your help,
Daryl

-------- Forwarded Message --------
Subject: 	Re: [Freeipa-users] ipa user-add slows down as more users are 
added
Date: 	Thu, 5 Nov 2015 16:37:29 -0600
From: 	Daryl Fonseca-Holt <Daryl.Fonseca-Holt at umanitoba.ca>
To: 	Rich Megginson <rmeggins at redhat.com>



On 11/04/15 17:25, Rich Megginson wrote:
> On 11/04/2015 04:07 PM, Rob Crittenden wrote:
>> Daryl Fonseca-Holt wrote:
>>> Hi All,
>>>
>>> I am testing migration from NIS with a custom MySQL backend to IPA. In
>>> our testing ipa user-add starts out at around 12 seconds per user but
>>> slows down as more users are add. By 5000+ users it is taking 90+
>>> seconds. We have 120,000+ users. I'm looking at 155 days to load all
>>> the
>>> users :(
>>>
>>> Per some performance tuning documentation I've increased
>>> nsslapd-cachememsize to 35,651,584 and am currently getting pretty high
>>> hit ratios (see below).
>
> Use dbmon.sh instead of db_stat - it will give you the entry cache
> information as well as a summary of the db cache information (instead
> of the quite verbose db_stat output).
>
Thanks for the tip. I've got some tuning to do. My dbcachefree is
negative and the roevicts are high.
>>> However, one thread of ns-slapd pegs out core at
>>> 100% and I can't get get it to add users any faster. I'm not seeing any
>>> I/O or memory swapping.
>> The problem is most likely the default IPA users group. As it gets
>> humongous adding new members slows it down.
>
> So could he disable the automember and memberof plugins?  Then have
> those work offline, after the users are added?
>
>>
>>> Suggestions would be appreciated. Multi-master will probably help but
>>> with that many accounts it would take a lot of masters to get additions
>>> done to a resonable (45 seconds or less?) time. Is there any guideline
>>> for number of users per master?
>> IPA is multi-master. All users are on all masters.
>>
>> If anything I'd expect that running imports on different masters would
>> slow things down as changes on multiple masters would need to get merged
>> together, particularly the default group.
>
> Right.
>
>>
>> Certainly bumping up the caches to match what the final expected sizes
>> is probably a good idea but I don't see it influencing import speed all
>> that much.
>
> Right.
>
>>
>> One idea I've had is to add the users in batches of 1000. What you'd do
>> is create 120 non-POSIX user groups, ipausers1..ipausers120, and add
>> them as members of ipausers.
>>
>> Then for each batch change the default ipausers group with:
>>
>>   $ ipa config-mod --defaultgroup=<name>
>>
>> This should keep the user-add command fairly peppy and keep the ipausers
>> group somewhat in check via the nesting.

I'm doing this but with a twist. I have an existing user base so I'm
going to use
the existing UID to hash to one of the ipauser<n> groups. Will do
round-robin
for future adds. Being an education institution we get a lot of new
accounts each fall.

>>
>> I imagine that the UI would blow up if you tried to view the ipausers
>> group as it tried to dereference 120k users.
>>
>> You'll probably also want to disable the compat module for the import.
>>
I'll give this a try too.
>> I assume you've already done some amount of testing with a smaller batch
>> of users to ensure they migrate ok, passwords work, etc?
>>
>> rob
>>
>

Thanks to both of you. It will be a while before I can report the fruits
of this
exercise. I'm reluctant to give up the 5000+ that I've already added so I'm
deleting them from ipauser and adding them to the appropriate ipauser<n>.

Meantime I've got a bunch of other scripts to write for some of the
custom stuff
we did with PAM and the old MySQL backend.

Regards,
Daryl

-- 
  --
  Daryl Fonseca-Holt
  IST/CNS/Unix Server Team
  University of Manitoba
  204.480.1079



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20151117/21601da0/attachment.htm>