[Freeipa-users] ipa user-add slows down as more users are added

[Rich Megginson]

On 11/04/2015 04:07 PM, Rob Crittenden wrote:
> Daryl Fonseca-Holt wrote:
>> Hi All,
>>
>> I am testing migration from NIS with a custom MySQL backend to IPA. In
>> our testing ipa user-add starts out at around 12 seconds per user but
>> slows down as more users are add. By 5000+ users it is taking 90+
>> seconds. We have 120,000+ users. I'm looking at 155 days to load all the
>> users :(
>>
>> Per some performance tuning documentation I've increased
>> nsslapd-cachememsize to 35,651,584 and am currently getting pretty high
>> hit ratios (see below).

Use dbmon.sh instead of db_stat - it will give you the entry cache 
information as well as a summary of the db cache information (instead of 
the quite verbose db_stat output).

>> However, one thread of ns-slapd pegs out core at
>> 100% and I can't get get it to add users any faster. I'm not seeing any
>> I/O or memory swapping.
> The problem is most likely the default IPA users group. As it gets
> humongous adding new members slows it down.

So could he disable the automember and memberof plugins?  Then have 
those work offline, after the users are added?

>
>> Suggestions would be appreciated. Multi-master will probably help but
>> with that many accounts it would take a lot of masters to get additions
>> done to a resonable (45 seconds or less?) time. Is there any guideline
>> for number of users per master?
> IPA is multi-master. All users are on all masters.
>
> If anything I'd expect that running imports on different masters would
> slow things down as changes on multiple masters would need to get merged
> together, particularly the default group.

Right.

>
> Certainly bumping up the caches to match what the final expected sizes
> is probably a good idea but I don't see it influencing import speed all
> that much.

Right.

>
> One idea I've had is to add the users in batches of 1000. What you'd do
> is create 120 non-POSIX user groups, ipausers1..ipausers120, and add
> them as members of ipausers.
>
> Then for each batch change the default ipausers group with:
>
>   $ ipa config-mod --defaultgroup=<name>
>
> This should keep the user-add command fairly peppy and keep the ipausers
> group somewhat in check via the nesting.
>
> I imagine that the UI would blow up if you tried to view the ipausers
> group as it tried to dereference 120k users.
>
> You'll probably also want to disable the compat module for the import.
>
> I assume you've already done some amount of testing with a smaller batch
> of users to ensure they migrate ok, passwords work, etc?
>
> rob
>