[Freeipa-users] ipa user-add slows down as more users are added

[Rob Crittenden]

Daryl Fonseca-Holt wrote:
> Hi All,
> 
> I am testing migration from NIS with a custom MySQL backend to IPA. In
> our testing ipa user-add starts out at around 12 seconds per user but
> slows down as more users are add. By 5000+ users it is taking 90+
> seconds. We have 120,000+ users. I'm looking at 155 days to load all the
> users :(
> 
> Per some performance tuning documentation I've increased
> nsslapd-cachememsize to 35,651,584 and am currently getting pretty high
> hit ratios (see below). However, one thread of ns-slapd pegs out core at
> 100% and I can't get get it to add users any faster. I'm not seeing any
> I/O or memory swapping.

The problem is most likely the default IPA users group. As it gets
humongous adding new members slows it down.

> Suggestions would be appreciated. Multi-master will probably help but
> with that many accounts it would take a lot of masters to get additions
> done to a resonable (45 seconds or less?) time. Is there any guideline
> for number of users per master?

IPA is multi-master. All users are on all masters.

If anything I'd expect that running imports on different masters would
slow things down as changes on multiple masters would need to get merged
together, particularly the default group.

Certainly bumping up the caches to match what the final expected sizes
is probably a good idea but I don't see it influencing import speed all
that much.

One idea I've had is to add the users in batches of 1000. What you'd do
is create 120 non-POSIX user groups, ipausers1..ipausers120, and add
them as members of ipausers.

Then for each batch change the default ipausers group with:

 $ ipa config-mod --defaultgroup=<name>

This should keep the user-add command fairly peppy and keep the ipausers
group somewhat in check via the nesting.

I imagine that the UI would blow up if you tried to view the ipausers
group as it tried to dereference 120k users.

You'll probably also want to disable the compat module for the import.

I assume you've already done some amount of testing with a smaller batch
of users to ensure they migrate ok, passwords work, etc?

rob