[Freeipa-users] Want faster user-add

[Petr Vobornik]

On 12/22/2015 10:24 AM, thierry bordaz wrote:
> On 12/21/2015 05:55 PM, Daryl Fonseca-Holt wrote:
>> Hi all,
>>
>> Environment: RHEL6 with IPA 3.0 at current RedHat level. 64-core
>> 256-GB RAM Oracle x4470 M2.
>>
>> During our migration from NIS on Solaris 140,000+ accounts will be
>> added. After tuning per the guides dbmon.sh shows no roevicts and we
>> get high cache hit ratios.
>>
>> Per a previous discussion with the list the input is broken down into
>> batches of less than 1,000 users and the default IPA group is changed
>> before each batch. This helped greatly.
>>
>> Adding all the users takes many hours. Initially ipa user-add takes an
>> average 2.3 seconds per user but degrades by the time there are
>> 140,000 users to an average 6.7 seconds per user.
>>
>> In tracing it appears that a significant portion of the time ipa
>> user-add takes is not the add itself, it is the query at the end that
>> displays the resulting user account. Is there any legit way to prevent
>> this query?
>>
>> The length of time it takes to migrate is not a big concern. The
>> concern is the start of the fall school term when we typically add
>> approximately 1,300 accounts per hour during the registration period
>> with our current system.
>>
>> All suggestions will be appreciated.
>>
>> Regards, Daryl
>>
> Hi Daryl,
>
> I can reproduce similar trend of user-add becoming slower and slower.
>
> Now in my tests (etime=7s) the time was spent half by authentication and
> half by ADD and MOD (update of ipausers group). I agree there are many
> direct SRCH (~10) but they all seems to be rapid.
>
> I know that the vast majority of the time is spent in DS schema-compat
> plugin. Disabling it, during provisioning, reduce the duration by ~3.
> Now I do not know if it is a valid option to disable this plugin during
> provisioning.
>
> thanks
> thierry

We must also distinguish performance on IPA 3.x (RHEL 6.x) and FreeIPA 
4.2/4.3

FreeIPA 4.2 got some performance improvements mostly related to group 
membership handling.

Improving user-add is one of primary goals for 4.4 release:
* https://fedorahosted.org/freeipa/ticket/5448

There are couple issues tracked about plugins output (also planned to be 
fixed in 4.4):
* https://fedorahosted.org/freeipa/ticket/5281
* https://fedorahosted.org/freeipa/ticket/5282
* https://fedorahosted.org/freeipa/ticket/4995

You can try to call user-add with --raw options but it won't help much 
because some parts ignore it. Other than that, there is not clean 
workaround.

When user is added, the user_add typically:
* adds the user to ipadefaultprimarygroup
* converts manger dn to human friedly value (should not cause perf. issues)
* set description to magic value to cause generation of user private 
group (calls user-mod)
* fetches password attributes and resolves ssh pub keys (does couple of 
searches)

An unsupported possibility - do on your own risk - is to remove these 
operations from user_add post_callback in ipalib/plugins/user.py (around 
line 535 on 6.7).

Other thing which might help is to remove 'memberof' and 
'memberofindirect' values from default_attributes in user.py (~line 
216). Note that it also affects other user-* commands.

All should be tried in testing environment.

Another performance improvement is to call IPA API directly and use 
batch command - with that it is possible to add e.g. 100 users with one 
call and save some network calls.

Example could be seen in this ugly script: 
https://pvoborni.fedorapeople.org/scripts/ipa-generate-users.sh
-- 
Petr Vobornik