[Freeipa-devel] Provisioning throughput

[thierry bordaz]

On 05/25/2016 09:31 PM, Rob Crittenden wrote:
> thierry bordaz wrote:
>>
>>
>> On 05/25/2016 08:49 PM, Rob Crittenden wrote:
>>> thierry bordaz wrote:
>>>>
>>>> Hello,
>>>>
>>>> Thanks for all the feedbacks. I updated the design accordingly and 
>>>> with
>>>> additional tests results
>>>> (http://www.freeipa.org/page/V4/Performance_Improvements#Proposed_improvements) 
>>>>
>>>>
>>>> Several improvements can be done, in particular in DS plugins 
>>>> (memberof,
>>>> retroCL), but for "easy" benefit provisioning will be done with 
>>>> memberof
>>>> disabled followed by fixup.
>>>>
>>>> It remains some aspects that are not clear to me:
>>>>
>>>>   * For best performance, DS tuning and provisioning/fixup would
>>>>     preferably be done under 'directory manager'
>>>>     That means prompting DM password and writing it into temporary 
>>>> file.
>>>>     Is that a concern ?
>>>>   * Fixup requires that we know the filters matching the provisioned
>>>>     entries. For example :
>>>>       o (objectClass=inetorgperson)
>>>>       o (objectClass=ipausergroup)
>>>>       o (objectClass=ipahost)
>>>>       o (objectClass=ipahostgroup)
>>>>       o (objectClass=ipasudorule)
>>>>       o (objectClass=ipahbacrule)
>>>>
>>>>         The set of objectclass could be hardcode or provided in the
>>>>         provisioning CLI option
>>>>         What to do if an entry in in the provision file does not match
>>>>         any of those filter ? Should it stop without starting the
>>>>         provisioning ?
>>>>   * The CLI doing the provisioning could be something like 'ipa
>>>>     provision <options>' or should it be a separated command e.g.
>>>>     ipa-bulk-load ?
>>>
>>> It depends. There is a migration command now, ipa migrate-ds, that
>>> adds records and is impacted by this. There is also the possibility of
>>> looping calls to ipa [user|group|etc]-add.
>>
>> I agree that migration and bulk load can be linked. If migration
>> dump/update a set of entries before filling them into a new instance it
>> could use bulk load.
>> For set loop of ipa <object>-add, I think they add many others direct
>> operations (mainly SRCH) before doing the ADD in order to check
>> coherency. bulk load looks more straightforward.
>
> I just wonder if some (all) of this could be done manually. Document 
> how to turn off memberof, do the import whatever way is appropriate, 
> then run the fixup? I'm not sure what you had in mind.
>
> I don't want to think small but do we expect to be importing a slew of 
> hosts, sudorules, etc? I guess the potential is there but would it be 
> on the same scale as users? If you focus only on users/groups does 
> that change the use case at all?
>
In fact, I am using such small scripts to prepare and run/monitor the 
provisioning.
If providing a set of scripts and document a procedure is enough I am 
fine with this.

>>> Would it be reasonable to require bulk import to be done on an IPA
>>> master so we can leverage the ldapi socket?
>> Do you mean using ldapi to reduce network latency or automember or
>> something else ?
>
> To avoid the DM password issues. ldapi autobinds to DM when the id is 
> root.

Yes I said automember but was thinking to autobind.
That is nice idea to avoid prompting DM password.
In addition, slapi-nis participating to slowing down the provisioning if 
it is using ldapi/DM slapi-nis will be offline by itself.

The limitation would be to run the provisioning on IPA master. During 
provisioning, membership attribute will be invalid (memberof not 
computed). Is it acceptable that IPA master contains invalid membership 
for some time ?

>
> rob
>