[Freeipa-devel] Provisioning throughput

[Martin Basti]

On 26.05.2016 11:12, Alexander Bokovoy wrote:
> On Thu, 26 May 2016, thierry bordaz wrote:
>>
>>
>> On 05/25/2016 09:31 PM, Rob Crittenden wrote:
>>> thierry bordaz wrote:
>>>>
>>>>
>>>> On 05/25/2016 08:49 PM, Rob Crittenden wrote:
>>>>> thierry bordaz wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> Thanks for all the feedbacks. I updated the design accordingly 
>>>>>> and with
>>>>>> additional tests results
>>>>>> (http://www.freeipa.org/page/V4/Performance_Improvements#Proposed_improvements) 
>>>>>>
>>>>>>
>>>>>>
>>>>>> Several improvements can be done, in particular in DS plugins 
>>>>>> (memberof,
>>>>>> retroCL), but for "easy" benefit provisioning will be done with 
>>>>>> memberof
>>>>>> disabled followed by fixup.
>>>>>>
>>>>>> It remains some aspects that are not clear to me:
>>>>>>
>>>>>>  * For best performance, DS tuning and provisioning/fixup would
>>>>>>    preferably be done under 'directory manager'
>>>>>>    That means prompting DM password and writing it into temporary 
>>>>>> file.
>>>>>>    Is that a concern ?
>>>>>>  * Fixup requires that we know the filters matching the provisioned
>>>>>>    entries. For example :
>>>>>>      o (objectClass=inetorgperson)
>>>>>>      o (objectClass=ipausergroup)
>>>>>>      o (objectClass=ipahost)
>>>>>>      o (objectClass=ipahostgroup)
>>>>>>      o (objectClass=ipasudorule)
>>>>>>      o (objectClass=ipahbacrule)
>>>>>>
>>>>>>        The set of objectclass could be hardcode or provided in the
>>>>>>        provisioning CLI option
>>>>>>        What to do if an entry in in the provision file does not 
>>>>>> match
>>>>>>        any of those filter ? Should it stop without starting the
>>>>>>        provisioning ?
>>>>>>  * The CLI doing the provisioning could be something like 'ipa
>>>>>>    provision <options>' or should it be a separated command e.g.
>>>>>>    ipa-bulk-load ?
>>>>>
>>>>> It depends. There is a migration command now, ipa migrate-ds, that
>>>>> adds records and is impacted by this. There is also the 
>>>>> possibility of
>>>>> looping calls to ipa [user|group|etc]-add.
>>>>
>>>> I agree that migration and bulk load can be linked. If migration
>>>> dump/update a set of entries before filling them into a new 
>>>> instance it
>>>> could use bulk load.
>>>> For set loop of ipa <object>-add, I think they add many others direct
>>>> operations (mainly SRCH) before doing the ADD in order to check
>>>> coherency. bulk load looks more straightforward.
>>>
>>> I just wonder if some (all) of this could be done manually. Document 
>>> how to turn off memberof, do the import whatever way is appropriate, 
>>> then run the fixup? I'm not sure what you had in mind.
>>>
>>> I don't want to think small but do we expect to be importing a slew 
>>> of hosts, sudorules, etc? I guess the potential is there but would 
>>> it be on the same scale as users? If you focus only on users/groups 
>>> does that change the use case at all?
>>>
>> In fact, I am using such small scripts to prepare and run/monitor the 
>> provisioning.
>> If providing a set of scripts and document a procedure is enough I am 
>> fine with this.
>>
>>>>> Would it be reasonable to require bulk import to be done on an IPA
>>>>> master so we can leverage the ldapi socket?
>>>> Do you mean using ldapi to reduce network latency or automember or
>>>> something else ?
>>>
>>> To avoid the DM password issues. ldapi autobinds to DM when the id 
>>> is root.
>>
>> Yes I said automember but was thinking to autobind.
>> That is nice idea to avoid prompting DM password.
>> In addition, slapi-nis participating to slowing down the provisioning 
>> if it is using ldapi/DM slapi-nis will be offline by itself.
>>
>> The limitation would be to run the provisioning on IPA master. During 
>> provisioning, membership attribute will be invalid (memberof not 
>> computed). Is it acceptable that IPA master contains invalid 
>> membership for some time ?
> Consider provisioning to be at the same level as running
> ipa-server-upgrade -- access via 389/636 ports is not allowed, LDAPI is
> the only interface enabled which implies there would be no problem if we
> set expectations right: provisioning mode is offline.
I agree

Martin^2