[Freeipa-devel] Provisioning throughput

[Alexander Bokovoy]

On Thu, 26 May 2016, thierry bordaz wrote:
>
>
>On 05/25/2016 09:31 PM, Rob Crittenden wrote:
>>thierry bordaz wrote:
>>>
>>>
>>>On 05/25/2016 08:49 PM, Rob Crittenden wrote:
>>>>thierry bordaz wrote:
>>>>>
>>>>>Hello,
>>>>>
>>>>>Thanks for all the feedbacks. I updated the design accordingly 
>>>>>and with
>>>>>additional tests results
>>>>>(http://www.freeipa.org/page/V4/Performance_Improvements#Proposed_improvements)
>>>>>
>>>>>
>>>>>Several improvements can be done, in particular in DS plugins 
>>>>>(memberof,
>>>>>retroCL), but for "easy" benefit provisioning will be done 
>>>>>with memberof
>>>>>disabled followed by fixup.
>>>>>
>>>>>It remains some aspects that are not clear to me:
>>>>>
>>>>>  * For best performance, DS tuning and provisioning/fixup would
>>>>>    preferably be done under 'directory manager'
>>>>>    That means prompting DM password and writing it into 
>>>>>temporary file.
>>>>>    Is that a concern ?
>>>>>  * Fixup requires that we know the filters matching the provisioned
>>>>>    entries. For example :
>>>>>      o (objectClass=inetorgperson)
>>>>>      o (objectClass=ipausergroup)
>>>>>      o (objectClass=ipahost)
>>>>>      o (objectClass=ipahostgroup)
>>>>>      o (objectClass=ipasudorule)
>>>>>      o (objectClass=ipahbacrule)
>>>>>
>>>>>        The set of objectclass could be hardcode or provided in the
>>>>>        provisioning CLI option
>>>>>        What to do if an entry in in the provision file does not match
>>>>>        any of those filter ? Should it stop without starting the
>>>>>        provisioning ?
>>>>>  * The CLI doing the provisioning could be something like 'ipa
>>>>>    provision <options>' or should it be a separated command e.g.
>>>>>    ipa-bulk-load ?
>>>>
>>>>It depends. There is a migration command now, ipa migrate-ds, that
>>>>adds records and is impacted by this. There is also the possibility of
>>>>looping calls to ipa [user|group|etc]-add.
>>>
>>>I agree that migration and bulk load can be linked. If migration
>>>dump/update a set of entries before filling them into a new instance it
>>>could use bulk load.
>>>For set loop of ipa <object>-add, I think they add many others direct
>>>operations (mainly SRCH) before doing the ADD in order to check
>>>coherency. bulk load looks more straightforward.
>>
>>I just wonder if some (all) of this could be done manually. Document 
>>how to turn off memberof, do the import whatever way is appropriate, 
>>then run the fixup? I'm not sure what you had in mind.
>>
>>I don't want to think small but do we expect to be importing a slew 
>>of hosts, sudorules, etc? I guess the potential is there but would 
>>it be on the same scale as users? If you focus only on users/groups 
>>does that change the use case at all?
>>
>In fact, I am using such small scripts to prepare and run/monitor the 
>provisioning.
>If providing a set of scripts and document a procedure is enough I am 
>fine with this.
>
>>>>Would it be reasonable to require bulk import to be done on an IPA
>>>>master so we can leverage the ldapi socket?
>>>Do you mean using ldapi to reduce network latency or automember or
>>>something else ?
>>
>>To avoid the DM password issues. ldapi autobinds to DM when the id 
>>is root.
>
>Yes I said automember but was thinking to autobind.
>That is nice idea to avoid prompting DM password.
>In addition, slapi-nis participating to slowing down the provisioning 
>if it is using ldapi/DM slapi-nis will be offline by itself.
>
>The limitation would be to run the provisioning on IPA master. During 
>provisioning, membership attribute will be invalid (memberof not 
>computed). Is it acceptable that IPA master contains invalid 
>membership for some time ?
Consider provisioning to be at the same level as running
ipa-server-upgrade -- access via 389/636 ports is not allowed, LDAPI is
the only interface enabled which implies there would be no problem if we
set expectations right: provisioning mode is offline.
-- 
/ Alexander Bokovoy