[Freeipa-devel] Provisioning throughput

[Alexander Bokovoy]

On Thu, 26 May 2016, thierry bordaz wrote:
>
>
>On 05/26/2016 09:32 AM, Alexander Bokovoy wrote:
>>On Wed, 25 May 2016, Rob Crittenden wrote:
>>>thierry bordaz wrote:
>>>>
>>>>
>>>>On 05/25/2016 08:49 PM, Rob Crittenden wrote:
>>>>>thierry bordaz wrote:
>>>>>>
>>>>>>Hello,
>>>>>>
>>>>>>Thanks for all the feedbacks. I updated the design 
>>>>>>accordingly and with
>>>>>>additional tests results
>>>>>>(http://www.freeipa.org/page/V4/Performance_Improvements#Proposed_improvements)
>>>>>>
>>>>>>
>>>>>>Several improvements can be done, in particular in DS 
>>>>>>plugins (memberof,
>>>>>>retroCL), but for "easy" benefit provisioning will be done 
>>>>>>with memberof
>>>>>>disabled followed by fixup.
>>>>>>
>>>>>>It remains some aspects that are not clear to me:
>>>>>>
>>>>>> * For best performance, DS tuning and provisioning/fixup would
>>>>>>   preferably be done under 'directory manager'
>>>>>>   That means prompting DM password and writing it into 
>>>>>>temporary file.
>>>>>>   Is that a concern ?
>>>>>> * Fixup requires that we know the filters matching the provisioned
>>>>>>   entries. For example :
>>>>>>     o (objectClass=inetorgperson)
>>>>>>     o (objectClass=ipausergroup)
>>>>>>     o (objectClass=ipahost)
>>>>>>     o (objectClass=ipahostgroup)
>>>>>>     o (objectClass=ipasudorule)
>>>>>>     o (objectClass=ipahbacrule)
>>>>>>
>>>>>>       The set of objectclass could be hardcode or provided in the
>>>>>>       provisioning CLI option
>>>>>>       What to do if an entry in in the provision file does not match
>>>>>>       any of those filter ? Should it stop without starting the
>>>>>>       provisioning ?
>>>>>> * The CLI doing the provisioning could be something like 'ipa
>>>>>>   provision <options>' or should it be a separated command e.g.
>>>>>>   ipa-bulk-load ?
>>>>>
>>>>>It depends. There is a migration command now, ipa migrate-ds, that
>>>>>adds records and is impacted by this. There is also the possibility of
>>>>>looping calls to ipa [user|group|etc]-add.
>>>>
>>>>I agree that migration and bulk load can be linked. If migration
>>>>dump/update a set of entries before filling them into a new instance it
>>>>could use bulk load.
>>>>For set loop of ipa <object>-add, I think they add many others direct
>>>>operations (mainly SRCH) before doing the ADD in order to check
>>>>coherency. bulk load looks more straightforward.
>>>
>>>I just wonder if some (all) of this could be done manually. 
>>>Document how to turn off memberof, do the import whatever way is 
>>>appropriate, then run the fixup? I'm not sure what you had in 
>>>mind.
>>>
>>>I don't want to think small but do we expect to be importing a 
>>>slew of hosts, sudorules, etc? I guess the potential is there but 
>>>would it be on the same scale as users? If you focus only on 
>>>users/groups does that change the use case at all?
>>I tend to agree with Rob on this. Maybe we should have a simple
>>script/update file that does preparatory work and another one that
>>returns configuration into the right state and document how to use them.
>
>Ok.
>rereading the thread I realize we are talking of 
>user/usergroups/host/hostgroup.
>
>Provisioning such entries is not that bad.
>For example 5Kusers/hosts are provisioned in 5min without memberof and 
>19min with memberof
>
>The real problem is provisioning sudorules and hbacrules where the 
>impact of memberof is very important.
>For example 100 sudorules are provisioned in 30s without memberof and 
>2h with memberof.
>
>Do you think provisioning should also considere sudorules/hbac or only 
>user/usergroups/host/hostgroup ?
I think it should consider all objects we support in the default
configuration.
-- 
/ Alexander Bokovoy