[Freeipa-devel] Provisioning throughput

[Martin Basti]

On 26.05.2016 11:24, thierry bordaz wrote:
>
>
> On 05/26/2016 11:12 AM, Alexander Bokovoy wrote:
>> On Thu, 26 May 2016, thierry bordaz wrote:
>>>
>>>
>>> On 05/25/2016 09:31 PM, Rob Crittenden wrote:
>>>> thierry bordaz wrote:
>>>>>
>>>>>
>>>>> On 05/25/2016 08:49 PM, Rob Crittenden wrote:
>>>>>> thierry bordaz wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> Thanks for all the feedbacks. I updated the design accordingly 
>>>>>>> and with
>>>>>>> additional tests results
>>>>>>> (http://www.freeipa.org/page/V4/Performance_Improvements#Proposed_improvements) 
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Several improvements can be done, in particular in DS plugins 
>>>>>>> (memberof,
>>>>>>> retroCL), but for "easy" benefit provisioning will be done with 
>>>>>>> memberof
>>>>>>> disabled followed by fixup.
>>>>>>>
>>>>>>> It remains some aspects that are not clear to me:
>>>>>>>
>>>>>>>  * For best performance, DS tuning and provisioning/fixup would
>>>>>>>    preferably be done under 'directory manager'
>>>>>>>    That means prompting DM password and writing it into 
>>>>>>> temporary file.
>>>>>>>    Is that a concern ?
>>>>>>>  * Fixup requires that we know the filters matching the provisioned
>>>>>>>    entries. For example :
>>>>>>>      o (objectClass=inetorgperson)
>>>>>>>      o (objectClass=ipausergroup)
>>>>>>>      o (objectClass=ipahost)
>>>>>>>      o (objectClass=ipahostgroup)
>>>>>>>      o (objectClass=ipasudorule)
>>>>>>>      o (objectClass=ipahbacrule)
>>>>>>>
>>>>>>>        The set of objectclass could be hardcode or provided in the
>>>>>>>        provisioning CLI option
>>>>>>>        What to do if an entry in in the provision file does not 
>>>>>>> match
>>>>>>>        any of those filter ? Should it stop without starting the
>>>>>>>        provisioning ?
>>>>>>>  * The CLI doing the provisioning could be something like 'ipa
>>>>>>>    provision <options>' or should it be a separated command e.g.
>>>>>>>    ipa-bulk-load ?
>>>>>>
>>>>>> It depends. There is a migration command now, ipa migrate-ds, that
>>>>>> adds records and is impacted by this. There is also the 
>>>>>> possibility of
>>>>>> looping calls to ipa [user|group|etc]-add.
>>>>>
>>>>> I agree that migration and bulk load can be linked. If migration
>>>>> dump/update a set of entries before filling them into a new 
>>>>> instance it
>>>>> could use bulk load.
>>>>> For set loop of ipa <object>-add, I think they add many others direct
>>>>> operations (mainly SRCH) before doing the ADD in order to check
>>>>> coherency. bulk load looks more straightforward.
>>>>
>>>> I just wonder if some (all) of this could be done manually. 
>>>> Document how to turn off memberof, do the import whatever way is 
>>>> appropriate, then run the fixup? I'm not sure what you had in mind.
>>>>
>>>> I don't want to think small but do we expect to be importing a slew 
>>>> of hosts, sudorules, etc? I guess the potential is there but would 
>>>> it be on the same scale as users? If you focus only on users/groups 
>>>> does that change the use case at all?
>>>>
>>> In fact, I am using such small scripts to prepare and run/monitor 
>>> the provisioning.
>>> If providing a set of scripts and document a procedure is enough I 
>>> am fine with this.
>>>
>>>>>> Would it be reasonable to require bulk import to be done on an IPA
>>>>>> master so we can leverage the ldapi socket?
>>>>> Do you mean using ldapi to reduce network latency or automember or
>>>>> something else ?
>>>>
>>>> To avoid the DM password issues. ldapi autobinds to DM when the id 
>>>> is root.
>>>
>>> Yes I said automember but was thinking to autobind.
>>> That is nice idea to avoid prompting DM password.
>>> In addition, slapi-nis participating to slowing down the 
>>> provisioning if it is using ldapi/DM slapi-nis will be offline by 
>>> itself.
>>>
>>> The limitation would be to run the provisioning on IPA master. 
>>> During provisioning, membership attribute will be invalid (memberof 
>>> not computed). Is it acceptable that IPA master contains invalid 
>>> membership for some time ?
>> Consider provisioning to be at the same level as running
>> ipa-server-upgrade -- access via 389/636 ports is not allowed, LDAPI is
>> the only interface enabled which implies there would be no problem if we
>> set expectations right: provisioning mode is offline.
>
> Yes I agree, provisioning mode is offline.
> My concern is about side effects on the rest of the topology if we are 
> putting IPA master offline (is password update possible on replica ?).
>
>
> thierry
>

How long it takes until memberof data are recreated using replication? 
(IIRC and memberof attributes are not replicated)