[Freeipa-devel] Provisioning throughput

[thierry bordaz]

On 05/26/2016 11:26 AM, Martin Basti wrote:
>
>
> On 26.05.2016 11:24, thierry bordaz wrote:
>>
>>
>> On 05/26/2016 11:12 AM, Alexander Bokovoy wrote:
>>> On Thu, 26 May 2016, thierry bordaz wrote:
>>>>
>>>>
>>>> On 05/25/2016 09:31 PM, Rob Crittenden wrote:
>>>>> thierry bordaz wrote:
>>>>>>
>>>>>>
>>>>>> On 05/25/2016 08:49 PM, Rob Crittenden wrote:
>>>>>>> thierry bordaz wrote:
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> Thanks for all the feedbacks. I updated the design accordingly 
>>>>>>>> and with
>>>>>>>> additional tests results
>>>>>>>> (http://www.freeipa.org/page/V4/Performance_Improvements#Proposed_improvements) 
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Several improvements can be done, in particular in DS plugins 
>>>>>>>> (memberof,
>>>>>>>> retroCL), but for "easy" benefit provisioning will be done with 
>>>>>>>> memberof
>>>>>>>> disabled followed by fixup.
>>>>>>>>
>>>>>>>> It remains some aspects that are not clear to me:
>>>>>>>>
>>>>>>>>  * For best performance, DS tuning and provisioning/fixup would
>>>>>>>>    preferably be done under 'directory manager'
>>>>>>>>    That means prompting DM password and writing it into 
>>>>>>>> temporary file.
>>>>>>>>    Is that a concern ?
>>>>>>>>  * Fixup requires that we know the filters matching the 
>>>>>>>> provisioned
>>>>>>>>    entries. For example :
>>>>>>>>      o (objectClass=inetorgperson)
>>>>>>>>      o (objectClass=ipausergroup)
>>>>>>>>      o (objectClass=ipahost)
>>>>>>>>      o (objectClass=ipahostgroup)
>>>>>>>>      o (objectClass=ipasudorule)
>>>>>>>>      o (objectClass=ipahbacrule)
>>>>>>>>
>>>>>>>>        The set of objectclass could be hardcode or provided in the
>>>>>>>>        provisioning CLI option
>>>>>>>>        What to do if an entry in in the provision file does not 
>>>>>>>> match
>>>>>>>>        any of those filter ? Should it stop without starting the
>>>>>>>>        provisioning ?
>>>>>>>>  * The CLI doing the provisioning could be something like 'ipa
>>>>>>>>    provision <options>' or should it be a separated command e.g.
>>>>>>>>    ipa-bulk-load ?
>>>>>>>
>>>>>>> It depends. There is a migration command now, ipa migrate-ds, that
>>>>>>> adds records and is impacted by this. There is also the 
>>>>>>> possibility of
>>>>>>> looping calls to ipa [user|group|etc]-add.
>>>>>>
>>>>>> I agree that migration and bulk load can be linked. If migration
>>>>>> dump/update a set of entries before filling them into a new 
>>>>>> instance it
>>>>>> could use bulk load.
>>>>>> For set loop of ipa <object>-add, I think they add many others 
>>>>>> direct
>>>>>> operations (mainly SRCH) before doing the ADD in order to check
>>>>>> coherency. bulk load looks more straightforward.
>>>>>
>>>>> I just wonder if some (all) of this could be done manually. 
>>>>> Document how to turn off memberof, do the import whatever way is 
>>>>> appropriate, then run the fixup? I'm not sure what you had in mind.
>>>>>
>>>>> I don't want to think small but do we expect to be importing a 
>>>>> slew of hosts, sudorules, etc? I guess the potential is there but 
>>>>> would it be on the same scale as users? If you focus only on 
>>>>> users/groups does that change the use case at all?
>>>>>
>>>> In fact, I am using such small scripts to prepare and run/monitor 
>>>> the provisioning.
>>>> If providing a set of scripts and document a procedure is enough I 
>>>> am fine with this.
>>>>
>>>>>>> Would it be reasonable to require bulk import to be done on an IPA
>>>>>>> master so we can leverage the ldapi socket?
>>>>>> Do you mean using ldapi to reduce network latency or automember or
>>>>>> something else ?
>>>>>
>>>>> To avoid the DM password issues. ldapi autobinds to DM when the id 
>>>>> is root.
>>>>
>>>> Yes I said automember but was thinking to autobind.
>>>> That is nice idea to avoid prompting DM password.
>>>> In addition, slapi-nis participating to slowing down the 
>>>> provisioning if it is using ldapi/DM slapi-nis will be offline by 
>>>> itself.
>>>>
>>>> The limitation would be to run the provisioning on IPA master. 
>>>> During provisioning, membership attribute will be invalid (memberof 
>>>> not computed). Is it acceptable that IPA master contains invalid 
>>>> membership for some time ?
>>> Consider provisioning to be at the same level as running
>>> ipa-server-upgrade -- access via 389/636 ports is not allowed, LDAPI is
>>> the only interface enabled which implies there would be no problem 
>>> if we
>>> set expectations right: provisioning mode is offline.
>>
>> Yes I agree, provisioning mode is offline.
>> My concern is about side effects on the rest of the topology if we 
>> are putting IPA master offline (is password update possible on 
>> replica ?).
>>
>>
>> thierry
>>
>
> How long it takes until memberof data are recreated using replication? 
> (IIRC and memberof attributes are not replicated)

I have not precise data on what will be the replication latency.
IMHO provisioning will be "fast" on the server where we run the 
provisioning. Then the entries will be replicated to replicas where 
memberof is enabled. replication latency of 
users/usergroups/host/hostgroup should also be quite low, let say few 
minutes behind. Now the replication latency for entries like 
sudorules/hbacrules would be important, probably several hours.

thierry