[Freeipa-devel] IPA Server upgrade 4.2 design

Tue Mar 3 10:00:45 UTC 2015

On 03/03/15 10:55, Jan Cholasta wrote:
> Dne 3.3.2015 v 09:55 Martin Basti napsal(a):
>> On 03/03/15 09:33, Jan Cholasta wrote:
>>> Dne 3.3.2015 v 09:06 Martin Basti napsal(a):
>>>> On 03/03/15 07:31, Jan Cholasta wrote:
>>>>> Dne 2.3.2015 v 13:51 Martin Basti napsal(a):
>>>>>> On 02/03/15 13:12, Jan Cholasta wrote:
>>>>>>> Dne 2.3.2015 v 12:23 Martin Kosek napsal(a):
>>>>>>>> On 03/02/2015 07:49 AM, Jan Cholasta wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Dne 24.2.2015 v 19:10 Martin Basti napsal(a):
>>>>>>>>>> Hello all,
>>>>>>>>>>
>>>>>>>>>> please read the design page, any objections/suggestions
>>>>>>>>>> appreciated
>>>>>>>>>> http://www.freeipa.org/page/V4/Server_Upgrade_Refactoring
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 1)
>>>>>>>>>
>>>>>>>>> "
>>>>>>>>> * Merge server update commands into the one command
>>>>>>>>> (ipa-server-upgrade)
>>>>>>>>> "
>>>>>>>>>
>>>>>>>>> So there is "ipa-server-install" to install the server,
>>>>>>>>> "ipa-server-install
>>>>>>>>> --uninstall" to uninstall the server and "ipa-server-upgrade" to
>>>>>>>>> upgrade the
>>>>>>>>> server. Maybe we should bring some consistency here and have one
>>>>>>>>> of:
>>>>>>>>>
>>>>>>>>>   a) "ipa-server-install [--install]", "ipa-server-install
>>>>>>>>> --uninstall",
>>>>>>>>> "ipa-server-install --upgrade"
>>>>>>>>>
>>>>>>>>>   b) "ipa-server-install [install]", "ipa-server-install
>>>>>>>>> uninstall",
>>>>>>>>> "ipa-server-install upgrade"
>>>>>>>>>
>>>>>>>>>   c) "ipa-server-install", "ipa-server-uninstall",
>>>>>>>>> "ipa-server-upgrade"
>>>>>>>>
>>>>>>>> Long term, I think we want C. Besides other advantages, it will 
>>>>>>>> let
>>>>>>>> us have
>>>>>>>> independent sets of options, based on what you want to do.
>>>>>>>>
>>>>>>>>> 2)
>>>>>>>>>
>>>>>>>>> "
>>>>>>>>> * Prevent to run IPA service, if code version and configuration
>>>>>>>>> version does
>>>>>>>>> not match
>>>>>>>>>    * ipactl should execute ipa-server-upgrade if needed
>>>>>>>>> "
>>>>>>>>>
>>>>>>>>> There should be no configuration version, configuration update
>>>>>>>>> should be run
>>>>>>>>> always. It's fast and hence does not need to be optimized like 
>>>>>>>>> data
>>>>>>>>> update by
>>>>>>>>> using a monolithic version number, which brings more than a few
>>>>>>>>> problems on its
>>>>>>>>> own.
>>>>>>>>
>>>>>>>> I do not agree in this section. Why would you like to run it 
>>>>>>>> always,
>>>>>>>> even if it
>>>>>>>> was fast? No run is still faster than fast run.
>>>>>>>
>>>>>>> In the ideal case the installer would be idempotent and upgrade 
>>>>>>> would
>>>>>>> be re-running the installer and we should aim to do just that. We
>>>>>>> kind
>>>>>>> of do that already, but there is a lot of code duplication in
>>>>>>> installers and ipa-upgradeconfig (I would like to fix that when
>>>>>>> refactoring installers). IMO it's better to always make 100% 
>>>>>>> sure the
>>>>>>> configuration is correct rather than to save a second or two.
>>>>>> I doesn't like this idea, if user wants to fix something, the one
>>>>>> should
>>>>>> use --skip-version-check option, and the IPA upgrade will be 
>>>>>> executed.
>>>>>
>>>>> Well, what I don't like is dealing with meaningless version numbers.
>>>>> They are causing us grief in API versioning and I don't see why it
>>>>> would be any different here.
>>>> However you must keep the version because of schema and data 
>>>> upgrade, so
>>>> why not to execute update as one batch instead of doing config upgrade
>>>> all the time, and then data upgrade only if required.
>>>
>>> Because there is no exact mapping between version number and what
>>> features are actually available. A state file is tons better than a
>>> single version number.
>>>
>>>>
>>>> Some configuration upgrades, like adding new DNS related services,
>>>> requires new schema, how we can handle this?
>>>
>>> This does not sound right. Could you be more specific?
>> at least ipa-dnskeysyncd service, requires updated schema for keys
>> metadata.
>> This service is mandratory for DNS, so it is newly configured during
>> upgrade.
>> Now it works because schema update is the first, so during configuration
>> upgrade we have actual schema.
>
> Right, but what's your point? We are not discussing order of updates 
> here, I'm perfectly fine with schema updates being done before 
> configuration updates.
So you want to run schema update before configuration upgrade every restart?
OR you want to run schema update if needed based on version, and then 
run configuration upgrade?

>
>>>
>>>> Running schema upgrade every time?
>>>>>
>>>>>> What if a service changes in a way, the IPA configuration will not
>>>>>> work?
>>>>>
>>>>> Then it's a bug and needs to be fixed, like any other bug. IIRC there
>>>>> was only one or two occurences of such bug in the past 3 years (I
>>>>> remember sshd_config), so I don't think you have a strong case here.
>>>> Ok
>>>>>
>>>>>> The user will need to change it manually, but after each restart,
>>>>>> upgrade will change the value back into IPA required configuration
>>>>>> which
>>>>>> will not work.
>>>>>
>>>>> Says who? It's our code, we can do whatever we want, it doesn't have
>>>>> to be dumb like this.
>>>>>
>>>>>> Yes, we have upgrade state file, but then the comparing of one
>>>>>> value is
>>>>>> faster then checking each state if was executed.
>>>>>
>>>>> How faster is that, like, few milliseconds? Are you seriously
>>>>> considering this the right optimization in a process that is
>>>>> magnitudes slower?
>>>> Ok the speed is not so important, but I still do not like the idea of
>>>> executing the code which is not needed to be executed, because I know
>>>> the version is the same as was before last restart, so nothing 
>>>> changed.
>>>
>>> Weren't "clever" optimizations like this what got us into this whole
>>> refactoring bussiness in the first place?
>> The "clever" optimizations worked in past, but IPA grown and now
>> contains constraints/requirements which nobody expected.
>
> Yes, then why do we need another one, especially so when it does not 
> provide any significant speed-up?
>
>> What if we
>> will need some update which needs to execute time-consuming system check
>> during every upgrade in future?
>
> Then we deal with the optimization in the future, instead of doing it 
> prematurely now.
>
>> User can always run the upgrade manually, with --skip-version-check, and
>> then configuration plugins will decide if the upgrade is needed.
>>
>>>
>>>>>
>>>>>>
>>>>>> My personal opinion is, application should not try to fix itself 
>>>>>> every
>>>>>> restart.
>>>>>
>>>>> One could say that application should not try to upgrade itself every
>>>>> restart, but here we are, doing it anyway...
>>>> I want to do upgrade only if needed not every restart.
>>>
>>> If there is nothing to upgrade, nothing will be upgraded. The effect
>>> is the same.
>>>
>>>>>
>>>>>>>
>>>>>>>> In the past, I do not recall
>>>>>>>> ipa-upgradeconfig as being really fast, especially 
>>>>>>>> certmonger/Dogtag
>>>>>>>> related
>>>>>>>> updates were really slow thank to service restarts, etc.
>>>>>>>
>>>>>>> Correct, but I was talking about configuration file updates, not
>>>>>>> (re)starts, which have to always be done in ipactl anyway.
>>>>>>>
>>>>>>>>
>>>>>>>>> 3)
>>>>>>>>>
>>>>>>>>> "
>>>>>>>>> * Prevent user to use ipa-upgradeconfig and ipa-ldap-updater
>>>>>>>>> "
>>>>>>>>>
>>>>>>>>> Even without arguments? Is ipactl now the only right place to
>>>>>>>>> trigger manual
>>>>>>>>> update?
>>>>>> Sorry, I will add more details there, if this is not clear.
>>>>>> ipa-upgrateconfig will be removed
>>>>>> ipa-ldap-updater will not be able to do overall update, you will
>>>>>> need to
>>>>>> specify options and update file.
>>>>>>>>>
>>>>>>>>> 4)
>>>>>>>>>
>>>>>>>>> "
>>>>>>>>> Plugins are called from update files, using new directive
>>>>>>>>> update-plugin:<plugin-name>
>>>>>>>>> "
>>>>>>>>>
>>>>>>>>> Why "update-plugin" and not just "plugin"? Do you expect other
>>>>>>>>> kinds
>>>>>>>>> of plugins
>>>>>>>>> to be called from update files in the future? (I certainly 
>>>>>>>>> don't.)
>>>>>>>>
>>>>>>>> I have no strong feelings on this one, but IMO it is always
>>>>>>>> better to
>>>>>>>> have some
>>>>>>>> "plan B" if we choose to indeed implement some yet unforeseen 
>>>>>>>> plugin
>>>>>>>> based
>>>>>>>> capability...
>>>>>>>
>>>>>>> I doubt that will happen, but if it does, we can always add
>>>>>>> "plan-b-plugin" directive.
>>>>>> I do not insist on "update-plugin", I just wanted to be more 
>>>>>> specific
>>>>>> which type of plugin is expected there.
>>>>>
>>>>> Well, the names of the files end with .update and they are located in
>>>>> /usr/share/ipa/updates, I think that's enough hints as to what 
>>>>> type of
>>>>> plugin is expected.
>>>>>
>>>> okay
>>>>>>>
>>>>>>>>
>>>>>>>>> 5)
>>>>>>>>>
>>>>>>>>> "
>>>>>>>>> New class UpdatePlugin is used for all update plugins.
>>>>>>>>> "
>>>>>>>>>
>>>>>>>>> Just reuse the existing Updater class, no need to reinvent the
>>>>>>>>> wheel.
>>>>>>>>>
>>>>>>>>> 6)
>>>>>>>>>
>>>>>>>>> I wonder why configuration update is done after data update 
>>>>>>>>> and not
>>>>>>>>> before. I
>>>>>>>>> know it's been like that for a long time, but it seems kind of
>>>>>>>>> unnatural to me,
>>>>>>>>> especially now when schema update is separate from data update.
>>>>>>>>> (Rob?)
>>>>>> We need schema update first, but I haven't found any services which
>>>>>> need
>>>>>> to have updated data (I might be wrong)
>>>>>>>>>
>>>>>>>>> 7)
>>>>>>>>>
>>>>>>>>> "
>>>>>>>>> keep --test option and fix the plugins which do not respect the
>>>>>>>>> option
>>>>>>>>> "
>>>>>>>>>
>>>>>>>>> Just a note, I believe this ticket is related:
>>>>>>>>> <https://fedorahosted.org/freeipa/ticket/3448>.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Good work overall!
>>>>>>>>>
>>>>>>>>> Honza
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>


-- 
Martin Basti