[Freeipa-devel] Designing better API compatibility

[Petr Spacek]

On 20.3.2015 15:51, Nathaniel McCallum wrote:
> On Fri, 2015-03-20 at 09:58 -0400, Simo Sorce wrote:
>> On Fri, 2015-03-20 at 14:38 +0100, Martin Kosek wrote:
>>> On 03/20/2015 02:19 PM, Simo Sorce wrote:
>>>> On Fri, 2015-03-20 at 14:13 +0100, Martin Kosek wrote:
>>>>> Hi guys,
>>>>>
>>>>> I would like to resurrect the discussion we had during 
>>>>> DevConf.cz time, about
>>>>> API compatibility in the FreeIPA server.
>>>>>
>>>>> So right now, we maintain the backward compatibility, old 
>>>>> clients can talk to
>>>>> newer servers. Forward compatibility is not maintained. 
>>>>> Unfortunately, this is
>>>>> not very helpful in real deployments, where the server will 
>>>>> often be some
>>>>> RHEL/CentOS system and the client may be the newest Fedora - 
>>>>> with newer API
>>>>> than the server. This is the toughest part we need to solve.
>>>>>
>>>>> There 3 main areas we wanted to attack with respect to 
>>>>> compatibility:
>>>>>
>>>>> 1) API publishing and/or API browser
>>>>> This is mostly documentation/interactive browser to see the 
>>>>> supported API of
>>>>> the server. It should not be difficult, it would just consume 
>>>>> the metadata
>>>>> already generated by the server.
>>>>>
>>>>> Ticket: https://fedorahosted.org/freeipa/ticket/3129
>>>>>
>>>>> 2) Forward compatibility of the direct API consumers
>>>>> Until now, to keep newer clients working against older server, 
>>>>> we are using the
>>>>> following trick in the ipa-client-install:
>>>>>
>>>>> https://git.fedorahosted.org/cgit/freeipa.git/tree/ipa-client/ipa-install/ipa-client-install#
>>>>> n1649
>>>>>
>>>>> It mostly works, one just needs to know the minimal version 
>>>>> that needs to be
>>>>> supported. It would be more user friendly, however, if this 
>>>>> check is done on
>>>>> the server automatically, without user having to research it. 
>>>>> This applies both
>>>>> for ipalib python lib consumer and for direct JSON-RPC 
>>>>> consumers.
>>>>>
>>>>> Ticket: https://fedorahosted.org/freeipa/ticket/4739
>>>>>
>>>>> 3) Forward compatibility of the "ipa" client tool
>>>>> There are different approaches how to fix this, the generally 
>>>>> accepted idea was
>>>>> to implement very thin client, which would download and cache 
>>>>> metadata from the
>>>>> server on the client and generate the CLI from it. We would 
>>>>> only need to have
>>>>> separate client only plugins, basically implementing 
>>>>> interactive_promt_callback
>>>>> from existing server side plugins.
>>>>>
>>>>> Tickets: https://fedorahosted.org/freeipa/ticket/4739,
>>>>> https://fedorahosted.org/freeipa/ticket/4768
>>>>>
>>>>>
>>>>> Now, question is what we can do in 4.2. I do not think we can 
>>>>> manage to rewrite
>>>>> "ipa" command in the thin client, but we should do at least 
>>>>> some portion of 1)
>>>>> and 3).
>>>>>
>>>>> I could not decipher that from our Devconf.cz notes. To me, 
>>>>> the simplest way of
>>>>> fixing forward compatibility seems to be following steps 
>>>>> (besides not making
>>>>> API backwards incompatible - i.e. what we do already):
>>>>>
>>>>> - keep sending API version from client to server
>>>>> - server should not refuse newer API versions
>>>>> - only raise error when an unknown option or unknown command 
>>>>> is used
>>>>
>>>> This is not sufficient, older 3.3 and 4.x servers can't be 
>>>> changed and we MUST be compatible with those.
>>>> Basically the plan MUST work with already released servers, this 
>>>> is a constraint that cannot be releaxed, please work within this 
>>>> limitations.
>>>
>>> Correct. I see 2 approaches here:
>>>
>>> a) Thin client, which simply downloads metadata from the (old) 
>>> server and won't 
>>> use unsupported commands/parameters
>>> b) Not-so-thin client that knows the minimal API versions of  
>>> commands/parameters (can be annotated in the code), that would 
>>> ping the server 
>>> first to identify it's version, validate that the chosen set of  
>>> commands/parameters is supported on that server and then send the 
>>> commands with 
>>> that version.
>>
>> If we have a recognizable error the client can take an optimistic 
>> approach, send the command normally, if it gets an error that the 
>> server does not understand it, it checks the version in the reply 
>> and falls back to an older "baseline" version of the command (if 
>> possible) or bails out with an error.
> 
> My understanding was that:
> 
> 1. We already publish all the information necessary to implement a 
> thin client, and have for some time.
We certainly have *some* data but real thin client will most likely require
some changes. Some information like return types and so on are missing.

> 2. Thus, the thin client would work on both new and old versions since 
> it just simply translates from user input into JSON/XML.
> 
> 3. Only plugins with specific client behavior would need to be ported 
> to the thin client. A prime example of this is otptoken-add-yubikey.
> 
> My preference is solidly for implementing the thin client first. Once 
> we have decoupled the client from the current plugin framework, server-
> side changes can be made in isolation. This decoupling is the move 
> that is essentially necessary to provide proper API versioning. And if 
> this can't land for 4.2, land it in the next release. I'd rather do 
> API-stability correctly and a release later than rushed with 
> compromises. We have to live with this forever.
+ all votes I have :-)

-- 
Petr^2 Spacek