[EnMasse] Upcoming REST API changes

[Ulf Lilleengen]

On 05/08/2018 10:54 PM, Ulf Lilleengen wrote:
> On 05/08/2018 02:27 PM, Gordon Sim wrote:
>> On 08/05/18 11:47, Ulf Lilleengen wrote:
>>>
>>> tir. 8. mai 2018 kl. 03:23 skrev Gordon Sim <gsim at redhat.com 
>>> <mailto:gsim at redhat.com>>:
>>>
>>>     On 08/05/18 11:13, Ulf Lilleengen wrote:
>>>      > One of my goals with this work is that we in the future could
>>>     migrate
>>>      > all components today working on configmaps to use this API 
>>> instead
>>>      > (through the kubernetes master). This would allow us to change
>>>     how we
>>>      > persist the data from configmaps to something more fitting.  My
>>>     worry is
>>>      > that if we put all addresses into the address space definition,
>>>     we would
>>>      > end up facing issues in standard-controller and agent due to 
>>> status
>>>      > updates needing to fetch/query a lot of data when one of the
>>>     addresses
>>>      > change.
>>>
>>>     At present they need to do that anyway, as there is no reliable 
>>> way of
>>>     tracking changes (especially deletions) except by periodically
>>>     retrieving the whole list.
>>>
>>>
>>> The periodic resync  is less frequent than the status checks which is 
>>> what typically trigger the watches a lot when configuring many 
>>> addresses. If we put addresses into then address space definition, 
>>> then all addresses is read on every update to any address, whereas 
>>> today only a single address is read when the status is changed.
>>
>> As you say, if you are reading at the granularity of the address 
>> space, then a small update involves extra data transfer. This is the 
>> same as happens on the resync.
>>
>> Again, as you rightly point out, the frequency at which this happens 
>> is relevant.
>>
>> The granularity and frequency are often related though.
>>
>> A single logical update that adds (or deletes or modifies) thousands 
>> of addresses results in thousands of notifications when watching at 
>> the granularity of the address.
>>
>> If instead you work at the granularity of the address space, for 
>> updates and reads/watches, my feeling is that the frequency of changes 
>> will be much less and therefore that the extra data transferred is not 
>> prohibitive especially in comparison to the resync that happens anyway.
>>
>> I.e. though at present during an update we get a storm of 
>> notifications with a much higher frequency than the resync interval, 
>> my instinct is that in general logical updates will be probably even 
>> less frequent than the resync.
> 
> Agreed with this logic, it might not be an issue then. So just thinking 
> about the resource then, it could be something like:
> 
> apiVersion: enmasse.io/v1
> kind: AddressSpace
> metadata:
>    name: myspace
> spec:
>    type: standard
>    plan: small-standard
>    addresses:
>      - address: myqueue
>        type: queue
>        plan: small-queue
>      - address: mytopic
>        type: topic
>        plan: small-topic
> status:
>    isReady: true
>    addresses:
>      - address: myqueue
>        phase: Configuring
>        isReady: false
>        messages:
>          - "Autolink not present on router"
>          - "Address not present on router"
>      - address: mytopic
>        phase: Active
>        isReady: true
> 
> Making status a subresource, we can then also assign permissions so that 
> only the internal controllers may update the status whereas the user 
> only get read permissions (same as for pods).
> 
> I think doing this would be a great simplification from a user POV, so I 
> really hope it can work.
> 
> As a first step, I can modify the API server to consume this format and 
> convert to/from the configmaps we use today.
> 
> Wdyt? Are there any tests we should/can do to ensure that this approach 
> will actually work? Any other concerns?
> 

Responding to myself here :) Combining the addresses in the address 
space requires some changes to the address controller as well if we want 
to avoid rewriting standard-controller, agent etc.

The namespace where the address configmaps are stored is not created 
until the address controller processes the address space. The API server 
cannot create these configmaps unless the namespace already exists. To 
get around this, we could try to have the API server create the 
namespace when handling the http request, instead of the 
address-controller creating it.

Another alternative is to just dive into the process of rewriting the 
standard-controller, agent etc. to use the API which would move us 
towards the goal of pluggable storage faster, but at the risk of 
changing many things at once.

-- 
Ulf