[EnMasse] AddressController performance / ConfigMap lookups

Fri Apr 6 09:28:23 UTC 2018

On 04/06/2018 09:53 AM, Lohmann Carsten (INST/ECS4) wrote:
> Hi,
> 

Hi,

Thanks for your great testing efforts and for sharing the results! It 
has been really helpful!

> here are some test results, creating 10k addresses - 5k brokered, 5k standard-anycast, 1 broker, 1 router instance:
> 
> - the duration for creating 2 addresses at a time has increased during the test.
> With ~10k existing addresses, adding 2 addresses took 2s:
> --------------------
> HttpAddressService:105 - appendAddresses: [telemetry/test3_78, event/test3_78]
> ConfigMapAddressApi:94 - listAddresses: list() took 934ms
> ConfigMapAddressApi:94 - listAddresses: list() took 1148ms
> HttpAddressService:48 - appendAddresses [telemetry/test3_78, event/test3_78] end (result: 9948 items) - requestProcessing took 2213ms
> --------------------
> This is mainly because of the time it takes to list the Address-ConfigMaps (see also #1117).
> 
> - On the K8s api-server, processing time for ConfigMap requests reached 500-1000ms:
> GET /api/v1/namespaces/hono/configmaps?labelSelector=type%20notin%20(address-config,address-space,address-space-plan,address-plan): (600.477208ms)
> GET /api/v1/namespaces/hono/configmaps?labelSelector=type%3Daddress-space: (614.516495ms)
> 
> - The load on the K8s api-server was inconspicuous during the test.
> However, when the qdrouter pod was recreated (once because of an error and once I've deleted the pod), there was increased load (up to 3) on the api-server and the K8s-Dashboard wasn't reacting for a while.
> This is because of many requests to modify the isReady-state in the address-configmaps - see #1131 for that.
> 
> - Concerning the StandardController, the 30s interval check looked like this in the end:
> Total: 9948, Active: 9948, Configuring: 0, Pending: 0, Terminating: 0, Failed: 0
> onUpdate: done; total: 5711 ms, resolvedPlan: 2 ms, calculatedUsage: 5 ms, checkedQuota: 0 ms, listClusters: 665 ms, provisionResources: 0 ms, checkStatuses: 5031 ms, deprovisionUnused: 3 ms, replaceAddresses: 0 ms, gcTerminating: 0 ms
> 
> ---
> So, I think, things are still looking ok for this many addresses, with improvements concerning address creation time and behavior on qdrouter restarts being discussed in corresponding tickets.
> Going further up with the number of addresses, there should probably be some other storage mechanism.
> 
> 
> Thinking further about how to reduce the dependence on the K8s API Server/Etcd and concerning the Custom API Server proposal:
> 
> My first question here would be, if the EnMasse Custom API Server (moved out of the AddressController) should be written in Go, using the same API-machinery as the core K8s APIs.
> That would make achieving K8s API compatibility easier but on the other hand would probably mean quite a lot of effort, even with the "apiserver-builder" [1] (which is still labeled
> "only a proof of concept, and is not production ready").
> When sticking with a Java-Implementation for the API Server instead, the question would be whether it needs to be separate from the AddressController then. From the Agent point of view, that wouldn't matter.
> 

 From the prototype I made, it is a relatively small chunk of java code 
to enable the custom API server, mainly because we already have the code 
to serve the 'resources' already. Rewriting it in Go I think is a 
significant effort, but we could benefit from the k8s go libraries.

IMO the API server part should be separate from the address controller, 
because I think that will make upgrades more flexible. For instance, if 
we roll out a new storage back-end for the API server, the 
address-controller can just continue to run unaffected.

You can also potentially deploy EnMasse without the API server, or even 
without the address controller and only a single address space + api server.

> Concerning the migration path:
> I guess that migrating storage of addresses from ConfigMaps towards K8s CustomResourceDefinitions (CRDs) could be an intermediate step, as that would already have some benefits (like omission of (de)serialization overhead, direct update of status field). And changing Agent and Standard/AddressController to work with K8s custom resources instead of ConfigMaps would need to be done in the same way when going towards API Aggregation/Custom API server, I guess.
> The thing already mentioned about CRDs in OpenShift was of course the question about needed cluster-admin access. But I guess deciding on using a custom API server in the end would then require cluster-admin access anyway.

Going with CRDs first, we need to keep 2 implementations in each 
component supporting both configmaps and CRDs and we either have to 
write/read both variants to make upgrades easy. With API Aggregation, we 
create the same API as you would get with CRDs, and we can gradually 
migrate each component that uses configmaps to use that API. I'm not 
sure how easy it is to migrate from CRDs to API Aggregation since you 
would end up with the CRDs and API Aggregation requiring the same API paths.

With API aggregation, when all components are migrated to use the new 
API, one option is to then to keep the configmap implementation, and 
then gradually work on an alternative implementation where data is 
stored in etcd, and all components using the API would be unaffected by 
that.

Regarding cluster admin access, I think that if we want to get good 
integration with kubernetes and OpenShift, we need to make use of these 
features.

There are potentially other uses of an API aggregation server around 
credentials. One could imagine an AddressSpaceRoles resource that 
contains users and groups that the API server will store directly in 
keycloak, simplifying creating authorization policies for messaging.

> 
>> It does, however, bring a dependency on the REST API to be registered with the
>> kubernetes master before agent, standard-controller and address-controller can
>> work.
> Does the Agent or the AddressController (the new one without APIServer) have to talk to the (new) EnMasse API Server via the K8s API Server?
> Or can it directly communicate with the EnMasse API Server and do authentication/authorisation separately.
> Then that dependency wouldn't necessarily be there.
> 

It can do so separately, but we should make that require a valid TLS 
client cert or something so we can control who can use that API (the 
internal components + k8s for instance).

Best regards,

Ulf

> Best regards
> Carsten
> 
> [1] https://github.com/kubernetes-incubator/apiserver-builder/blob/master/README.md
> 
> 
>> -----Ursprüngliche Nachricht-----
>> Von: Ulf Lilleengen [mailto:lulf at redhat.com]
>> Gesendet: Freitag, 23. März 2018 09:12
>> An: Lohmann Carsten (INST/ECS4) <Carsten.Lohmann at bosch-si.com>
>> Cc: enmasse at redhat.com
>> Betreff: Re: AW: [EnMasse] AddressController performance / ConfigMap lookups
>>
>> Hi,
>>
>> I've done some investigation into the custom resources approach and how we
>> could migrate to that. If we use api aggregation/custom api server[1] instead of
>> custom resource definitions (which is sort of a quick way to get custom apis),
>> there is a path for gradually moving towards configurable persistence.
>>
>> The way api aggregation works is that one register the address controller as an API
>> server handling v1.enmasse.io API. This is already implemented in [2], and allows
>> you to proxy the REST API through the kubernetes master.
>>
>> This is, however, not sufficient to get 'native' support in kubectl and oc for
>> manipulating the resources. I then studied the service catalog API server
>> implementation and extended the REST API to support the paths /apis,
>> /apis/enmasse.io, and changed the format for /apis/enmasse.io/v1 based on the
>> types expected when using api aggregation[3].
>>
>> With those, relatively minor, API changes, I am able to create address spaces
>> using 'kubectl create -f space.yaml' containing only
>>
>> kind: AddressSpace
>> apiVersion: v1
>> metadata:
>>     name: default
>> spec:
>>     type: standard
>>     plan: standard-unlimited
>>
>> This will in turn call the REST API path /apis/enmasse.io/v1/addressspaces that we
>> already have implemented, which will store it as a configmap.
>>
>> Now, imagine we also do the same for addresses (not much more work but
>> requires some API changes). Then we have an API for manipulating address
>> spaces and addresses as custom resources through the kubernetes API, but the
>> federation to the REST API and how they are stored is hidden.
>>
>> At that point, the standard controller, agent and the controller part of the address
>> controller still operate on the configmaps.
>>
>> A potential next step, however, is to split the REST API part out from the address-
>> controller process into its own 'api server' component.
>> Then, we can gradually migrate standard-controller, address-controller and agent to
>> operate on the new custom resources REST API rather than configmaps REST
>> API. Once this is done, the fact that configmaps are used is just an implementation
>> detail known to the 'api server'. This is exactly how all kubernetes components use
>> the 'core api server' today (it happens to use etcd underneath).
>>
>> The last step would then to have the 'api server' support alternative ways to store
>> the resources. In fact, the service catalog api server uses etcd directly to store
>> resources but aspires (according to some TODOs in the code) to support multiple
>> persistence configurations.
>>
>> I think what's nice with this approach is that we can longer term gradually move
>> towards what you proposed initially Carsten, and shorter term gain a huge
>> improvement in the usability of the API (being able to use kubectl and oc directly)
>> without changing our implementation.
>>
>> It does, however, bring a dependency on the REST API to be registered with the
>> kubernetes master before agent, standard-controller and address-controller can
>> work. I think it is good for EnMasse as a whole, but perhaps not for the ability to
>> deploy just the address space, so we should consider that if this is a direction we
>> want to go in.
>>
>> [1]
>> https://kubernetes.io/docs/concepts/api-extension/custom-resources/#api-server-
>> aggregation
>> [2]
>> https://github.com/EnMasseProject/enmasse/blob/master/templates/include/api-
>> service.json
>> [3]
>> https://kubernetes.io/docs/reference/generated/federation/v1/definitions/#_v1_apir
>> esource
>>
>> Best regards,
>>
>> Ulf
>>
> 

-- 
Ulf