[EnMasse] AddressController performance / ConfigMap lookups

Fri Apr 6 07:53:18 UTC 2018

Hi,

here are some test results, creating 10k addresses - 5k brokered, 5k standard-anycast, 1 broker, 1 router instance:

- the duration for creating 2 addresses at a time has increased during the test. 
With ~10k existing addresses, adding 2 addresses took 2s:
--------------------
HttpAddressService:105 - appendAddresses: [telemetry/test3_78, event/test3_78]
ConfigMapAddressApi:94 - listAddresses: list() took 934ms
ConfigMapAddressApi:94 - listAddresses: list() took 1148ms
HttpAddressService:48 - appendAddresses [telemetry/test3_78, event/test3_78] end (result: 9948 items) - requestProcessing took 2213ms
--------------------
This is mainly because of the time it takes to list the Address-ConfigMaps (see also #1117).

- On the K8s api-server, processing time for ConfigMap requests reached 500-1000ms:
GET /api/v1/namespaces/hono/configmaps?labelSelector=type%20notin%20(address-config,address-space,address-space-plan,address-plan): (600.477208ms)
GET /api/v1/namespaces/hono/configmaps?labelSelector=type%3Daddress-space: (614.516495ms)

- The load on the K8s api-server was inconspicuous during the test. 
However, when the qdrouter pod was recreated (once because of an error and once I've deleted the pod), there was increased load (up to 3) on the api-server and the K8s-Dashboard wasn't reacting for a while.
This is because of many requests to modify the isReady-state in the address-configmaps - see #1131 for that.

- Concerning the StandardController, the 30s interval check looked like this in the end:
Total: 9948, Active: 9948, Configuring: 0, Pending: 0, Terminating: 0, Failed: 0
onUpdate: done; total: 5711 ms, resolvedPlan: 2 ms, calculatedUsage: 5 ms, checkedQuota: 0 ms, listClusters: 665 ms, provisionResources: 0 ms, checkStatuses: 5031 ms, deprovisionUnused: 3 ms, replaceAddresses: 0 ms, gcTerminating: 0 ms

---
So, I think, things are still looking ok for this many addresses, with improvements concerning address creation time and behavior on qdrouter restarts being discussed in corresponding tickets.
Going further up with the number of addresses, there should probably be some other storage mechanism.

Thinking further about how to reduce the dependence on the K8s API Server/Etcd and concerning the Custom API Server proposal:

My first question here would be, if the EnMasse Custom API Server (moved out of the AddressController) should be written in Go, using the same API-machinery as the core K8s APIs. 
That would make achieving K8s API compatibility easier but on the other hand would probably mean quite a lot of effort, even with the "apiserver-builder" [1] (which is still labeled 
"only a proof of concept, and is not production ready").
When sticking with a Java-Implementation for the API Server instead, the question would be whether it needs to be separate from the AddressController then. From the Agent point of view, that wouldn't matter.

Concerning the migration path:
I guess that migrating storage of addresses from ConfigMaps towards K8s CustomResourceDefinitions (CRDs) could be an intermediate step, as that would already have some benefits (like omission of (de)serialization overhead, direct update of status field). And changing Agent and Standard/AddressController to work with K8s custom resources instead of ConfigMaps would need to be done in the same way when going towards API Aggregation/Custom API server, I guess. 
The thing already mentioned about CRDs in OpenShift was of course the question about needed cluster-admin access. But I guess deciding on using a custom API server in the end would then require cluster-admin access anyway.

> It does, however, bring a dependency on the REST API to be registered with the
> kubernetes master before agent, standard-controller and address-controller can
> work.
Does the Agent or the AddressController (the new one without APIServer) have to talk to the (new) EnMasse API Server via the K8s API Server? 
Or can it directly communicate with the EnMasse API Server and do authentication/authorisation separately. 
Then that dependency wouldn't necessarily be there.

Best regards
Carsten

[1] https://github.com/kubernetes-incubator/apiserver-builder/blob/master/README.md

> -----Ursprüngliche Nachricht-----
> Von: Ulf Lilleengen [mailto:lulf at redhat.com]
> Gesendet: Freitag, 23. März 2018 09:12
> An: Lohmann Carsten (INST/ECS4) <Carsten.Lohmann at bosch-si.com>
> Cc: enmasse at redhat.com
> Betreff: Re: AW: [EnMasse] AddressController performance / ConfigMap lookups
> 
> Hi,
> 
> I've done some investigation into the custom resources approach and how we
> could migrate to that. If we use api aggregation/custom api server[1] instead of
> custom resource definitions (which is sort of a quick way to get custom apis),
> there is a path for gradually moving towards configurable persistence.
> 
> The way api aggregation works is that one register the address controller as an API
> server handling v1.enmasse.io API. This is already implemented in [2], and allows
> you to proxy the REST API through the kubernetes master.
> 
> This is, however, not sufficient to get 'native' support in kubectl and oc for
> manipulating the resources. I then studied the service catalog API server
> implementation and extended the REST API to support the paths /apis,
> /apis/enmasse.io, and changed the format for /apis/enmasse.io/v1 based on the
> types expected when using api aggregation[3].
> 
> With those, relatively minor, API changes, I am able to create address spaces
> using 'kubectl create -f space.yaml' containing only
> 
> kind: AddressSpace
> apiVersion: v1
> metadata:
>    name: default
> spec:
>    type: standard
>    plan: standard-unlimited
> 
> This will in turn call the REST API path /apis/enmasse.io/v1/addressspaces that we
> already have implemented, which will store it as a configmap.
> 
> Now, imagine we also do the same for addresses (not much more work but
> requires some API changes). Then we have an API for manipulating address
> spaces and addresses as custom resources through the kubernetes API, but the
> federation to the REST API and how they are stored is hidden.
> 
> At that point, the standard controller, agent and the controller part of the address
> controller still operate on the configmaps.
> 
> A potential next step, however, is to split the REST API part out from the address-
> controller process into its own 'api server' component.
> Then, we can gradually migrate standard-controller, address-controller and agent to
> operate on the new custom resources REST API rather than configmaps REST
> API. Once this is done, the fact that configmaps are used is just an implementation
> detail known to the 'api server'. This is exactly how all kubernetes components use
> the 'core api server' today (it happens to use etcd underneath).
> 
> The last step would then to have the 'api server' support alternative ways to store
> the resources. In fact, the service catalog api server uses etcd directly to store
> resources but aspires (according to some TODOs in the code) to support multiple
> persistence configurations.
> 
> I think what's nice with this approach is that we can longer term gradually move
> towards what you proposed initially Carsten, and shorter term gain a huge
> improvement in the usability of the API (being able to use kubectl and oc directly)
> without changing our implementation.
> 
> It does, however, bring a dependency on the REST API to be registered with the
> kubernetes master before agent, standard-controller and address-controller can
> work. I think it is good for EnMasse as a whole, but perhaps not for the ability to
> deploy just the address space, so we should consider that if this is a direction we
> want to go in.
> 
> [1]
> https://kubernetes.io/docs/concepts/api-extension/custom-resources/#api-server-
> aggregation
> [2]
> https://github.com/EnMasseProject/enmasse/blob/master/templates/include/api-
> service.json
> [3]
> https://kubernetes.io/docs/reference/generated/federation/v1/definitions/#_v1_apir
> esource
> 
> Best regards,
> 
> Ulf
> 