[EnMasse] AddressController performance / ConfigMap lookups
Ulf Lilleengen
lulf at redhat.com
Fri Mar 23 08:11:53 UTC 2018
Hi,
I've done some investigation into the custom resources approach and how
we could migrate to that. If we use api aggregation/custom api server[1]
instead of custom resource definitions (which is sort of a quick way to
get custom apis), there is a path for gradually moving towards
configurable persistence.
The way api aggregation works is that one register the address
controller as an API server handling v1.enmasse.io API. This is already
implemented in [2], and allows you to proxy the REST API through the
kubernetes master.
This is, however, not sufficient to get 'native' support in kubectl and
oc for manipulating the resources. I then studied the service catalog
API server implementation and extended the REST API to support the paths
/apis, /apis/enmasse.io, and changed the format for /apis/enmasse.io/v1
based on the types expected when using api aggregation[3].
With those, relatively minor, API changes, I am able to create address
spaces using 'kubectl create -f space.yaml' containing only
kind: AddressSpace
apiVersion: v1
metadata:
name: default
spec:
type: standard
plan: standard-unlimited
This will in turn call the REST API path
/apis/enmasse.io/v1/addressspaces that we already have implemented,
which will store it as a configmap.
Now, imagine we also do the same for addresses (not much more work but
requires some API changes). Then we have an API for manipulating address
spaces and addresses as custom resources through the kubernetes API, but
the federation to the REST API and how they are stored is hidden.
At that point, the standard controller, agent and the controller part of
the address controller still operate on the configmaps.
A potential next step, however, is to split the REST API part out from
the address-controller process into its own 'api server' component.
Then, we can gradually migrate standard-controller, address-controller
and agent to operate on the new custom resources REST API rather than
configmaps REST API. Once this is done, the fact that configmaps are
used is just an implementation detail known to the 'api server'. This is
exactly how all kubernetes components use the 'core api server' today
(it happens to use etcd underneath).
The last step would then to have the 'api server' support alternative
ways to store the resources. In fact, the service catalog api server
uses etcd directly to store resources but aspires (according to some
TODOs in the code) to support multiple persistence configurations.
I think what's nice with this approach is that we can longer term
gradually move towards what you proposed initially Carsten, and shorter
term gain a huge improvement in the usability of the API (being able to
use kubectl and oc directly) without changing our implementation.
It does, however, bring a dependency on the REST API to be registered
with the kubernetes master before agent, standard-controller and
address-controller can work. I think it is good for EnMasse as a whole,
but perhaps not for the ability to deploy just the address space, so we
should consider that if this is a direction we want to go in.
[1]
https://kubernetes.io/docs/concepts/api-extension/custom-resources/#api-server-aggregation
[2]
https://github.com/EnMasseProject/enmasse/blob/master/templates/include/api-service.json
[3]
https://kubernetes.io/docs/reference/generated/federation/v1/definitions/#_v1_apiresource
Best regards,
Ulf
On 03/12/2018 02:48 PM, Lohmann Carsten (INST/ECS4) wrote:
> Hi Ulf,
>
> in general, we were thinking whether it wouldn't be better to use a
> database (e.g. MongoDB) instead of ConfigMaps for address-persistence.
>
> When thinking about a scenario with 10000+ addresses, having that many
> ConfigMaps seems .. odd. Kind of going beyond what the ConfigMap
> mechanism as Pod configuration data is probably intended for (although
> there don't seem to be hard limits in that sense).
>
> Database-persistence could possibly offer better performance and
> simplify backup-strategies.
>
> Also when updating EnMasse by re-deploying the EnMasse components
> (having deleted the K8s-namespace first), it seems easier to have the
> addresses in the database untouched by this instead of having to
> re-create the addresses/ConfigMaps.
>
> I think from an architectural point of view, the question is, whether
> the addresses are considered cluster-state information (therefore
> belonging in the etcd datastore) or application-specific data.
>
> With the current address ConfigMap content structure modelled after K8s
> resources, it's obviously handled more as cluster-state information.
>
> (In that sense, is it planned to switch addresses to be proper K8s
> custom resources instead of ConfigMaps?).
>
> But, I think the addresses could also be viewed as application-specific
> data and in that sense better be stored externally.
>
> WDYT? Have you thought about replacing the ConfigMap persistence? Do you
> see limitations with the current ConfigMap-based approach thinking about
> 10000+ addresses?
>
> Or would you see other persistence options?
>
> From an implementation standpoint, having a separate persistence
> implementation seems quite straightforward in the address/standard
> controller with the AddressApi interface already in place.
>
> Then the agent component would have to be changed as well (maybe
> changing it to use the AddressController REST API?).
>
> Best regards
>
> Carsten
>
>
> *Von:*Ulf Lilleengen [mailto:lulf at redhat.com]
> *Gesendet:* Montag, 5. März 2018 15:00
> *An:* Lohmann Carsten (INST/ECS4) <Carsten.Lohmann at bosch-si.com>
> *Cc:* enmasse at redhat.com
> *Betreff:* Re: [EnMasse] AddressController performance / ConfigMap lookups
>
> Hi Carsten,
>
> Yes, the getSchema() will lookup the configmaps every time. I suspected
> that we might need to cache it, but decided to see how often this would
> actually be used before optimizing it. Sounds like it should be
> optimized :). I'm a bit surprised that it takes this long to get this
> information from the kubernetes master though.
>
> Using watchResources instead to cache it sounds like a sensible thing to
> do to me.
>
> Best regards,
>
> Ulf
>
> On Mon, Mar 5, 2018 at 2:17 PM, Lohmann Carsten (INST/ECS4)
> <Carsten.Lohmann at bosch-si.com <mailto:Carsten.Lohmann at bosch-si.com>> wrote:
>
> Hi,
>
> We have noticed some performance issues when using the
> AddressController REST API.
>
> Here is a log excerpt with added debug output concerning the
> addition of 2 addresses:
>
> ---------------------
>
> 2018-03-02 16:31:43.281 [vert.x-worker-thread-15] DEBUG
> HttpAddressService:94 - appendAddresses:
> [telemetry/tst_8432b9a6d2194c3f8c6328706eb455a0,
> event/tst_8432b9a6d2194c3f8c6328706eb455a0]
>
> 2018-03-02 16:31:43.290 [vert.x-worker-thread-15] DEBUG
> ConfigMapAddressSpaceApi:38 - getAddressSpaceWithName: get() took 8ms
>
> 2018-03-02 16:31:43.291 [vert.x-worker-thread-15] DEBUG
> AddressApiHelper:47 - verifyAuthorized took 0ms
>
> 2018-03-02 16:31:43.305 [vert.x-worker-thread-15] DEBUG
> ConfigMapSchemaApi:55 - listConfigMaps type=address-space-plan,
> resultList.size=1 took 14ms
>
> 2018-03-02 16:31:43.321 [vert.x-worker-thread-15] DEBUG
> ConfigMapSchemaApi:55 - listConfigMaps type=address-plan,
> resultList.size=6 took 15ms
>
> 2018-03-02 16:31:43.336 [vert.x-worker-thread-15] DEBUG
> ConfigMapSchemaApi:55 - listConfigMaps type=resource-definition,
> resultList.size=4 took 14ms
>
> 2018-03-02 16:31:43.351 [vert.x-worker-thread-15] DEBUG
> ConfigMapSchemaApi:55 - listConfigMaps type=address-plan,
> resultList.size=6 took 14ms
>
> 2018-03-02 16:31:43.365 [vert.x-worker-thread-15] DEBUG
> ConfigMapSchemaApi:55 - listConfigMaps type=resource-definition,
> resultList.size=4 took 14ms
>
> 2018-03-02 16:31:43.380 [vert.x-worker-thread-15] DEBUG
> ConfigMapSchemaApi:55 - listConfigMaps type=resource-definition,
> resultList.size=4 took 14ms
>
> 2018-03-02 16:31:43.397 [vert.x-worker-thread-15] DEBUG
> ConfigMapSchemaApi:55 - listConfigMaps type=resource-definition,
> resultList.size=4 took 16ms
>
> 2018-03-02 16:31:43.412 [vert.x-worker-thread-15] DEBUG
> ConfigMapSchemaApi:55 - listConfigMaps type=resource-definition,
> resultList.size=4 took 15ms
>
> 2018-03-02 16:31:43.427 [vert.x-worker-thread-15] DEBUG
> ConfigMapSchemaApi:55 - listConfigMaps type=resource-definition,
> resultList.size=4 took 14ms
>
> 2018-03-02 16:31:43.442 [vert.x-worker-thread-15] DEBUG
> ConfigMapSchemaApi:55 - listConfigMaps type=resource-definition,
> resultList.size=4 took 14ms
>
> 2018-03-02 16:31:43.456 [vert.x-worker-thread-15] DEBUG
> ConfigMapSchemaApi:55 - listConfigMaps type=resource-definition,
> resultList.size=4 took 14ms
>
> 2018-03-02 16:31:43.456 [vert.x-worker-thread-15] DEBUG
> ConfigMapSchemaApi:166 - getSchema took 165ms
>
> 2018-03-02 16:31:43.473 [vert.x-worker-thread-15] DEBUG
> ConfigMapAddressApi:89 - listAddresses: list() took 16ms
>
> 2018-03-02 16:31:43.494 [vert.x-worker-thread-15] DEBUG
> ConfigMapAddressApi:103 - createAddress: create() took 20ms
>
> 2018-03-02 16:31:43.506 [vert.x-worker-thread-15] DEBUG
> ConfigMapAddressApi:103 - createAddress: create() took 12ms
>
> 2018-03-02 16:31:43.524 [vert.x-worker-thread-15] DEBUG
> ConfigMapAddressApi:89 - listAddresses: list() took 17ms
>
> 2018-03-02 16:31:43.524 [vert.x-worker-thread-15] DEBUG
> HttpAddressService:48 - appendAddresses
> [telemetry/tst_8432b9a6d2194c3f8c6328706eb455a0,
> event/tst_8432b9a6d2194c3f8c6328706eb455a0] end (result: 56 items) -
> requestProcessing took 242ms
>
> --------------------
>
> What becomes obvious here is that the "
> ConfigMapSchemaApi.getSchema" invocation is quite expensive with its
> "listConfigMaps" calls.
>
> The duration of 160ms is quite typical in our environment.
>
> We even had times where the API server took longer for the requests
> and where the output looked like this:
>
> --------------------
>
> 2018-03-02 17:20:07.197 [vert.x-worker-thread-13] DEBUG
> HttpAddressService:94 - appendAddresses:
> [telemetry/tst_a77831ab936849c4b8a1ba8d15d2e018,
> event/tst_a77831ab936849c4b8a1ba8d15d2e018]
>
> 2018-03-02 17:20:07.345 [vert.x-worker-thread-13] DEBUG
> ConfigMapAddressSpaceApi:38 - getAddressSpaceWithName: get() took 147ms
>
> 2018-03-02 17:20:07.345 [vert.x-worker-thread-13] DEBUG
> AddressApiHelper:47 - verifyAuthorized took 0ms
>
> 2018-03-02 17:20:07.444 [vert.x-worker-thread-13] DEBUG
> ConfigMapSchemaApi:55 - listConfigMaps type=address-space-plan,
> resultList.size=1 took 98ms
>
> 2018-03-02 17:20:07.591 [vert.x-worker-thread-13] DEBUG
> ConfigMapSchemaApi:55 - listConfigMaps type=address-plan,
> resultList.size=6 took 147ms
>
> 2018-03-02 17:20:07.714 [vert.x-worker-thread-13] DEBUG
> ConfigMapSchemaApi:55 - listConfigMaps type=resource-definition,
> resultList.size=4 took 122ms
>
> 2018-03-02 17:20:07.834 [vert.x-worker-thread-13] DEBUG
> ConfigMapSchemaApi:55 - listConfigMaps type=address-plan,
> resultList.size=6 took 120ms
>
> 2018-03-02 17:20:07.981 [vert.x-worker-thread-13] DEBUG
> ConfigMapSchemaApi:55 - listConfigMaps type=resource-definition,
> resultList.size=4 took 146ms
>
> 2018-03-02 17:20:08.131 [vert.x-worker-thread-13] DEBUG
> ConfigMapSchemaApi:55 - listConfigMaps type=resource-definition,
> resultList.size=4 took 149ms
>
> 2018-03-02 17:20:08.254 [vert.x-worker-thread-13] DEBUG
> ConfigMapSchemaApi:55 - listConfigMaps type=resource-definition,
> resultList.size=4 took 122ms
>
> 2018-03-02 17:20:08.374 [vert.x-worker-thread-13] DEBUG
> ConfigMapSchemaApi:55 - listConfigMaps type=resource-definition,
> resultList.size=4 took 120ms
>
> 2018-03-02 17:20:08.494 [vert.x-worker-thread-13] DEBUG
> ConfigMapSchemaApi:55 - listConfigMaps type=resource-definition,
> resultList.size=4 took 119ms
>
> 2018-03-02 17:20:08.641 [vert.x-worker-thread-13] DEBUG
> ConfigMapSchemaApi:55 - listConfigMaps type=resource-definition,
> resultList.size=4 took 146ms
>
> 2018-03-02 17:20:08.761 [vert.x-worker-thread-13] DEBUG
> ConfigMapSchemaApi:55 - listConfigMaps type=resource-definition,
> resultList.size=4 took 120ms
>
> 2018-03-02 17:20:08.761 [vert.x-worker-thread-13] DEBUG
> ConfigMapSchemaApi:166 - getSchema took 1416ms
>
> 2018-03-02 17:20:08.913 [vert.x-worker-thread-13] DEBUG
> ConfigMapAddressApi:89 - listAddresses: list() took 151ms
>
> 2018-03-02 17:20:09.145 [vert.x-worker-thread-13] DEBUG
> ConfigMapAddressApi:103 - createAddress: create() took 231ms
>
> 2018-03-02 17:20:09.265 [vert.x-worker-thread-13] DEBUG
> ConfigMapAddressApi:103 - createAddress: create() took 119ms
>
> 2018-03-02 17:20:09.426 [vert.x-worker-thread-13] DEBUG
> ConfigMapAddressApi:89 - listAddresses: list() took 160ms
>
> 2018-03-02 17:20:09.426 [vert.x-worker-thread-13] DEBUG
> HttpAddressService:48 - appendAddresses
> [telemetry/tst_a77831ab936849c4b8a1ba8d15d2e018,
> event/tst_a77831ab936849c4b8a1ba8d15d2e018] end (result: 66 items) -
> requestProcessing took 2229ms
>
> --------------------
>
> Possible performance improvements concerning the API server aside,
> there is the question whether there is room to make "getSchema" faster.
>
> To me it looks like the K8s resources requested there (address space
> plans, address plans, resource definitions) are fairly static and
> could therefore be cached/kept in memory.
>
> I guess updates on these K8s resources could be handled via
> ConfigMapAddressAPI.watchResources (?).
>
> WDYT? Would that be feasible?
>
> Best regards
>
> *Carsten Lohmann
> *
> (INST/ECS4)
> Bosch Software Innovations GmbH | Ullsteinstr. 128 | 12109 Berlin |
> GERMANY| www.bosch-si.com <http://www.bosch-si.com>
>
> Sitz: Berlin, Registergericht: Amtsgericht Charlottenburg; HRB 148411 B
> Aufsichtsratsvorsitzender: Dr.-Ing. Thorsten Lücke;
> Geschäftsführung: Dr. Stefan Ferber, Michael Hahn
>
>
>
> _______________________________________________
> enmasse mailing list
> enmasse at redhat.com <mailto:enmasse at redhat.com>
> https://www.redhat.com/mailman/listinfo/enmasse
>
--
Ulf
More information about the enmasse
mailing list