[EnMasse] AddressController performance / ConfigMap lookups

[Ulf Lilleengen]

On 03/13/2018 10:40 AM, Gordon Sim wrote:
> On 13/03/18 09:16, Ulf Lilleengen wrote:
>> If one address status needs to be changed, the controller then have to 
>> read/write 1000 addresses worth of data, so I wouldn't say that it is 
>> more manageable. If the problem is that the kubernetes master is 
>> overloaded by enmasse use, this could even make it worse.
> 
> It is certainly speculative, but often smaller numbers of slightly 
> larger data items are easier to manage and cause less load. While 
> updating status would indeed cause some unneeded data transfer, 
> retrieval of the addresses (which I would argue is by far the more 
> common operation, assuming the addresses change relatively infrequently 
> on a running system) would involve far fewer calls. Likewise for a bulk 
> update of addresses (10,000 PUTs will generally cause more load that 10, 
> even if the latter have more data in them).
> 

I think we already read addresses in batches by reading a ConfigMapList 
(though maybe that translates to 10000 reads from etcd, I'm not sure if 
it supports it natively).

A PUT with AddressList currently generates 10000 configmap PUTs. 
However, I think that could be improved by moving to custom resources as 
well, as you get 'native' support in the kubernetes API for doing a list 
PUT.

> In any case though, I would suggest we start with some actual 
> experiments on pushing the scale wrt the number of addresses and find 
> out where the actual bottlenecks are and figure out how to alleviate them.

Fully agreed, numbers don't lie (usually :))

Best regards,

Ulf