[Container-tools] First impressions on packaging a multi-tier Nulecule application on Kubernetes

[Ratnadeep Debnath]

Hi all,

Lately, I have been working on packaging sentry[1] for Nulecule. And,
I chose to use Kubernetes as  a provider for this, so that I could
explore kubernetes too. However, my journey with kubernetes hasn't
been that smooth. I will list down the issues that bothered me the
most.

Linking containers and inconsistencies (Docker vs Kubernetes)
==============================================

In Docker, we use **--link** option to explicitly link other
containers to a container. The argument **--link** takes is of the
form:
some-redis:myredis

where **some-redis** is a container name and **myredis** is an alias
for **some-redis** in the target container.

What it does is that:
- It exposes certain properties of **some-redis** container in the
target container formatted as <ALIAS>_*, e.g, MYREDIS_PORT_6379_TCP,
MYREDIS_PORT_6379_TCP_ADDR, etc.
- It also exposes the other, user defined environment variables from
**some-redis** to the target container formatter as <ALIAS>_ENV_*,
e.g, MYREDIS_ENV_FOO, etc.
- It also adds host entry for **some-redis** in /etc/hosts for service
discovery insider internal docker network.

In contrast to the above, what Kubernetes does implicitly, is that it
exposes the properties, like port info, tcp addr, etc. of all the
active services in a container in the form:
<Service name>_PROP_NAME
However, I could not see it sharing the user defined ENV variables of
a service, which Docker does through linking. Also, it does not add
any host entries of the active services in /etc/hosts

I had to actually customize[3] the official Sentry Docker image to make it work.

Complicated DNS setup
==================
 Kubernetes supports enabling cluster addon for DNS, however it's not
that straightforward. In the docs[2], it says that just put the addons
in /etc/kubernetes/addons and restart the services and the addons
should start working. But, what to put, what should be the directory
structure, nothing is clearly mentioned.

Also, I tried running our nulecule example for skydns[3], however, it
could not bring up the etcd pod as it reported errors like:

kubernetes create snapshot directory error: mkdir
/var/etcd/data/member: permission denied

It seemed to originate due to some permission issue from creating
volumes with **emptyDir** option. However, I could not find a work
around for that. I think skydns setup should also be a part of adb
bundle.

I got the nulecule example working by removing the **volume** related
options, but it was not a valid solution.

To summarize, my experience with Kubernetes was not that smooth. It
could be because of my ignorance, or some genuine issues, but at the
end of the day, getting started should not be so painful. Also,
regarding the above inconsistencies between Docker and Kubernetes, in
the way container linking works, work needs to be done to level up
these inconsistencies to provide the user a smooth user experience.


[1]: https://hub.docker.com/_/sentry
[2]: https://github.com/kubernetes/kubernetes/tree/master/cluster/addons
[3]: https://github.com/rtnpro/docker-sentry/commit/e286e953ddd213f5f0331e2a22a229b1f51a1eb8

Regards,
rtnpro
-- 
Ratnadeep Debnath,
https://www.waartaa.com
GPG Fingerprint: 033C 8041 A0E9 CDBA 2E02  B785 2119 5486 F245 DFD6