[Container-tools] First impressions on packaging a multi-tier Nulecule application on Kubernetes

[Dusty Mabe]

On 09/11/2015 08:17 AM, Ratnadeep Debnath wrote:
> Hi all,
>
> Lately, I have been working on packaging sentry[1] for Nulecule. And,
> I chose to use Kubernetes as  a provider for this, so that I could
> explore kubernetes too. However, my journey with kubernetes hasn't
> been that smooth. I will list down the issues that bothered me the
> most.
>
> Linking containers and inconsistencies (Docker vs Kubernetes)
> ==============================================
>
> In Docker, we use **--link** option to explicitly link other
> containers to a container. The argument **--link** takes is of the
> form:
> some-redis:myredis
>
> where **some-redis** is a container name and **myredis** is an alias
> for **some-redis** in the target container.
>
> What it does is that:
> - It exposes certain properties of **some-redis** container in the
> target container formatted as <ALIAS>_*, e.g, MYREDIS_PORT_6379_TCP,
> MYREDIS_PORT_6379_TCP_ADDR, etc.
> - It also exposes the other, user defined environment variables from
> **some-redis** to the target container formatter as <ALIAS>_ENV_*,
> e.g, MYREDIS_ENV_FOO, etc.
> - It also adds host entry for **some-redis** in /etc/hosts for service
> discovery insider internal docker network.
>
> In contrast to the above, what Kubernetes does implicitly, is that it
> exposes the properties, like port info, tcp addr, etc. of all the
> active services in a container in the form:
> <Service name>_PROP_NAME
> However, I could not see it sharing the user defined ENV variables of
> a service, which Docker does through linking. Also, it does not add
> any host entries of the active services in /etc/hosts
>
> I had to actually customize[3] the official Sentry Docker image to make it work.
>
> Complicated DNS setup
> ==================
>   Kubernetes supports enabling cluster addon for DNS, however it's not
> that straightforward. In the docs[2], it says that just put the addons
> in /etc/kubernetes/addons and restart the services and the addons
> should start working. But, what to put, what should be the directory
> structure, nothing is clearly mentioned.

Did you figure it out? If so could you add it back to the official 
documentation to make it better for the future?

>
> Also, I tried running our nulecule example for skydns[3], however, it
> could not bring up the etcd pod as it reported errors like:
>
> kubernetes create snapshot directory error: mkdir
> /var/etcd/data/member: permission denied
>
> It seemed to originate due to some permission issue from creating
> volumes with **emptyDir** option. However, I could not find a work
> around for that. I think skydns setup should also be a part of adb
> bundle.
>
> I got the nulecule example working by removing the **volume** related
> options, but it was not a valid solution.
>
> To summarize, my experience with Kubernetes was not that smooth. It
> could be because of my ignorance, or some genuine issues, but at the
> end of the day, getting started should not be so painful. Also,
> regarding the above inconsistencies between Docker and Kubernetes, in
> the way container linking works, work needs to be done to level up
> these inconsistencies to provide the user a smooth user experience.
>
>
> [1]: https://hub.docker.com/_/sentry
> [2]: https://github.com/kubernetes/kubernetes/tree/master/cluster/addons
> [3]: https://github.com/rtnpro/docker-sentry/commit/e286e953ddd213f5f0331e2a22a229b1f51a1eb8
>
> Regards,
> rtnpro