[Container-tools] First impressions on packaging a multi-tier Nulecule application on Kubernetes

[Mark Lamourine]

----- Original Message -----
> Hi all,
> 
> Lately, I have been working on packaging sentry[1] for Nulecule. And,
> I chose to use Kubernetes as  a provider for this, so that I could
> explore kubernetes too. However, my journey with kubernetes hasn't
> been that smooth. I will list down the issues that bothered me the
> most.
> 
> Linking containers and inconsistencies (Docker vs Kubernetes)
> ==============================================
> 
> In Docker, we use **--link** option to explicitly link other
> containers to a container. The argument **--link** takes is of the
> form:
> some-redis:myredis
> 
> where **some-redis** is a container name and **myredis** is an alias
> for **some-redis** in the target container.
> 
> What it does is that:
> - It exposes certain properties of **some-redis** container in the
> target container formatted as <ALIAS>_*, e.g, MYREDIS_PORT_6379_TCP,
> MYREDIS_PORT_6379_TCP_ADDR, etc.
> - It also exposes the other, user defined environment variables from
> **some-redis** to the target container formatter as <ALIAS>_ENV_*,
> e.g, MYREDIS_ENV_FOO, etc.
> - It also adds host entry for **some-redis** in /etc/hosts for service
> discovery insider internal docker network.
> 
> In contrast to the above, what Kubernetes does implicitly, is that it
> exposes the properties, like port info, tcp addr, etc. of all the
> active services in a container in the form:
> <Service name>_PROP_NAME
> However, I could not see it sharing the user defined ENV variables of
> a service, which Docker does through linking. Also, it does not add
> any host entries of the active services in /etc/hosts

I would suggest that Docker links should be deprecated. Unless something has changed recently they only work when two containers reside on the same host.  They also make inappropriate changes to the host config (modifying /etc/hosts).  I think they also claim host network ports, which means that collisions are likely and difficult to resolve.

Container images designed to communicate using links will never be able to scale up to a container cloud whether it's Kubernetes or Mesos or anything else.

> 
> I had to actually customize[3] the official Sentry Docker image to make it
> work.
> 
> Complicated DNS setup
> ==================
>  Kubernetes supports enabling cluster addon for DNS, however it's not
> that straightforward. In the docs[2], it says that just put the addons
> in /etc/kubernetes/addons and restart the services and the addons
> should start working. But, what to put, what should be the directory
> structure, nothing is clearly mentioned.
> 
> Also, I tried running our nulecule example for skydns[3], however, it
> could not bring up the etcd pod as it reported errors like:
> 
> kubernetes create snapshot directory error: mkdir
> /var/etcd/data/member: permission denied

Is /var/etcd/data purely in the container or is it attempting to mount something from the host? If the latter, then it has to run as a privileged container.  It probably shouldn't be doing that, as the etcd you're creating is solely for the user of your Sentry service.

> 
> It seemed to originate due to some permission issue from creating
> volumes with **emptyDir** option. However, I could not find a work
> around for that. I think skydns setup should also be a part of adb
> bundle.
> 
> I got the nulecule example working by removing the **volume** related
> options, but it was not a valid solution.

Why not?  What are the characteristics of the storage that was being placed in the volumes?  Where is that data ultimately supposed to reside and who uses it?

> 
> To summarize, my experience with Kubernetes was not that smooth. It
> could be because of my ignorance, or some genuine issues, but at the
> end of the day, getting started should not be so painful. Also,
> regarding the above inconsistencies between Docker and Kubernetes, in
> the way container linking works, work needs to be done to level up
> these inconsistencies to provide the user a smooth user experience.
> 
> 
> [1]: https://hub.docker.com/_/sentry
> [2]: https://github.com/kubernetes/kubernetes/tree/master/cluster/addons
> [3]:
> https://github.com/rtnpro/docker-sentry/commit/e286e953ddd213f5f0331e2a22a229b1f51a1eb8
> 
> Regards,
> rtnpro
> --
> Ratnadeep Debnath,
> https://www.waartaa.com
> GPG Fingerprint: 033C 8041 A0E9 CDBA 2E02  B785 2119 5486 F245 DFD6
> 
> _______________________________________________
> Container-tools mailing list
> Container-tools at redhat.com
> https://www.redhat.com/mailman/listinfo/container-tools
> 

-- 
Mark Lamourine <mlamouri at redhat.com>
Sr. Software Developer, Cloud Strategy
Red Hat, 314 Littleton Road, Westford MA 01886
Voice: +1 978 392 1093
http://people.redhat.com/~mlamouri
markllama @ irc://irc.freenod.org*lopsa