[Container-tools] Atomic Developer Bundle and OpenShift

[Maciej Szulik]

On 11/04/2015 05:06 AM, Praveen Kumar wrote:
> On Tue, Nov 3, 2015 at 10:35 PM, Langdon White <langdon at redhat.com> wrote:
>>
>>
>> On 11/03/2015 04:21 AM, Maciej Szulik wrote:
>>>
>>> My $0.02 in the topic, to clear out the situation and save some time.
>>> OpenShift is built on top of k8s, so when running OpenShift you actually
>>> run k8s with OpenShift. See for example this piece of start code [1] and
>>> you'll see how we start k8s controllers, on which OS heavily relays on.
>>> That's why running both OpenShift and k8s is not possible without
>>> too much of a hassle. Additionally there's one more point to remember,
>>> when two k8s master will be running they will start fighting for the
>>> pods they've started and nodes they control, which will result in more
>>> mess you would want to have.
>>> Given above arguments it's perfectly understandable to show and explain
>>> you should have one or the other running at any given point in time.
>>> Besides when running OS you still have full access to k8s api in
>>> any way you can have when running plain k8s, iow. both kubectl and
>>> REST API work, this can be seen by looking again at OS source code [2]
>>> or logs from OS start.
>>> I hope that explains more, and if you have more questions don't hesitate
>>> to ask.
>>>
>>> Maciej
>>>
>>>
>>> [1]
>>> https://github.com/openshift/origin/blob/master/pkg/cmd/server/start/start_master.go#L528-L557
>>> [2]
>>> https://github.com/openshift/origin/blob/master/pkg/cmd/server/kubernetes/master.go#L50-L56
>>>
>>
>> So.. the problem is that the two versions are different. So, if you are
>> gonna deploy in prod on RHEL/CentOS (or atomic), i believe you get the
>> rhel/centos version of k8s. As a result, it is not a good test to use the
>> OpenShift k8s. So, I think we all understand that OpenShift is using k8s,
>> the problem is it is the wrong version. IIUC.

Langdon, can you please specify what you mean by OpenShift is using
wrong version of k8s? We're trying hard to follow upstream, our
current version is 20 days old [1] with quite a lot of cherry-picks
that are crucial for our 1.1 release soon to happen.

[1] 
https://github.com/kubernetes/kubernetes/commit/4c8e6f47ec23f390978e651232b375f5f9cde3c7

> Yes, this is what our current issue is (version mismatch). Yesterday I
> again had chat with Maciej and Michal about same. According to them in
> future this will be taken care of by openshift team (same version of
> k8s they are going to use with OS also). I also did some experiment[0]
> around k8s api which openshift provide and there is some network issue
> for service (I have to follow up with Maciej for same).
>
>>
>> I also understand it is painful to bind to different IPs and potential
>> conflicts on docker, however, I think it is worth the experimentation to
>> simplify the experience for the end user.
>
> Yes it is painful but I got a response to try bind() function[1] with
> different IP + services which I will check today and see if that
> workout and let you folks know. k8s and etcd services have options to
> run it at different port then default and when I used this option, I
> was able to run k8s services along with openshift but then whatever
> pods were running before on plain k8s  just crushed as soon as OS
> service started. So even we try to map different service with

This is the case I was talking about. K8s is very aggressive when it
comes to managing docker containers. If you watch carefully every
container started by k8s has a `k8s_` prefix, with this in mind you
might imagine what happens when two k8s (the stadalone one and the one
OpenShift one) run, trying to take over all the containers with that
prefix. And containers is not the only part that k8s very aggressively
manages, others that I can think of will be networking which provides
access to the internal parts of the created infrastructure. The bind
problem Praveen is facing now is the first one on the long list of
problems, I imagine, you'll have to deal with.

> different IP to make sure port conflict not occur then also this issue
> will bug us. (Maciej also faced same issue when I had discussion with
> him and then suggested to use single service at given time)

I'm in contact with Praveen about solving problems he's having to run
services on top of the internal OpenShift k8s, but most of them are
due to the security constraints OpenShift provides to restrict
access to the k8s infrastructure.

>>
>> If the experiment fails, or proves to be too much effort, then, yes, I think
>> turning them each "on and off" is probably the best answer.
>
> Yes, this is our last option.
>
> [0] http://fpaste.org/286696/60948414/
> [1] http://linux.die.net/man/2/bind
>