[Ansible-service-broker] pod can't start when deployed on origin 3.10
Jason Montleon
jmontleo at redhat.com
Tue Aug 14 12:49:11 UTC 2018
Sorry, I misunderstood. It sounds like you are using openshift-ansible
for the install? I thought you were using oc cluster up
--enable=automation-service-broker.
You can use these settings for your host to modify the image used during
an openshift-ansible install.
ansible_service_broker_image_prefix
ansible_service_broker_image_tag
You could also modify the patch command to work with your installation
post install. Something along the lines of:
oc patch dc -n openshift-ansible-service-broker asb
--patch='{"spec":{"template":{"spec":{"containers":[{"name": "asb",
"image":"ansibleplaybookbundle/origin-ansible-service-broker:release-1.2"}]}}}}'
You might do an 'oc get dc -n openshift-ansible-service-broker' and 'oc
edit dc -n openshift-ansible-service-broker' to get the correct dc and
container names. I am no 100% certain of these off the top of my head.
If the default behavior for openshift-ansible with origin isn't what
you'd expect it's probably worth filing a bug so we can try to improve it.
On 08/14/2018 04:01 AM, Charles Moulliard wrote:
> I have modified manually the ASB's DC to change the image from latest to
> 1.3 and I can see the Playbooks now [1]
>
> [1]
> https://www.dropbox.com/s/uvyjfsg3tx75p0s/Screenshot%202018-08-14%2010.00.36.png?dl=0
>
> On Tue, Aug 14, 2018 at 9:52 AM, Georgios Andrianakis
> <gandrian at redhat.com <mailto:gandrian at redhat.com>> wrote:
>
> Hi
>
> On Mon, Aug 13, 2018 at 8:40 PM Jason Montleon <jmontleo at redhat.com
> <mailto:jmontleo at redhat.com>> wrote:
>
> Someone recently filed a bug for the v3.10/latest version
> mismatch. We
> have some ideas how to handle it better and are working on
> implementing
> it for 3.11 (and possibly 3.10).
>
> For now it's pretty easy to modify the dc image to get the
> release you want.
>
> oc patch dc -n openshift-automation-service-broker
> openshift-automation-service-broker
> --patch='{"spec":{"template":{"spec":{"containers":[{"name":
> "penshift-automation-service-broker",
> "image":"ansibleplaybookbundle/origin-ansible-service-broker:release-1.2"}]}}}}'
>
>
> I tried applying the patch after my Ansible Service Broker
> installation was failing in the same exact way that Charles
> mentions, but the odd thing is that I don't have the
> /openshift-automation-service-b//roker namespace/. Is that to be
> expected or perhaps some bug?
>
>
> release-1.0 corresponds to 3.7
> release-1.1 3.9
> release-1.2 3.10
> latest what will become release-1.3 3.11
>
> I'm not sure anything earlier than release-1.2 will work with the
> resources created by the APB so I'd stick to release-1.2 and
> latest if
> you can for now.
>
> On 08/13/2018 12:52 PM, Charles Moulliard wrote:
> > Is there a reason why we don't install the APB docker image
> tagged for
> > openshift 3.10 but instead latest ?
> >
> >
> https://github.com/openshift/openshift-ansible/blob/release-3.10/roles/ansible_service_broker/defaults/main.yml#L26
> <https://github.com/openshift/openshift-ansible/blob/release-3.10/roles/ansible_service_broker/defaults/main.yml#L26>
> >
> >
> >
> > On Mon, Aug 13, 2018 at 6:42 PM, Charles Moulliard
> <cmoullia at redhat.com <mailto:cmoullia at redhat.com>
> > <mailto:cmoullia at redhat.com <mailto:cmoullia at redhat.com>>> wrote:
> >
> > FYI, events still report that the pod is unhealthy
> >
> > 6:39:49 PMWarningUnhealthy Readiness probe failed: Get
> > https://10.128.0.16:1338/healthz
> <https://10.128.0.16:1338/healthz>
> <https://10.128.0.16:1338/healthz
> <https://10.128.0.16:1338/healthz>>:
> > dial tcp 10.128.0.16:1338 <http://10.128.0.16:1338>
> <http://10.128.0.16:1338>: getsockopt:
> > connection refused
> > 4 times in the last minute
> > 6:39:45 PMWarningUnhealthy Liveness probe failed: Get
> > https://10.128.0.16:1338/healthz
> <https://10.128.0.16:1338/healthz>
> <https://10.128.0.16:1338/healthz
> <https://10.128.0.16:1338/healthz>>:
> > dial tcp 10.128.0.16:1338 <http://10.128.0.16:1338>
> <http://10.128.0.16:1338>: getsockopt:
> > connection refused
> > 2 times in the last
> >
> > and of course, when I browse the catalog, I don't see APB
> playbooks :-(
> >
> >
> > On Mon, Aug 13, 2018 at 6:40 PM, David Zager
> <dzager at redhat.com <mailto:dzager at redhat.com>
> > <mailto:dzager at redhat.com <mailto:dzager at redhat.com>>> wrote:
> >
> > An update to the liveness and readiness probes for
> 3.10 may be
> > appropriate in openshift-ansible.
> >
> > We are intentionally keeping the
> `dh-automation-broker-apb` out
> > of the list of services. I suspect the issue here is
> simply with
> > the log level of "not adding spec
> dh-automation-broker-apb to
> > list of services due to error transforming to service
> - Could
> > not find the parameter type for: "?
> >
> > On Mon, Aug 13, 2018 at 12:35 PM Charles Moulliard
> > <cmoullia at redhat.com <mailto:cmoullia at redhat.com>
> <mailto:cmoullia at redhat.com <mailto:cmoullia at redhat.com>>> wrote:
> >
> > To get the pod up and running, I have removed the
> > threshold's parameters from the APB's deployment
> resource
> > ->liveness and readinessProbes.
> >
> > But I see this error now
> >
> > time="2018-08-13T16:30:09Z" level=info msg="Broker
> > successfully bootstrapped on startup"
> > time="2018-08-13T16:30:09Z" level=info
> msg="Listening on
> > https://[::]:1338"
> > time="2018-08-13T16:30:09Z" level=info
> msg="Ansible Service
> > Broker Starting"
> > ERROR: logging before flag.Parse: I0813
> 16:30:09.600491
> > 1 serve.go:89] Serving securely on [::]:1338
> > time="2018-08-13T16:30:20Z" level=info
> > msg="AnsibleBroker::Catalog"
> > time="2018-08-13T16:30:20Z" level=error msg="not
> adding spec
> > dh-automation-broker-apb to list of services due
> to error
> > transforming to service - Could not find the
> parameter type
> > for: "
> > 10.128.0.1 - - [13/Aug/2018:16:30:20 +0000] "GET
> > /ansible-service-broker/v2/catalog HTTP/1.1" 200
> 273918
> > time="2018-08-13T16:30:40Z" level=info
> > msg="AnsibleBroker::Catalog"
> > time="2018-08-13T16:30:40Z" level=error msg="not
> adding spec
> > dh-automation-broker-apb to list of services due
> to error
> > transforming to service - Could not find the
> parameter type
> > for: "
> > 10.128.0.1 - - [13/Aug/2018:16:30:40 +0000] "GET
> > /ansible-service-broker/v2/catalog HTTP/1.1" 200
> 273918
> >
> > On Fri, Aug 10, 2018 at 8:31 AM, Georgios Andrianakis
> > <gandrian at redhat.com <mailto:gandrian at redhat.com>
> <mailto:gandrian at redhat.com <mailto:gandrian at redhat.com>>> wrote:
> >
> > Hello,
> >
> > I am seeing the same exact error as Charles
> under the
> > same conditions -> installing ASB on
> Openshift Origin 3.10.
> >
> > Regards
> >
> > On Thu, Aug 9, 2018 at 8:36 PM David Zager
> > <dzager at redhat.com <mailto:dzager at redhat.com>
> <mailto:dzager at redhat.com <mailto:dzager at redhat.com>>> wrote:
> >
> > After reading through the logs and seeing
> the way
> > the cut off, I'm curious if it's related
> to the
> > `initialDelaySeconds` in the broker's
> deployment
> > config
> >
> <https://github.com/openshift/openshift-ansible/blob/master/roles/ansible_service_broker/templates/asb_dc.yaml.j2#L59-L72 <https://github.com/openshift/openshift-ansible/blob/master/roles/ansible_service_broker/templates/asb_dc.yaml.j2#L59-L72>>.
> > With the upstream broker-apb we increased
> this from
> > 15 seconds to 120
> >
> <https://github.com/openshift/ansible-service-broker/blob/master/apb/defaults/main.yml#L17 <https://github.com/openshift/ansible-service-broker/blob/master/apb/defaults/main.yml#L17>>.
> > I suspect that could be the cause.
> >
> >
> > On Thu, Aug 9, 2018 at 11:37 AM Jason
> Montleon
> > <jmontleo at redhat.com
> <mailto:jmontleo at redhat.com> <mailto:jmontleo at redhat.com
> <mailto:jmontleo at redhat.com>>>
> > wrote:
> >
> > I made some comments in irc this morning.
> >
> > Our downstream CI test are passing as
> of this
> > morning using
> > openshift-ansible with downstream
> images. From
> > his gist logs it looked
> > like the broker started, but I am not
> clear on
> > what's happening to cause
> > the getsockopt error.
> >
> >
> https://gist.github.com/cmoulliard/716468a9e9beb21ba860806a30e3ddcb
> <https://gist.github.com/cmoulliard/716468a9e9beb21ba860806a30e3ddcb>
> >
> <https://gist.github.com/cmoulliard/716468a9e9beb21ba860806a30e3ddcb <https://gist.github.com/cmoulliard/716468a9e9beb21ba860806a30e3ddcb>>
> >
> > It seems like the service name
> resolved, but it
> > couldn't connect at the
> > address "172.30.206.144". Perhaps
> there is a
> > misconfiguration with the
> > docker or openshift networking?
> >
> > On 08/09/2018 11:30 AM, John Matthews
> wrote:
> > > +Jason/David who may be able to help
> > >
> > >
> > > Hi Charles,
> > >
> > > "I don't see anymore the apb-etcd
> pod wuth
> > openshift 3.10, it is normal ?"
> > >
> > > Yes, this is normal for 3.10, we
> moved to
> > CRDs for the data we store, so
> > > we dropped our own etcd and rely
> on regular
> > k8s API.
> > >
> > >
> > > "Error fetching catalog. Error
> getting broker
> > catalog:
> > >
> >
> Gethttps://asb.openshift-ansible-service-broker.svc:1338/ansible-service-broker/v2/catalog
> >
> > >
> >
> <https://asb.openshift-ansible-service-broker.svc:1338/ansible-service-broker/v2/catalog <https://asb.openshift-ansible-service-broker.svc:1338/ansible-service-broker/v2/catalog>
> >
> <https://asb.openshift-ansible-service-broker.svc:1338/ansible-service-broker/v2/catalog <https://asb.openshift-ansible-service-broker.svc:1338/ansible-service-broker/v2/catalog>>>:
> >
> > > dial tcp172.30.206.144:1338
> > <http://172.30.206.144:1338/>:
> getsockopt:
> > > no route to host"
> > >
> > > Please double check the namespace the
> > automation broker is running in.
> > > I wonder if there is a bug we
> missed with
> > "openshift-ansible" deploying
> > > with origin.
> > > I believe the majority of our QE
> scenarios
> > are with openshift-ansible
> > > with downstream OCP or 'oc cluster
> up' with
> > origin.
> > >
> > > Guessing, but perhaps there is a
> mismatch
> > between actual namespace
> > > broker is in versus what we are
> registering
> > to service-catalog.
> > >
> > >
> > >
> > >
> > > On Thu, Aug 9, 2018 at 4:29 AM,
> Charles
> > Moulliard <cmoullia at redhat.com
> <mailto:cmoullia at redhat.com>
> > <mailto:cmoullia at redhat.com
> <mailto:cmoullia at redhat.com>>
> > > <mailto:cmoullia at redhat.com
> <mailto:cmoullia at redhat.com>
> > <mailto:cmoullia at redhat.com
> <mailto:cmoullia at redhat.com>>>> wrote:
> > >
> > > Additional info. The
> ClusterServiceBroker
> > reports this error
> > >
> > > Error fetching catalog. Error
> getting
> > broker catalog: Get
> > >
> >
> https://asb.openshift-ansible-service-broker.svc:1338/ansible-service-broker/v2/catalog
> <https://asb.openshift-ansible-service-broker.svc:1338/ansible-service-broker/v2/catalog>
> >
> <https://asb.openshift-ansible-service-broker.svc:1338/ansible-service-broker/v2/catalog <https://asb.openshift-ansible-service-broker.svc:1338/ansible-service-broker/v2/catalog>>
> > >
> >
> <https://asb.openshift-ansible-service-broker.svc:1338/ansible-service-broker/v2/catalog <https://asb.openshift-ansible-service-broker.svc:1338/ansible-service-broker/v2/catalog> <https://asb.openshift-ansible-service-broker.svc:1338/ansible-service-broker/v2/catalog <https://asb.openshift-ansible-service-broker.svc:1338/ansible-service-broker/v2/catalog>>>:
> > > dial tcp 172.30.206.144:1338
> <http://172.30.206.144:1338>
> > <http://172.30.206.144:1338>
> > <http://172.30.206.144:1338>:
> > > getsockopt: no route to host
> > >
> > > On Thu, Aug 9, 2018 at 10:22
> AM, Charles
> > Moulliard
> > > <cmoullia at redhat.com
> <mailto:cmoullia at redhat.com>
> > <mailto:cmoullia at redhat.com
> <mailto:cmoullia at redhat.com>>
> > <mailto:cmoullia at redhat.com
> <mailto:cmoullia at redhat.com>
> > <mailto:cmoullia at redhat.com
> <mailto:cmoullia at redhat.com>>>> wrote:
> > >
> > > Hi
> > >
> > > I can't install ASB on
> origin 3.10
> > anymore using the playbook
> > >
> >
> "openshift-ansible/playbooks/openshift-service-catalog/config.yml"
> > >
> > > The apb deployment starts,
> a pod is
> > created and after x minutes
> > > I see the status =
> "Failed" but the
> > pod's log doesn't report ERROR
> > >
> > > Log :
> > >
> >
> https://gist.github.com/cmoulliard/716468a9e9beb21ba860806a30e3ddcb
> <https://gist.github.com/cmoulliard/716468a9e9beb21ba860806a30e3ddcb>
> >
> <https://gist.github.com/cmoulliard/716468a9e9beb21ba860806a30e3ddcb <https://gist.github.com/cmoulliard/716468a9e9beb21ba860806a30e3ddcb>>
> > >
> >
> <https://gist.github.com/cmoulliard/716468a9e9beb21ba860806a30e3ddcb <https://gist.github.com/cmoulliard/716468a9e9beb21ba860806a30e3ddcb> <https://gist.github.com/cmoulliard/716468a9e9beb21ba860806a30e3ddcb <https://gist.github.com/cmoulliard/716468a9e9beb21ba860806a30e3ddcb>>>
> > >
> > > How can we troubleshoot
> and resolve
> > such error ?
> > > I don't see anymore the
> apb-etcd pod
> > wuth openshift 3.10, it is
> > > normal ?
> > >
> > > Regards
> > >
> > > Charles
> > >
> > >
> > >
> > >
> >
> _______________________________________________
> > > Ansible-service-broker mailing
> list
> > > Ansible-service-broker at redhat.com
> <mailto:Ansible-service-broker at redhat.com>
> >
> <mailto:Ansible-service-broker at redhat.com
> <mailto:Ansible-service-broker at redhat.com>>
> > >
> <mailto:Ansible-service-broker at redhat.com
> <mailto:Ansible-service-broker at redhat.com>
> >
> <mailto:Ansible-service-broker at redhat.com
> <mailto:Ansible-service-broker at redhat.com>>>
> > >
> >
> https://www.redhat.com/mailman/listinfo/ansible-service-broker
> <https://www.redhat.com/mailman/listinfo/ansible-service-broker>
> >
> <https://www.redhat.com/mailman/listinfo/ansible-service-broker <https://www.redhat.com/mailman/listinfo/ansible-service-broker>>
> > >
> >
> <https://www.redhat.com/mailman/listinfo/ansible-service-broker <https://www.redhat.com/mailman/listinfo/ansible-service-broker> <https://www.redhat.com/mailman/listinfo/ansible-service-broker <https://www.redhat.com/mailman/listinfo/ansible-service-broker>>>
> > >
> > >
> >
> > --
> > Jason Montleon | email:
> jmontleo at redhat.com <mailto:jmontleo at redhat.com>
> > <mailto:jmontleo at redhat.com
> <mailto:jmontleo at redhat.com>>
> > Software Engineer | gpg key: 0x069E3022
> > Red Hat, Inc. | irc: jmontleo
> > desk: 978-392-3930
> <tel:%28978%29%20392-3930> |
> > cell: 508-496-0663
> <tel:%28508%29%20496-0663>
> >
> >
> _______________________________________________
> > Ansible-service-broker mailing list
> > Ansible-service-broker at redhat.com
> <mailto:Ansible-service-broker at redhat.com>
> > <mailto:Ansible-service-broker at redhat.com
> <mailto:Ansible-service-broker at redhat.com>>
> >
> https://www.redhat.com/mailman/listinfo/ansible-service-broker
> <https://www.redhat.com/mailman/listinfo/ansible-service-broker>
> >
> <https://www.redhat.com/mailman/listinfo/ansible-service-broker <https://www.redhat.com/mailman/listinfo/ansible-service-broker>>
> >
> >
> >
> >
>
> --
> Jason Montleon | email: jmontleo at redhat.com
> <mailto:jmontleo at redhat.com>
> Software Engineer | gpg key: 0x069E3022
> Red Hat, Inc. | irc: jmontleo
> desk: 978-392-3930 | cell: 508-496-0663
>
>
--
Jason Montleon | email: jmontleo at redhat.com
Software Engineer | gpg key: 0x069E3022
Red Hat, Inc. | irc: jmontleo
desk: 978-392-3930 | cell: 508-496-0663
More information about the Ansible-service-broker
mailing list