[Ansible-service-broker] pod can't start when deployed on origin 3.10
Jason Montleon
jmontleo at redhat.com
Mon Aug 13 19:59:25 UTC 2018
FYI: https://bugzilla.redhat.com/show_bug.cgi?id=1612876#c1
On 08/13/2018 01:40 PM, Jason Montleon wrote:
> Someone recently filed a bug for the v3.10/latest version mismatch. We
> have some ideas how to handle it better and are working on implementing
> it for 3.11 (and possibly 3.10).
>
> For now it's pretty easy to modify the dc image to get the release you
> want.
>
> oc patch dc -n openshift-automation-service-broker
> openshift-automation-service-broker
> --patch='{"spec":{"template":{"spec":{"containers":[{"name":
> "penshift-automation-service-broker",
> "image":"ansibleplaybookbundle/origin-ansible-service-broker:release-1.2"}]}}}}'
>
>
> release-1.0 corresponds to 3.7
> release-1.1 3.9
> release-1.2 3.10
> latest what will become release-1.3 3.11
>
> I'm not sure anything earlier than release-1.2 will work with the
> resources created by the APB so I'd stick to release-1.2 and latest if
> you can for now.
>
> On 08/13/2018 12:52 PM, Charles Moulliard wrote:
>> Is there a reason why we don't install the APB docker image tagged for
>> openshift 3.10 but instead latest ?
>>
>> https://github.com/openshift/openshift-ansible/blob/release-3.10/roles/ansible_service_broker/defaults/main.yml#L26
>>
>>
>>
>>
>> On Mon, Aug 13, 2018 at 6:42 PM, Charles Moulliard
>> <cmoullia at redhat.com <mailto:cmoullia at redhat.com>> wrote:
>>
>> FYI, events still report that the pod is unhealthy
>>
>> 6:39:49 PMWarningUnhealthy Readiness probe failed: Get
>> https://10.128.0.16:1338/healthz <https://10.128.0.16:1338/healthz>:
>> dial tcp 10.128.0.16:1338 <http://10.128.0.16:1338>: getsockopt:
>> connection refused
>> 4 times in the last minute
>> 6:39:45 PMWarningUnhealthy Liveness probe failed: Get
>> https://10.128.0.16:1338/healthz <https://10.128.0.16:1338/healthz>:
>> dial tcp 10.128.0.16:1338 <http://10.128.0.16:1338>: getsockopt:
>> connection refused
>> 2 times in the last
>>
>> and of course, when I browse the catalog, I don't see APB
>> playbooks :-(
>>
>>
>> On Mon, Aug 13, 2018 at 6:40 PM, David Zager <dzager at redhat.com
>> <mailto:dzager at redhat.com>> wrote:
>>
>> An update to the liveness and readiness probes for 3.10 may be
>> appropriate in openshift-ansible.
>>
>> We are intentionally keeping the `dh-automation-broker-apb` out
>> of the list of services. I suspect the issue here is simply with
>> the log level of "not adding spec dh-automation-broker-apb to
>> list of services due to error transforming to service - Could
>> not find the parameter type for: "?
>>
>> On Mon, Aug 13, 2018 at 12:35 PM Charles Moulliard
>> <cmoullia at redhat.com <mailto:cmoullia at redhat.com>> wrote:
>>
>> To get the pod up and running, I have removed the
>> threshold's parameters from the APB's deployment resource
>> ->liveness and readinessProbes.
>>
>> But I see this error now
>>
>> time="2018-08-13T16:30:09Z" level=info msg="Broker
>> successfully bootstrapped on startup"
>> time="2018-08-13T16:30:09Z" level=info msg="Listening on
>> https://[::]:1338"
>> time="2018-08-13T16:30:09Z" level=info msg="Ansible Service
>> Broker Starting"
>> ERROR: logging before flag.Parse: I0813 16:30:09.600491
>> 1 serve.go:89] Serving securely on [::]:1338
>> time="2018-08-13T16:30:20Z" level=info
>> msg="AnsibleBroker::Catalog"
>> time="2018-08-13T16:30:20Z" level=error msg="not adding spec
>> dh-automation-broker-apb to list of services due to error
>> transforming to service - Could not find the parameter type
>> for: "
>> 10.128.0.1 - - [13/Aug/2018:16:30:20 +0000] "GET
>> /ansible-service-broker/v2/catalog HTTP/1.1" 200 273918
>> time="2018-08-13T16:30:40Z" level=info
>> msg="AnsibleBroker::Catalog"
>> time="2018-08-13T16:30:40Z" level=error msg="not adding spec
>> dh-automation-broker-apb to list of services due to error
>> transforming to service - Could not find the parameter type
>> for: "
>> 10.128.0.1 - - [13/Aug/2018:16:30:40 +0000] "GET
>> /ansible-service-broker/v2/catalog HTTP/1.1" 200 273918
>>
>> On Fri, Aug 10, 2018 at 8:31 AM, Georgios Andrianakis
>> <gandrian at redhat.com <mailto:gandrian at redhat.com>> wrote:
>>
>> Hello,
>>
>> I am seeing the same exact error as Charles under the
>> same conditions -> installing ASB on Openshift Origin
>> 3.10.
>>
>> Regards
>>
>> On Thu, Aug 9, 2018 at 8:36 PM David Zager
>> <dzager at redhat.com <mailto:dzager at redhat.com>> wrote:
>>
>> After reading through the logs and seeing the way
>> the cut off, I'm curious if it's related to the
>> `initialDelaySeconds` in the broker's deployment
>> config
>>
>> <https://github.com/openshift/openshift-ansible/blob/master/roles/ansible_service_broker/templates/asb_dc.yaml.j2#L59-L72>.
>>
>> With the upstream broker-apb we increased this from
>> 15 seconds to 120
>>
>> <https://github.com/openshift/ansible-service-broker/blob/master/apb/defaults/main.yml#L17>.
>>
>> I suspect that could be the cause.
>>
>>
>> On Thu, Aug 9, 2018 at 11:37 AM Jason Montleon
>> <jmontleo at redhat.com <mailto:jmontleo at redhat.com>>
>> wrote:
>>
>> I made some comments in irc this morning.
>>
>> Our downstream CI test are passing as of this
>> morning using
>> openshift-ansible with downstream images. From
>> his gist logs it looked
>> like the broker started, but I am not clear on
>> what's happening to cause
>> the getsockopt error.
>>
>>
>> https://gist.github.com/cmoulliard/716468a9e9beb21ba860806a30e3ddcb
>>
>> <https://gist.github.com/cmoulliard/716468a9e9beb21ba860806a30e3ddcb>
>>
>> It seems like the service name resolved, but it
>> couldn't connect at the
>> address "172.30.206.144". Perhaps there is a
>> misconfiguration with the
>> docker or openshift networking?
>>
>> On 08/09/2018 11:30 AM, John Matthews wrote:
>> > +Jason/David who may be able to help
>> >
>> >
>> > Hi Charles,
>> >
>> > "I don't see anymore the apb-etcd pod wuth
>> openshift 3.10, it is normal ?"
>> >
>> > Yes, this is normal for 3.10, we moved to
>> CRDs for the data we store, so
>> > we dropped our own etcd and rely on regular
>> k8s API.
>> >
>> >
>> > "Error fetching catalog. Error getting broker
>> catalog:
>> >
>>
>> Gethttps://asb.openshift-ansible-service-broker.svc:1338/ansible-service-broker/v2/catalog
>>
>>
>> >
>>
>> <https://asb.openshift-ansible-service-broker.svc:1338/ansible-service-broker/v2/catalog
>>
>>
>> <https://asb.openshift-ansible-service-broker.svc:1338/ansible-service-broker/v2/catalog>>:
>>
>>
>> > dial tcp172.30.206.144:1338
>> <http://172.30.206.144:1338/>: getsockopt:
>> > no route to host"
>> >
>> > Please double check the namespace the
>> automation broker is running in.
>> > I wonder if there is a bug we missed with
>> "openshift-ansible" deploying
>> > with origin.
>> > I believe the majority of our QE scenarios
>> are with openshift-ansible
>> > with downstream OCP or 'oc cluster up' with
>> origin.
>> >
>> > Guessing, but perhaps there is a mismatch
>> between actual namespace
>> > broker is in versus what we are registering
>> to service-catalog.
>> >
>> >
>> >
>> >
>> > On Thu, Aug 9, 2018 at 4:29 AM, Charles
>> Moulliard <cmoullia at redhat.com
>> <mailto:cmoullia at redhat.com>
>> > <mailto:cmoullia at redhat.com
>> <mailto:cmoullia at redhat.com>>> wrote:
>> >
>> > Additional info. The ClusterServiceBroker
>> reports this error
>> >
>> > Error fetching catalog. Error getting
>> broker catalog: Get
>> >
>>
>> https://asb.openshift-ansible-service-broker.svc:1338/ansible-service-broker/v2/catalog
>>
>>
>> <https://asb.openshift-ansible-service-broker.svc:1338/ansible-service-broker/v2/catalog>
>>
>> >
>> <https://asb.openshift-ansible-service-broker.svc:1338/ansible-service-broker/v2/catalog
>> <https://asb.openshift-ansible-service-broker.svc:1338/ansible-service-broker/v2/catalog>>:
>>
>> > dial tcp 172.30.206.144:1338
>> <http://172.30.206.144:1338>
>> <http://172.30.206.144:1338>:
>> > getsockopt: no route to host
>> >
>> > On Thu, Aug 9, 2018 at 10:22 AM, Charles
>> Moulliard
>> > <cmoullia at redhat.com
>> <mailto:cmoullia at redhat.com>
>> <mailto:cmoullia at redhat.com
>> <mailto:cmoullia at redhat.com>>> wrote:
>> >
>> > Hi
>> >
>> > I can't install ASB on origin 3.10
>> anymore using the playbook
>> >
>> "openshift-ansible/playbooks/openshift-service-catalog/config.yml"
>> >
>> > The apb deployment starts, a pod is
>> created and after x minutes
>> > I see the status = "Failed" but the
>> pod's log doesn't report ERROR
>> >
>> > Log :
>> >
>>
>> https://gist.github.com/cmoulliard/716468a9e9beb21ba860806a30e3ddcb
>>
>> <https://gist.github.com/cmoulliard/716468a9e9beb21ba860806a30e3ddcb>
>> >
>> <https://gist.github.com/cmoulliard/716468a9e9beb21ba860806a30e3ddcb
>> <https://gist.github.com/cmoulliard/716468a9e9beb21ba860806a30e3ddcb>>
>> >
>> > How can we troubleshoot and resolve
>> such error ?
>> > I don't see anymore the apb-etcd pod
>> wuth openshift 3.10, it is
>> > normal ?
>> >
>> > Regards
>> >
>> > Charles
>> >
>> >
>> >
>> >
>> _______________________________________________
>> > Ansible-service-broker mailing list
>> > Ansible-service-broker at redhat.com
>> <mailto:Ansible-service-broker at redhat.com>
>> > <mailto:Ansible-service-broker at redhat.com
>> <mailto:Ansible-service-broker at redhat.com>>
>> >
>>
>> https://www.redhat.com/mailman/listinfo/ansible-service-broker
>>
>> <https://www.redhat.com/mailman/listinfo/ansible-service-broker>
>> >
>> <https://www.redhat.com/mailman/listinfo/ansible-service-broker
>> <https://www.redhat.com/mailman/listinfo/ansible-service-broker>>
>> >
>> >
>>
>> -- Jason Montleon
>> | email: jmontleo at redhat.com
>> <mailto:jmontleo at redhat.com>
>> Software Engineer | gpg key: 0x069E3022
>> Red Hat, Inc. | irc: jmontleo
>> desk: 978-392-3930 <tel:%28978%29%20392-3930> |
>> cell: 508-496-0663 <tel:%28508%29%20496-0663>
>>
>> _______________________________________________
>> Ansible-service-broker mailing list
>> Ansible-service-broker at redhat.com
>> <mailto:Ansible-service-broker at redhat.com>
>>
>> https://www.redhat.com/mailman/listinfo/ansible-service-broker
>>
>> <https://www.redhat.com/mailman/listinfo/ansible-service-broker>
>>
>>
>>
>>
>
--
Jason Montleon | email: jmontleo at redhat.com
Software Engineer | gpg key: 0x069E3022
Red Hat, Inc. | irc: jmontleo
desk: 978-392-3930 | cell: 508-496-0663
More information about the Ansible-service-broker
mailing list