[EnMasse] EnMasse 0.12.0: Addresses not getting ready
Ulf Lilleengen
lulf at redhat.com
Tue Aug 8 13:40:04 UTC 2017
Hi Carsten,
I don't think the build has gone through yet, but i'll verify later today
as I have been able to reproduce it. I'll give an update once I think it
should work.
Ulf
On Tue, Aug 8, 2017 at 3:14 PM, Lohmann Carsten (INST/ECS4) <
Carsten.Lohmann at bosch-si.com> wrote:
> Hi Ulf,
>
> thanks for the fast reply and fix. Unfortunately, with the new version
> there is still such an error:
> ---
> 2017-08-08 13:03:51 WARN WatcherVerticle:92 - Exception doing resource
> update
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:653)
> at java.util.ArrayList.get(ArrayList.java:429)
> at enmasse.controller.standard.StandardController.
> ingressToEndpoint(StandardController.java:192)
> at java.util.stream.ReferencePipeline$3$1.accept(
> ReferencePipeline.java:193)
> at java.util.stream.ReferencePipeline$2$1.accept(
> ReferencePipeline.java:175)
> at java.util.ArrayList$ArrayListSpliterator.
> forEachRemaining(ArrayList.java:1374)
> at java.util.stream.AbstractPipeline.copyInto(
> AbstractPipeline.java:481)
> at java.util.stream.AbstractPipeline.wrapAndCopyInto(
> AbstractPipeline.java:471)
> at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(
> ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(
> AbstractPipeline.java:234)
> at java.util.stream.ReferencePipeline.collect(
> ReferencePipeline.java:499)
> at enmasse.controller.standard.StandardController.updateEndpoints(
> StandardController.java:158)
> at enmasse.controller.standard.StandardController.
> resourcesUpdated(StandardController.java:93)
> at enmasse.controller.common.WatcherVerticle.run(
> WatcherVerticle.java:89)
> at java.lang.Thread.run(Thread.java:748)
> ---
>
>
> Best regards
>
> Carsten Lohmann
>
> (INST/ECS4)
> Bosch Software Innovations GmbH | Schöneberger Ufer 89-91 | 10785 Berlin |
> GERMANY | www.bosch-si.com
> Tel. +49 30 726112-130 | Fax +49 30 726112-100 |
> carsten.lohmann at bosch-si.com
>
> Sitz: Berlin, Registergericht: Amtsgericht Charlottenburg; HRB 148411 B
> Aufsichtsratsvorsitzender: Dr.-Ing. Thorsten Lücke; Geschäftsführung:
> Dr.-Ing. Rainer Kallenbach, Michael Hahn
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: Ulf Lilleengen [mailto:lulf at redhat.com]
> Gesendet: Dienstag, 8. August 2017 11:32
> An: Lohmann Carsten (INST/ECS4) <Carsten.Lohmann at bosch-si.com>;
> enmasse at redhat.com
> Betreff: Re: [EnMasse] EnMasse 0.12.0: Addresses not getting ready
>
> Hi Carsten,
>
> Thanks for the detailed bug report. After looking at the code, I see that
> the exception is what is preventing the status to get updated.
>
> I'll try to reproduce this as the tls entry should have been added afaict
> from looking at the code. I will apply a fix, and tag a 0.12.1.
> Sorry for the inconvenience!
>
> Best regards,
>
> Ulf
>
> On 08. aug. 2017 10:20, Lohmann Carsten (INST/ECS4) wrote:
> > Hi,
> >
> > we've done an upgrade to EnMasse 0.12.0 and are now having the issue,
> > that addresses registered via the address controller REST interface
> > are not getting ready ("isReady" stays false).
> >
> > Test was done on a local minikube cluster, using the
> > kubernetes/enmasse.yaml deployment file.
> >
> > When looking at the log of the address-controller, there is this
> > exception (occurring repeatedly):
> >
> > ---
> >
> > 2017-08-07 14:53:32 WARN WatcherVerticle:92 - Exception doing
> > resource update
> >
> > java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> >
> > at java.util.ArrayList.rangeCheck(ArrayList.java:653)
> >
> > at java.util.ArrayList.get(ArrayList.java:429)
> >
> > at
> > enmasse.controller.standard.StandardController.ingressToEndpoint(Stand
> > ardController.java:191)
> >
> > at
> > java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:1
> > 93)
> >
> > at
> > java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:1
> > 75)
> >
> > at
> > java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.ja
> > va:1374)
> >
> > at
> > java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> >
> > at
> > java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.jav
> > a:471)
> >
> > at
> > java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:
> > 708)
> >
> > at
> > java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> >
> > at
> > java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> >
> > at
> > enmasse.controller.standard.StandardController.updateEndpoints(Standar
> > dController.java:158)
> >
> > at
> > enmasse.controller.standard.StandardController.resourcesUpdated(Standa
> > rdController.java:93)
> >
> > at
> > enmasse.controller.common.WatcherVerticle.run(WatcherVerticle.java:89)
> >
> > at java.lang.Thread.run(Thread.java:748)
> >
> > ---
> >
> > Looking at StandardController.ingressToEndpoint(), line 191 is:
> >
> > builder.setHost(ingress.getSpec().getTls().get(0).getHosts().get(0));
> >
> > I guess the ingress here is the "messaging" ingress (it has a
> > "io.enmasse.certSecret" annotation set).
> >
> > It looks like this:
> >
> > ---
> >
> > {
> >
> > "apiVersion": "extensions/v1beta1",
> >
> > "kind": "Ingress",
> >
> > "metadata": {
> >
> > "annotations": {
> >
> > "addressSpace": "default",
> >
> > "io.enmasse.certSecret": "external-certs-messaging"
> >
> > },
> >
> > "creationTimestamp": "2017-08-07T14:53:31Z",
> >
> > "generation": 1,
> >
> > "name": "messaging",
> >
> > "namespace": "hono",
> >
> > "resourceVersion": "509798",
> >
> > "selfLink":
> > "/apis/extensions/v1beta1/namespaces/hono/ingresses/messaging",
> >
> > "uid": "2de5c2fb-7b80-11e7-adee-080027c98449"
> >
> > },
> >
> > "spec": {
> >
> > "backend": {
> >
> > "serviceName": "messaging",
> >
> > "servicePort": "amqps"
> >
> > }
> >
> > },
> >
> > "status": {
> >
> > "loadBalancer": {}
> >
> > }
> >
> > }
> >
> > ---
> >
> > So, no "tls" below "spec".
> >
> > Here is some more log output:
> >
> > ragent:
> >
> > ----
> >
> > Mon, 07 Aug 2017 14:55:47 GMT ragent info Router connected from
> > Router.qdrouterd-1128330706-25p37
> >
> > Mon, 07 Aug 2017 14:55:47 GMT ragent info EXTRACT, results: ,
> > attributes:
> > name,identity,type,prefix,distribution,waypoint,ingressPhase,egressPha
> > se
> >
> > Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved addresses for
> > Router.qdrouterd-1128330706-25p37: {}
> >
> > Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved link routes for
> > Router.qdrouterd-1128330706-25p37: raw->
> > [{"name":"override.lwt_in","identity":"1","type":"org.apache.qpid.disp
> > atch.router.config.linkRoute","prefix":"$lwt","distribution":"linkBala
> > nced","connection":null,"containerId":"lwt-service","dir":"in","operSt
> > atus":"active"},{"name":"override.lwt_out","identity":"2","type":"org.
> > apache.qpid.dispatch.router.config.linkRoute","prefix":"$lwt","distrib
> > ution":"linkBalanced","connection":null,"containerId":"lwt-service","d
> > ir":"out","operStatus":"active"},{"name":"override.locate","identity":
> > "3","type":"org.apache.qpid.dispatch.router.config.linkRoute","prefix"
> > :"locate","distribution":"linkBalanced","connection":"subscription-ser
> > vice","containerId":null,"dir":"out","operStatus":"active"}]
> >
> > Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved link routes for
> > Router.qdrouterd-1128330706-25p37: {}
> >
> > Mon, 07 Aug 2017 14:55:47 GMT ragent info updating addresses for
> > Router.qdrouterd-1128330706-25p37
> >
> > Mon, 07 Aug 2017 14:55:47 GMT ragent info defining address
> > telemetry/DEFAULT_TENANT on router Router.qdrouterd-1128330706-25p37
> >
> > Mon, 07 Aug 2017 14:55:47 GMT ragent info defining in link route
> > event/DEFAULT_TENANT on router Router.qdrouterd-1128330706-25p37
> >
> > Mon, 07 Aug 2017 14:55:47 GMT ragent info defining out link route
> > event/DEFAULT_TENANT on router Router.qdrouterd-1128330706-25p37
> >
> > Mon, 07 Aug 2017 14:55:47 GMT ragent info addresses updated for
> > Router.qdrouterd-1128330706-25p37
> >
> > Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved listeners for
> > Router.qdrouterd-1128330706-25p37: 172.17.0.10:55672
> >
> > Mon, 07 Aug 2017 14:55:47 GMT ragent info
> > Router.qdrouterd-1128330706-25p37 not ready for connectivity check:
> > false false
> >
> > Mon, 07 Aug 2017 14:55:47 GMT ragent info EXTRACT, results:
> >
> > [..]
> >
> > Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved addresses for
> > Router.qdrouterd-1128330706-25p37:
> >
> > [..]
> >
> > Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved link routes for
> > Router.qdrouterd-1128330706-25p37: raw-> [..]
> >
> > Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved link routes for
> > Router.qdrouterd-1128330706-25p37:
> > {"event/DEFAULT_TENANT_in":{"name":"event/
> >
> > [..]
> >
> > Mon, 07 Aug 2017 14:55:47 GMT ragent info updating addresses for
> > Router.qdrouterd-1128330706-25p37
> >
> > Mon, 07 Aug 2017 14:55:47 GMT ragent info
> > Router.qdrouterd-1128330706-25p37 not ready for connectivity check:
> > true false
> >
> > Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved connectors for
> > Router.qdrouterd-1128330706-25p37:
> >
> > Mon, 07 Aug 2017 14:55:47 GMT ragent info checking connectivity for
> > Router.qdrouterd-1128330706-25p37
> >
> > Mon, 07 Aug 2017 14:55:47 GMT ragent info checking connectors on
> > router Router.qdrouterd-1128330706-25p37, missing=, stale=
> >
> > ----
> >
> > queue-scheduler:
> >
> > ----
> >
> > 2017-08-07 14:53:39 WARN MacAddressUtil:136 - Failed to find a usable
> > hardware address from the network interfaces; using random bytes:
> > e0:37:7a:1d:7d:a2:98:48
> >
> > 2017-08-07 14:53:40 INFO QueueScheduler:95 - QueueScheduler is up and
> > running
> >
> > 2017-08-07 14:53:40 ERROR ConfigServiceClient:65 - Error connecting to
> > configuration service
> >
> > io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection
> > refused: localhost/127.0.0.1:5672
> >
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native
> > Method)
> >
> > at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> >
> > at
> > io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocket
> > Channel.java:347)
> >
> > at
> > io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnec
> > t(AbstractNioChannel.java:340)
> >
> > at
> > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java
> > :630)
> >
> > at
> > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEven
> > tLoop.java:565)
> >
> > at
> > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.jav
> > a:479)
> >
> > at
> > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441)
> >
> > at
> > io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadE
> > ventExecutor.java:858)
> >
> > at java.lang.Thread.run(Thread.java:748)
> >
> > 2017-08-07 14:53:50 INFO ConfigServiceClient:45 - Connected to the
> > configuration service
> >
> > ----
> >
> > Any idea on what the reason could be or how to investigate further?
> >
> > Best regards
> >
> > *Carsten Lohmann
> > *
> > (INST/ECS4)
> > Bosch Software Innovations GmbH | Schöneberger Ufer 89-91 | 10785
> > Berlin
> > | GERMANY| www.bosch-si.com <http://www.bosch-si.com>
> > Tel. +49 30 726112-130 | Fax +49 30 726112-100 |
> > carsten.lohmann at bosch-si.com <mailto:carsten.lohmann at bosch-si.com>
> >
> > Sitz: Berlin, Registergericht: Amtsgericht Charlottenburg; HRB 148411
> > B
> > Aufsichtsratsvorsitzender: Dr.-Ing. Thorsten Lücke; Geschäftsführung:
> > Dr.-Ing. Rainer Kallenbach, Michael Hahn
> >
> >
> >
> >
> >
> > _______________________________________________
> > enmasse mailing list
> > enmasse at redhat.com
> > https://www.redhat.com/mailman/listinfo/enmasse
> >
>
> --
> Ulf
>
> _______________________________________________
> enmasse mailing list
> enmasse at redhat.com
> https://www.redhat.com/mailman/listinfo/enmasse
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/enmasse/attachments/20170808/101f912a/attachment.htm>
More information about the enmasse
mailing list