[EnMasse] EnMasse 0.12.0: Addresses not getting ready
Lohmann Carsten (INST/ECS4)
Carsten.Lohmann at bosch-si.com
Wed Aug 9 09:10:00 UTC 2017
Thanks! Now it works.
Best regards
Carsten Lohmann
(INST/ECS4)
Bosch Software Innovations GmbH | Schöneberger Ufer 89-91 | 10785 Berlin | GERMANY | www.bosch-si.com<http://www.bosch-si.com>
Tel. +49 30 726112-130 | Fax +49 30 726112-100 | carsten.lohmann at bosch-si.com<mailto:carsten.lohmann at bosch-si.com>
Sitz: Berlin, Registergericht: Amtsgericht Charlottenburg; HRB 148411 B
Aufsichtsratsvorsitzender: Dr.-Ing. Thorsten Lücke; Geschäftsführung: Dr.-Ing. Rainer Kallenbach, Michael Hahn
Von: Ulf Lilleengen [mailto:lulf at redhat.com]
Gesendet: Dienstag, 8. August 2017 22:24
An: Lohmann Carsten (INST/ECS4) <Carsten.Lohmann at bosch-si.com>
Cc: enmasse at redhat.com
Betreff: Re: [EnMasse] EnMasse 0.12.0: Addresses not getting ready
Hi again,
Please retry with 0.12.3 which now has a fix that actually works! I did test the previous fix also, but somehow it was still using the old docker image even if I retagged the image, so I made a new tag instead. I'm able to deploy 0.12.3, and isReady: true for all address types now after config has propagated. Apologies for the wait!
Ulf
On Tue, Aug 8, 2017 at 3:40 PM, Ulf Lilleengen <lulf at redhat.com<mailto:lulf at redhat.com>> wrote:
Hi Carsten,
I don't think the build has gone through yet, but i'll verify later today as I have been able to reproduce it. I'll give an update once I think it should work.
Ulf
On Tue, Aug 8, 2017 at 3:14 PM, Lohmann Carsten (INST/ECS4) <Carsten.Lohmann at bosch-si.com<mailto:Carsten.Lohmann at bosch-si.com>> wrote:
Hi Ulf,
thanks for the fast reply and fix. Unfortunately, with the new version there is still such an error:
---
2017-08-08 13:03:51 WARN WatcherVerticle:92 - Exception doing resource update
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(ArrayList.java:653)
at java.util.ArrayList.get(ArrayList.java:429)
at enmasse.controller.standard.St<http://enmasse.controller.standard.St>andardController.ingressToEndpoint(StandardController.java:192)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at enmasse.controller.standard.St<http://enmasse.controller.standard.St>andardController.updateEndpoints(StandardController.java:158)
at enmasse.controller.standard.St<http://enmasse.controller.standard.St>andardController.resourcesUpdated(StandardController.java:93)
at enmasse.controller.common.WatcherVerticle.run(WatcherVerticle.java:89)
at java.lang.Thread.run(Thread.java:748)
---
Best regards
Carsten Lohmann
(INST/ECS4)
Bosch Software Innovations GmbH | Schöneberger Ufer 89-91 | 10785 Berlin | GERMANY | www.bosch-si.com<http://www.bosch-si.com>
Tel. +49 30 726112-130<tel:%2B49%2030%20726112-130> | Fax +49 30 726112-100<tel:%2B49%2030%20726112-100> | carsten.lohmann at bosch-si.com<mailto:carsten.lohmann at bosch-si.com>
Sitz: Berlin, Registergericht: Amtsgericht Charlottenburg; HRB 148411 B
Aufsichtsratsvorsitzender: Dr.-Ing. Thorsten Lücke; Geschäftsführung: Dr.-Ing. Rainer Kallenbach, Michael Hahn
-----Ursprüngliche Nachricht-----
Von: Ulf Lilleengen [mailto:lulf at redhat.com<mailto:lulf at redhat.com>]
Gesendet: Dienstag, 8. August 2017 11:32
An: Lohmann Carsten (INST/ECS4) <Carsten.Lohmann at bosch-si.com<mailto:Carsten.Lohmann at bosch-si.com>>; enmasse at redhat.com<mailto:enmasse at redhat.com>
Betreff: Re: [EnMasse] EnMasse 0.12.0: Addresses not getting ready
Hi Carsten,
Thanks for the detailed bug report. After looking at the code, I see that the exception is what is preventing the status to get updated.
I'll try to reproduce this as the tls entry should have been added afaict from looking at the code. I will apply a fix, and tag a 0.12.1.
Sorry for the inconvenience!
Best regards,
Ulf
On 08. aug. 2017 10:20, Lohmann Carsten (INST/ECS4) wrote:
> Hi,
>
> we've done an upgrade to EnMasse 0.12.0 and are now having the issue,
> that addresses registered via the address controller REST interface
> are not getting ready ("isReady" stays false).
>
> Test was done on a local minikube cluster, using the
> kubernetes/enmasse.yaml deployment file.
>
> When looking at the log of the address-controller, there is this
> exception (occurring repeatedly):
>
> ---
>
> 2017-08-07 14:53:32 WARN WatcherVerticle:92 - Exception doing
> resource update
>
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>
> at java.util.ArrayList.rangeCheck(ArrayList.java:653)
>
> at java.util.ArrayList.get(ArrayList.java:429)
>
> at
> enmasse.controller.standard.St<http://enmasse.controller.standard.St>andardController.ingressToEndpoint(Stand
> ardController.java:191)
>
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:1
> 93)
>
> at
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:1
> 75)
>
> at
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.ja
> va:1374)
>
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
>
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.jav
> a:471)
>
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:
> 708)
>
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
>
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
>
> at
> enmasse.controller.standard.St<http://enmasse.controller.standard.St>andardController.updateEndpoints(Standar
> dController.java:158)
>
> at
> enmasse.controller.standard.St<http://enmasse.controller.standard.St>andardController.resourcesUpdated(Standa
> rdController.java:93)
>
> at
> enmasse.controller.common.WatcherVerticle.run(WatcherVerticle.java:89)
>
> at java.lang.Thread.run(Thread.java:748)
>
> ---
>
> Looking at StandardController.ingressToEndpoint(), line 191 is:
>
> builder.setHost(ingress.getSpec().getTls().get(0).getHosts().get(0));
>
> I guess the ingress here is the "messaging" ingress (it has a
> "io.enmasse.certSecret" annotation set).
>
> It looks like this:
>
> ---
>
> {
>
> "apiVersion": "extensions/v1beta1",
>
> "kind": "Ingress",
>
> "metadata": {
>
> "annotations": {
>
> "addressSpace": "default",
>
> "io.enmasse.certSecret": "external-certs-messaging"
>
> },
>
> "creationTimestamp": "2017-08-07T14:53:31Z",
>
> "generation": 1,
>
> "name": "messaging",
>
> "namespace": "hono",
>
> "resourceVersion": "509798",
>
> "selfLink":
> "/apis/extensions/v1beta1/namespaces/hono/ingresses/messaging",
>
> "uid": "2de5c2fb-7b80-11e7-adee-080027c98449"
>
> },
>
> "spec": {
>
> "backend": {
>
> "serviceName": "messaging",
>
> "servicePort": "amqps"
>
> }
>
> },
>
> "status": {
>
> "loadBalancer": {}
>
> }
>
> }
>
> ---
>
> So, no "tls" below "spec".
>
> Here is some more log output:
>
> ragent:
>
> ----
>
> Mon, 07 Aug 2017 14:55:47 GMT ragent info Router connected from
> Router.qdrouterd-1128330706-25p37
>
> Mon, 07 Aug 2017 14:55:47 GMT ragent info EXTRACT, results: ,
> attributes:
> name,identity,type,prefix,distribution,waypoint,ingressPhase,egressPha
> se
>
> Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved addresses for
> Router.qdrouterd-1128330706-25p37: {}
>
> Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved link routes for
> Router.qdrouterd-1128330706-25p37: raw->
> [{"name":"override.lwt_in","identity":"1","type":"org.apache.qpid.disp
> atch.router.config.linkRoute","prefix":"$lwt","distribution":"linkBala
> nced","connection":null,"containerId":"lwt-service","dir":"in","operSt
> atus":"active"},{"name":"override.lwt_out","identity":"2","type":"org.
> apache.qpid.dispatch.router.co<http://apache.qpid.dispatch.router.co>nfig.linkRoute","prefix":"$lwt","distrib
> ution":"linkBalanced","connection":null,"containerId":"lwt-service","d
> ir":"out","operStatus":"active"},{"name":"override.locate","identity":
> "3","type":"org.apache.qpid.dispatch.router.config.linkRoute","prefix"
> :"locate","distribution":"linkBalanced","connection":"subscription-ser
> vice","containerId":null,"dir":"out","operStatus":"active"}]
>
> Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved link routes for
> Router.qdrouterd-1128330706-25p37: {}
>
> Mon, 07 Aug 2017 14:55:47 GMT ragent info updating addresses for
> Router.qdrouterd-1128330706-25p37
>
> Mon, 07 Aug 2017 14:55:47 GMT ragent info defining address
> telemetry/DEFAULT_TENANT on router Router.qdrouterd-1128330706-25p37
>
> Mon, 07 Aug 2017 14:55:47 GMT ragent info defining in link route
> event/DEFAULT_TENANT on router Router.qdrouterd-1128330706-25p37
>
> Mon, 07 Aug 2017 14:55:47 GMT ragent info defining out link route
> event/DEFAULT_TENANT on router Router.qdrouterd-1128330706-25p37
>
> Mon, 07 Aug 2017 14:55:47 GMT ragent info addresses updated for
> Router.qdrouterd-1128330706-25p37
>
> Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved listeners for
> Router.qdrouterd-1128330706-25p37: 172.17.0.10:55672<http://172.17.0.10:55672>
>
> Mon, 07 Aug 2017 14:55:47 GMT ragent info
> Router.qdrouterd-1128330706-25p37 not ready for connectivity check:
> false false
>
> Mon, 07 Aug 2017 14:55:47 GMT ragent info EXTRACT, results:
>
> [..]
>
> Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved addresses for
> Router.qdrouterd-1128330706-25p37:
>
> [..]
>
> Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved link routes for
> Router.qdrouterd-1128330706-25p37: raw-> [..]
>
> Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved link routes for
> Router.qdrouterd-1128330706-25p37:
> {"event/DEFAULT_TENANT_in":{"name":"event/
>
> [..]
>
> Mon, 07 Aug 2017 14:55:47 GMT ragent info updating addresses for
> Router.qdrouterd-1128330706-25p37
>
> Mon, 07 Aug 2017 14:55:47 GMT ragent info
> Router.qdrouterd-1128330706-25p37 not ready for connectivity check:
> true false
>
> Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved connectors for
> Router.qdrouterd-1128330706-25p37:
>
> Mon, 07 Aug 2017 14:55:47 GMT ragent info checking connectivity for
> Router.qdrouterd-1128330706-25p37
>
> Mon, 07 Aug 2017 14:55:47 GMT ragent info checking connectors on
> router Router.qdrouterd-1128330706-25p37, missing=, stale=
>
> ----
>
> queue-scheduler:
>
> ----
>
> 2017-08-07 14:53:39 WARN MacAddressUtil:136 - Failed to find a usable
> hardware address from the network interfaces; using random bytes:
> e0:37:7a:1d:7d:a2:98:48
>
> 2017-08-07 14:53:40 INFO QueueScheduler:95 - QueueScheduler is up and
> running
>
> 2017-08-07 14:53:40 ERROR ConfigServiceClient:65 - Error connecting to
> configuration service
>
> io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection
> refused: localhost/127.0.0.1:5672<http://127.0.0.1:5672>
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native
> Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
>
> at
> io.netty.channel.socket.nio.Ni<http://io.netty.channel.socket.nio.Ni>oSocketChannel.doFinishConnect(NioSocket
> Channel.java:347)
>
> at
> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnec
> t(AbstractNioChannel.java:340)
>
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java
> :630)
>
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEven
> tLoop.java:565)
>
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.jav
> a:479)
>
> at
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441)
>
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadE
> ventExecutor.java:858)
>
> at java.lang.Thread.run(Thread.java:748)
>
> 2017-08-07 14:53:50 INFO ConfigServiceClient:45 - Connected to the
> configuration service
>
> ----
>
> Any idea on what the reason could be or how to investigate further?
>
> Best regards
>
> *Carsten Lohmann
> *
> (INST/ECS4)
> Bosch Software Innovations GmbH | Schöneberger Ufer 89-91 | 10785
> Berlin
> | GERMANY| www.bosch-si.com<http://www.bosch-si.com> <http://www.bosch-si.com>
> Tel. +49 30 726112-130<tel:%2B49%2030%20726112-130> | Fax +49 30 726112-100<tel:%2B49%2030%20726112-100> |
> carsten.lohmann at bosch-si.com<mailto:carsten.lohmann at bosch-si.com> <mailto:carsten.lohmann at bosch-si.com<mailto:carsten.lohmann at bosch-si.com>>
>
> Sitz: Berlin, Registergericht: Amtsgericht Charlottenburg; HRB 148411
> B
> Aufsichtsratsvorsitzender: Dr.-Ing. Thorsten Lücke; Geschäftsführung:
> Dr.-Ing. Rainer Kallenbach, Michael Hahn
>
>
>
>
>
> _______________________________________________
> enmasse mailing list
> enmasse at redhat.com<mailto:enmasse at redhat.com>
> https://www.redhat.com/mailman/listinfo/enmasse
>
--
Ulf
_______________________________________________
enmasse mailing list
enmasse at redhat.com<mailto:enmasse at redhat.com>
https://www.redhat.com/mailman/listinfo/enmasse
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/enmasse/attachments/20170809/41935bef/attachment.htm>
More information about the enmasse
mailing list