[EnMasse] EnMasse 0.12.0: Addresses not getting ready
Ulf Lilleengen
lulf at redhat.com
Tue Aug 8 20:24:02 UTC 2017
Hi again,
Please retry with 0.12.3 which now has a fix that actually works! I did
test the previous fix also, but somehow it was still using the old docker
image even if I retagged the image, so I made a new tag instead. I'm able
to deploy 0.12.3, and isReady: true for all address types now after config
has propagated. Apologies for the wait!
Ulf
On Tue, Aug 8, 2017 at 3:40 PM, Ulf Lilleengen <lulf at redhat.com> wrote:
> Hi Carsten,
>
> I don't think the build has gone through yet, but i'll verify later today
> as I have been able to reproduce it. I'll give an update once I think it
> should work.
>
> Ulf
>
> On Tue, Aug 8, 2017 at 3:14 PM, Lohmann Carsten (INST/ECS4) <
> Carsten.Lohmann at bosch-si.com> wrote:
>
>> Hi Ulf,
>>
>> thanks for the fast reply and fix. Unfortunately, with the new version
>> there is still such an error:
>> ---
>> 2017-08-08 13:03:51 WARN WatcherVerticle:92 - Exception doing resource
>> update
>> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>> at java.util.ArrayList.rangeCheck(ArrayList.java:653)
>> at java.util.ArrayList.get(ArrayList.java:429)
>> at enmasse.controller.standard.StandardController.ingressToEndp
>> oint(StandardController.java:192)
>> at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipel
>> ine.java:193)
>> at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipel
>> ine.java:175)
>> at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(
>> ArrayList.java:1374)
>> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.
>> java:481)
>> at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPi
>> peline.java:471)
>> at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Reduc
>> eOps.java:708)
>> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.
>> java:234)
>> at java.util.stream.ReferencePipeline.collect(ReferencePipeline
>> .java:499)
>> at enmasse.controller.standard.StandardController.updateEndpoin
>> ts(StandardController.java:158)
>> at enmasse.controller.standard.StandardController.resourcesUpda
>> ted(StandardController.java:93)
>> at enmasse.controller.common.WatcherVerticle.run(WatcherVerticl
>> e.java:89)
>> at java.lang.Thread.run(Thread.java:748)
>> ---
>>
>>
>> Best regards
>>
>> Carsten Lohmann
>>
>> (INST/ECS4)
>> Bosch Software Innovations GmbH | Schöneberger Ufer 89-91 | 10785 Berlin
>> | GERMANY | www.bosch-si.com
>> Tel. +49 30 726112-130 | Fax +49 30 726112-100 |
>> carsten.lohmann at bosch-si.com
>>
>> Sitz: Berlin, Registergericht: Amtsgericht Charlottenburg; HRB 148411 B
>> Aufsichtsratsvorsitzender: Dr.-Ing. Thorsten Lücke; Geschäftsführung:
>> Dr.-Ing. Rainer Kallenbach, Michael Hahn
>>
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Ulf Lilleengen [mailto:lulf at redhat.com]
>> Gesendet: Dienstag, 8. August 2017 11:32
>> An: Lohmann Carsten (INST/ECS4) <Carsten.Lohmann at bosch-si.com>;
>> enmasse at redhat.com
>> Betreff: Re: [EnMasse] EnMasse 0.12.0: Addresses not getting ready
>>
>> Hi Carsten,
>>
>> Thanks for the detailed bug report. After looking at the code, I see that
>> the exception is what is preventing the status to get updated.
>>
>> I'll try to reproduce this as the tls entry should have been added afaict
>> from looking at the code. I will apply a fix, and tag a 0.12.1.
>> Sorry for the inconvenience!
>>
>> Best regards,
>>
>> Ulf
>>
>> On 08. aug. 2017 10:20, Lohmann Carsten (INST/ECS4) wrote:
>> > Hi,
>> >
>> > we've done an upgrade to EnMasse 0.12.0 and are now having the issue,
>> > that addresses registered via the address controller REST interface
>> > are not getting ready ("isReady" stays false).
>> >
>> > Test was done on a local minikube cluster, using the
>> > kubernetes/enmasse.yaml deployment file.
>> >
>> > When looking at the log of the address-controller, there is this
>> > exception (occurring repeatedly):
>> >
>> > ---
>> >
>> > 2017-08-07 14:53:32 WARN WatcherVerticle:92 - Exception doing
>> > resource update
>> >
>> > java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>> >
>> > at java.util.ArrayList.rangeCheck(ArrayList.java:653)
>> >
>> > at java.util.ArrayList.get(ArrayList.java:429)
>> >
>> > at
>> > enmasse.controller.standard.StandardController.ingressToEndpoint(Stand
>> > ardController.java:191)
>> >
>> > at
>> > java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:1
>> > 93)
>> >
>> > at
>> > java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:1
>> > 75)
>> >
>> > at
>> > java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.ja
>> > va:1374)
>> >
>> > at
>> > java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
>> >
>> > at
>> > java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.jav
>> > a:471)
>> >
>> > at
>> > java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:
>> > 708)
>> >
>> > at
>> > java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
>> >
>> > at
>> > java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
>> >
>> > at
>> > enmasse.controller.standard.StandardController.updateEndpoints(Standar
>> > dController.java:158)
>> >
>> > at
>> > enmasse.controller.standard.StandardController.resourcesUpdated(Standa
>> > rdController.java:93)
>> >
>> > at
>> > enmasse.controller.common.WatcherVerticle.run(WatcherVerticle.java:89)
>> >
>> > at java.lang.Thread.run(Thread.java:748)
>> >
>> > ---
>> >
>> > Looking at StandardController.ingressToEndpoint(), line 191 is:
>> >
>> > builder.setHost(ingress.getSpec().getTls().get(0).getHosts().get(0));
>> >
>> > I guess the ingress here is the "messaging" ingress (it has a
>> > "io.enmasse.certSecret" annotation set).
>> >
>> > It looks like this:
>> >
>> > ---
>> >
>> > {
>> >
>> > "apiVersion": "extensions/v1beta1",
>> >
>> > "kind": "Ingress",
>> >
>> > "metadata": {
>> >
>> > "annotations": {
>> >
>> > "addressSpace": "default",
>> >
>> > "io.enmasse.certSecret": "external-certs-messaging"
>> >
>> > },
>> >
>> > "creationTimestamp": "2017-08-07T14:53:31Z",
>> >
>> > "generation": 1,
>> >
>> > "name": "messaging",
>> >
>> > "namespace": "hono",
>> >
>> > "resourceVersion": "509798",
>> >
>> > "selfLink":
>> > "/apis/extensions/v1beta1/namespaces/hono/ingresses/messaging",
>> >
>> > "uid": "2de5c2fb-7b80-11e7-adee-080027c98449"
>> >
>> > },
>> >
>> > "spec": {
>> >
>> > "backend": {
>> >
>> > "serviceName": "messaging",
>> >
>> > "servicePort": "amqps"
>> >
>> > }
>> >
>> > },
>> >
>> > "status": {
>> >
>> > "loadBalancer": {}
>> >
>> > }
>> >
>> > }
>> >
>> > ---
>> >
>> > So, no "tls" below "spec".
>> >
>> > Here is some more log output:
>> >
>> > ragent:
>> >
>> > ----
>> >
>> > Mon, 07 Aug 2017 14:55:47 GMT ragent info Router connected from
>> > Router.qdrouterd-1128330706-25p37
>> >
>> > Mon, 07 Aug 2017 14:55:47 GMT ragent info EXTRACT, results: ,
>> > attributes:
>> > name,identity,type,prefix,distribution,waypoint,ingressPhase,egressPha
>> > se
>> >
>> > Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved addresses for
>> > Router.qdrouterd-1128330706-25p37: {}
>> >
>> > Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved link routes for
>> > Router.qdrouterd-1128330706-25p37: raw->
>> > [{"name":"override.lwt_in","identity":"1","type":"org.apache.qpid.disp
>> > atch.router.config.linkRoute","prefix":"$lwt","distribution":"linkBala
>> > nced","connection":null,"containerId":"lwt-service","dir":"in","operSt
>> > atus":"active"},{"name":"override.lwt_out","identity":"2","type":"org.
>> > apache.qpid.dispatch.router.config.linkRoute","prefix":"$lwt","distrib
>> > ution":"linkBalanced","connection":null,"containerId":"lwt-service","d
>> > ir":"out","operStatus":"active"},{"name":"override.locate","identity":
>> > "3","type":"org.apache.qpid.dispatch.router.config.linkRoute","prefix"
>> > :"locate","distribution":"linkBalanced","connection":"subscription-ser
>> > vice","containerId":null,"dir":"out","operStatus":"active"}]
>> >
>> > Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved link routes for
>> > Router.qdrouterd-1128330706-25p37: {}
>> >
>> > Mon, 07 Aug 2017 14:55:47 GMT ragent info updating addresses for
>> > Router.qdrouterd-1128330706-25p37
>> >
>> > Mon, 07 Aug 2017 14:55:47 GMT ragent info defining address
>> > telemetry/DEFAULT_TENANT on router Router.qdrouterd-1128330706-25p37
>> >
>> > Mon, 07 Aug 2017 14:55:47 GMT ragent info defining in link route
>> > event/DEFAULT_TENANT on router Router.qdrouterd-1128330706-25p37
>> >
>> > Mon, 07 Aug 2017 14:55:47 GMT ragent info defining out link route
>> > event/DEFAULT_TENANT on router Router.qdrouterd-1128330706-25p37
>> >
>> > Mon, 07 Aug 2017 14:55:47 GMT ragent info addresses updated for
>> > Router.qdrouterd-1128330706-25p37
>> >
>> > Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved listeners for
>> > Router.qdrouterd-1128330706-25p37: 172.17.0.10:55672
>> >
>> > Mon, 07 Aug 2017 14:55:47 GMT ragent info
>> > Router.qdrouterd-1128330706-25p37 not ready for connectivity check:
>> > false false
>> >
>> > Mon, 07 Aug 2017 14:55:47 GMT ragent info EXTRACT, results:
>> >
>> > [..]
>> >
>> > Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved addresses for
>> > Router.qdrouterd-1128330706-25p37:
>> >
>> > [..]
>> >
>> > Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved link routes for
>> > Router.qdrouterd-1128330706-25p37: raw-> [..]
>> >
>> > Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved link routes for
>> > Router.qdrouterd-1128330706-25p37:
>> > {"event/DEFAULT_TENANT_in":{"name":"event/
>> >
>> > [..]
>> >
>> > Mon, 07 Aug 2017 14:55:47 GMT ragent info updating addresses for
>> > Router.qdrouterd-1128330706-25p37
>> >
>> > Mon, 07 Aug 2017 14:55:47 GMT ragent info
>> > Router.qdrouterd-1128330706-25p37 not ready for connectivity check:
>> > true false
>> >
>> > Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved connectors for
>> > Router.qdrouterd-1128330706-25p37:
>> >
>> > Mon, 07 Aug 2017 14:55:47 GMT ragent info checking connectivity for
>> > Router.qdrouterd-1128330706-25p37
>> >
>> > Mon, 07 Aug 2017 14:55:47 GMT ragent info checking connectors on
>> > router Router.qdrouterd-1128330706-25p37, missing=, stale=
>> >
>> > ----
>> >
>> > queue-scheduler:
>> >
>> > ----
>> >
>> > 2017-08-07 14:53:39 WARN MacAddressUtil:136 - Failed to find a usable
>> > hardware address from the network interfaces; using random bytes:
>> > e0:37:7a:1d:7d:a2:98:48
>> >
>> > 2017-08-07 14:53:40 INFO QueueScheduler:95 - QueueScheduler is up and
>> > running
>> >
>> > 2017-08-07 14:53:40 ERROR ConfigServiceClient:65 - Error connecting to
>> > configuration service
>> >
>> > io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection
>> > refused: localhost/127.0.0.1:5672
>> >
>> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native
>> > Method)
>> >
>> > at
>> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
>> >
>> > at
>> > io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocket
>> > Channel.java:347)
>> >
>> > at
>> > io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnec
>> > t(AbstractNioChannel.java:340)
>> >
>> > at
>> > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java
>> > :630)
>> >
>> > at
>> > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEven
>> > tLoop.java:565)
>> >
>> > at
>> > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.jav
>> > a:479)
>> >
>> > at
>> > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441)
>> >
>> > at
>> > io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadE
>> > ventExecutor.java:858)
>> >
>> > at java.lang.Thread.run(Thread.java:748)
>> >
>> > 2017-08-07 14:53:50 INFO ConfigServiceClient:45 - Connected to the
>> > configuration service
>> >
>> > ----
>> >
>> > Any idea on what the reason could be or how to investigate further?
>> >
>> > Best regards
>> >
>> > *Carsten Lohmann
>> > *
>> > (INST/ECS4)
>> > Bosch Software Innovations GmbH | Schöneberger Ufer 89-91 | 10785
>> > Berlin
>> > | GERMANY| www.bosch-si.com <http://www.bosch-si.com>
>> > Tel. +49 30 726112-130 | Fax +49 30 726112-100 |
>> > carsten.lohmann at bosch-si.com <mailto:carsten.lohmann at bosch-si.com>
>> >
>> > Sitz: Berlin, Registergericht: Amtsgericht Charlottenburg; HRB 148411
>> > B
>> > Aufsichtsratsvorsitzender: Dr.-Ing. Thorsten Lücke; Geschäftsführung:
>> > Dr.-Ing. Rainer Kallenbach, Michael Hahn
>> >
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > enmasse mailing list
>> > enmasse at redhat.com
>> > https://www.redhat.com/mailman/listinfo/enmasse
>> >
>>
>> --
>> Ulf
>>
>> _______________________________________________
>> enmasse mailing list
>> enmasse at redhat.com
>> https://www.redhat.com/mailman/listinfo/enmasse
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/enmasse/attachments/20170808/35dd337b/attachment.htm>
More information about the enmasse
mailing list