<div dir="ltr">Hi again,<div> </div><div>Please retry with 0.12.3 which now has a fix that actually works! I did test the previous fix also, but somehow it was still using the old docker image even if I retagged the image, so I made a new tag instead. I'm able to deploy 0.12.3, and isReady: true for all address types now after config has propagated. Apologies for the wait! </div><div> </div><div>Ulf</div><div> </div><div> </div></div><div class="gmail_extra"> <div class="gmail_quote">On Tue, Aug 8, 2017 at 3:40 PM, Ulf Lilleengen <<a href="mailto:lulf@redhat.com" target="_blank">lulf@redhat.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Carsten,<div> </div><div>I don't think the build has gone through yet, but i'll verify later today as I have been able to reproduce it. I'll give an update once I think it should work.</div><div> </div><div>Ulf</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"> <div class="gmail_quote">On Tue, Aug 8, 2017 at 3:14 PM, Lohmann Carsten (INST/ECS4) <<a href="mailto:Carsten.Lohmann@bosch-si.com" target="_blank">Carsten.Lohmann@bosch-si.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Ulf, thanks for the fast reply and fix. Unfortunately, with the new version there is still such an error: --- 2017-08-08 13:03:51 WARN WatcherVerticle:92 - Exception doing resource update java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 at java.util.ArrayList.rangeCheck(ArrayList.java:653) at java.util.ArrayList.get(ArrayList.java:429) at <a href="http://enmasse.controller.standard.St">enmasse.controller.standard.St</a>andardController.ingressToEndpoint(StandardController.java:192) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) at <a href="http://enmasse.controller.standard.St">enmasse.controller.standard.St</a>andardController.updateEndpoints(StandardController.java:158) at <a href="http://enmasse.controller.standard.St">enmasse.controller.standard.St</a>andardController.resourcesUpdated(StandardController.java:93) at enmasse.controller.common.WatcherVerticle.run(WatcherVerticle.java:89) at java.lang.Thread.run(Thread.java:748) --- Best regards Carsten Lohmann (INST/ECS4) Bosch Software Innovations GmbH | Schöneberger Ufer 89-91 | 10785 Berlin | GERMANY | <a href="http://www.bosch-si.com" rel="noreferrer" target="_blank">www.bosch-si.com</a> Tel. <a href="tel:%2B49%2030%20726112-130" value="+4930726112130" target="_blank">+49 30 726112-130</a> | Fax <a href="tel:%2B49%2030%20726112-100" value="+4930726112100" target="_blank">+49 30 726112-100</a> | <a href="mailto:carsten.lohmann@bosch-si.com" target="_blank">carsten.lohmann@bosch-si.com</a> Sitz: Berlin, Registergericht: Amtsgericht Charlottenburg; HRB 148411 B Aufsichtsratsvorsitzender: Dr.-Ing. Thorsten Lücke; Geschäftsführung: Dr.-Ing. Rainer Kallenbach, Michael Hahn -----Ursprüngliche Nachricht----- Von: Ulf Lilleengen [mailto:<a href="mailto:lulf@redhat.com" target="_blank">lulf@redhat.com</a>] Gesendet: Dienstag, 8. August 2017 11:32 An: Lohmann Carsten (INST/ECS4) <<a href="mailto:Carsten.Lohmann@bosch-si.com" target="_blank">Carsten.Lohmann@bosch-si.com</a>>; <a href="mailto:enmasse@redhat.com" target="_blank">enmasse@redhat.com</a> Betreff: Re: [EnMasse] EnMasse 0.12.0: Addresses not getting ready <div class="m_-4154368467929186218HOEnZb"><div class="m_-4154368467929186218h5"> Hi Carsten, Thanks for the detailed bug report. After looking at the code, I see that the exception is what is preventing the status to get updated. I'll try to reproduce this as the tls entry should have been added afaict from looking at the code. I will apply a fix, and tag a 0.12.1. Sorry for the inconvenience! Best regards, Ulf On 08. aug. 2017 10:20, Lohmann Carsten (INST/ECS4) wrote: > Hi, > > we've done an upgrade to EnMasse 0.12.0 and are now having the issue, > that addresses registered via the address controller REST interface > are not getting ready ("isReady" stays false). > > Test was done on a local minikube cluster, using the > kubernetes/enmasse.yaml deployment file. > > When looking at the log of the address-controller, there is this > exception (occurring repeatedly): > > --- > > 2017-08-07 14:53:32 WARN WatcherVerticle:92 - Exception doing > resource update > > java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 > > at java.util.ArrayList.rangeCheck(ArrayList.java:653) > > at java.util.ArrayList.get(ArrayList.java:429) > > at > <a href="http://enmasse.controller.standard.St">enmasse.controller.standard.St</a>andardController.ingressToEndpoint(Stand > ardController.java:191) > > at > java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:1 > 93) > > at > java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:1 > 75) > > at > java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.ja > va:1374) > > at > java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) > > at > java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.jav > a:471) > > at > java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java: > 708) > > at > java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) > > at > java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) > > at > <a href="http://enmasse.controller.standard.St">enmasse.controller.standard.St</a>andardController.updateEndpoints(Standar > dController.java:158) > > at > <a href="http://enmasse.controller.standard.St">enmasse.controller.standard.St</a>andardController.resourcesUpdated(Standa > rdController.java:93) > > at > enmasse.controller.common.WatcherVerticle.run(WatcherVerticle.java:89) > > at java.lang.Thread.run(Thread.java:748) > > --- > > Looking at StandardController.ingressToEndpoint(), line 191 is: > > builder.setHost(ingress.getSpec().getTls().get(0).getHosts().get(0)); > > I guess the ingress here is the "messaging" ingress (it has a > "io.enmasse.certSecret" annotation set). > > It looks like this: > > --- > > { > > "apiVersion": "extensions/v1beta1", > > "kind": "Ingress", > > "metadata": { > > "annotations": { > > "addressSpace": "default", > > "io.enmasse.certSecret": "external-certs-messaging" > > }, > > "creationTimestamp": "2017-08-07T14:53:31Z", > > "generation": 1, > > "name": "messaging", > > "namespace": "hono", > > "resourceVersion": "509798", > > "selfLink": > "/apis/extensions/v1beta1/namespaces/hono/ingresses/messaging", > > "uid": "2de5c2fb-7b80-11e7-adee-080027c98449" > > }, > > "spec": { > > "backend": { > > "serviceName": "messaging", > > "servicePort": "amqps" > > } > > }, > > "status": { > > "loadBalancer": {} > > } > > } > > --- > > So, no "tls" below "spec". > > Here is some more log output: > > ragent: > > ---- > > Mon, 07 Aug 2017 14:55:47 GMT ragent info Router connected from > Router.qdrouterd-1128330706-25p37 > > Mon, 07 Aug 2017 14:55:47 GMT ragent info EXTRACT, results: , > attributes: > name,identity,type,prefix,distribution,waypoint,ingressPhase,egressPha > se > > Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved addresses for > Router.qdrouterd-1128330706-25p37: {} > > Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved link routes for > Router.qdrouterd-1128330706-25p37: raw-> > [{"name":"override.lwt_in","identity":"1","type":"org.apache.qpid.disp > atch.router.config.linkRoute","prefix":"$lwt","distribution":"linkBala > nced","connection":null,"containerId":"lwt-service","dir":"in","operSt > atus":"active"},{"name":"override.lwt_out","identity":"2","type":"org. > <a href="http://apache.qpid.dispatch.router.co">apache.qpid.dispatch.router.co</a>nfig.linkRoute","prefix":"$lwt","distrib > ution":"linkBalanced","connection":null,"containerId":"lwt-service","d </div></div><div class="m_-4154368467929186218HOEnZb"><div class="m_-4154368467929186218h5">> ir":"out","operStatus":"active"},{"name":"override.locate","identity": > "3","type":"org.apache.qpid.dispatch.router.config.linkRoute","prefix" > :"locate","distribution":"linkBalanced","connection":"subscription-ser > vice","containerId":null,"dir":"out","operStatus":"active"}] > > Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved link routes for > Router.qdrouterd-1128330706-25p37: {} > > Mon, 07 Aug 2017 14:55:47 GMT ragent info updating addresses for > Router.qdrouterd-1128330706-25p37 > > Mon, 07 Aug 2017 14:55:47 GMT ragent info defining address > telemetry/DEFAULT_TENANT on router Router.qdrouterd-1128330706-25p37 > > Mon, 07 Aug 2017 14:55:47 GMT ragent info defining in link route > event/DEFAULT_TENANT on router Router.qdrouterd-1128330706-25p37 > > Mon, 07 Aug 2017 14:55:47 GMT ragent info defining out link route > event/DEFAULT_TENANT on router Router.qdrouterd-1128330706-25p37 > > Mon, 07 Aug 2017 14:55:47 GMT ragent info addresses updated for > Router.qdrouterd-1128330706-25p37 > > Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved listeners for > Router.qdrouterd-1128330706-25p37: <a href="http://172.17.0.10:55672" rel="noreferrer" target="_blank">172.17.0.10:55672</a> > > Mon, 07 Aug 2017 14:55:47 GMT ragent info > Router.qdrouterd-1128330706-25p37 not ready for connectivity check: > false false > > Mon, 07 Aug 2017 14:55:47 GMT ragent info EXTRACT, results: > > [..] > > Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved addresses for > Router.qdrouterd-1128330706-25p37: > > [..] > > Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved link routes for > Router.qdrouterd-1128330706-25p37: raw-> [..] > > Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved link routes for > Router.qdrouterd-1128330706-25p37: > {"event/DEFAULT_TENANT_in":{"name":"event/ > > [..] > > Mon, 07 Aug 2017 14:55:47 GMT ragent info updating addresses for > Router.qdrouterd-1128330706-25p37 > > Mon, 07 Aug 2017 14:55:47 GMT ragent info > Router.qdrouterd-1128330706-25p37 not ready for connectivity check: > true false > > Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved connectors for > Router.qdrouterd-1128330706-25p37: > > Mon, 07 Aug 2017 14:55:47 GMT ragent info checking connectivity for > Router.qdrouterd-1128330706-25p37 > > Mon, 07 Aug 2017 14:55:47 GMT ragent info checking connectors on > router Router.qdrouterd-1128330706-25p37, missing=, stale= > > ---- > > queue-scheduler: > > ---- > > 2017-08-07 14:53:39 WARN MacAddressUtil:136 - Failed to find a usable > hardware address from the network interfaces; using random bytes: > e0:37:7a:1d:7d:a2:98:48 > > 2017-08-07 14:53:40 INFO QueueScheduler:95 - QueueScheduler is up and > running > > 2017-08-07 14:53:40 ERROR ConfigServiceClient:65 - Error connecting to > configuration service > > io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection > refused: localhost/<a href="http://127.0.0.1:5672" rel="noreferrer" target="_blank">127.0.0.1:5672</a> > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native > Method) > > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) > > at > <a href="http://io.netty.channel.socket.nio.Ni">io.netty.channel.socket.nio.Ni</a>oSocketChannel.doFinishConnect(NioSocket > Channel.java:347) > > at > io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnec > t(AbstractNioChannel.java:340) > > at > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java > :630) > > at > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEven > tLoop.java:565) > > at > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.jav > a:479) > > at > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441) > > at > io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadE > ventExecutor.java:858) > > at java.lang.Thread.run(Thread.java:748) > > 2017-08-07 14:53:50 INFO ConfigServiceClient:45 - Connected to the > configuration service > > ---- > > Any idea on what the reason could be or how to investigate further? > > Best regards > > *Carsten Lohmann > * > (INST/ECS4) > Bosch Software Innovations GmbH | Schöneberger Ufer 89-91 | 10785 > Berlin > | GERMANY| <a href="http://www.bosch-si.com" rel="noreferrer" target="_blank">www.bosch-si.com</a> <<a href="http://www.bosch-si.com" rel="noreferrer" target="_blank">http://www.bosch-si.com</a>> > Tel. <a href="tel:%2B49%2030%20726112-130" value="+4930726112130" target="_blank">+49 30 726112-130</a> | Fax <a href="tel:%2B49%2030%20726112-100" value="+4930726112100" target="_blank">+49 30 726112-100</a> | > <a href="mailto:carsten.lohmann@bosch-si.com" target="_blank">carsten.lohmann@bosch-si.com</a> <mailto:<a href="mailto:carsten.lohmann@bosch-si.com" target="_blank">carsten.lohmann@bosch-si.com</a>> > > Sitz: Berlin, Registergericht: Amtsgericht Charlottenburg; HRB 148411 > B > Aufsichtsratsvorsitzender: Dr.-Ing. Thorsten Lücke; Geschäftsführung: > Dr.-Ing. Rainer Kallenbach, Michael Hahn > > > > > > _______________________________________________ > enmasse mailing list > <a href="mailto:enmasse@redhat.com" target="_blank">enmasse@redhat.com</a> > <a href="https://www.redhat.com/mailman/listinfo/enmasse" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/enmasse</a> > -- Ulf _______________________________________________ enmasse mailing list <a href="mailto:enmasse@redhat.com" target="_blank">enmasse@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/enmasse" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/enmasse</a> </div></div></blockquote></div> </div> </div></div></blockquote></div> </div>