<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="#0563C1" vlink="#954F72"> <div class="WordSection1"> Hi,<o:p></o:p> <o:p> </o:p> we've done an upgrade to EnMasse 0.12.0 and are now having the issue, that addresses registered via the address controller REST interface are not getting ready ("isReady" stays false).<o:p></o:p> <o:p> </o:p> Test was done on a local minikube cluster, using the kubernetes/enmasse.yaml deployment file.<o:p></o:p> <o:p> </o:p> <o:p> </o:p> When looking at the log of the address-controller, there is this exception (occurring repeatedly):<o:p></o:p> ---<o:p></o:p> 2017-08-07 14:53:32 WARN WatcherVerticle:92 - Exception doing resource update<o:p></o:p> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0<o:p></o:p> at java.util.ArrayList.rangeCheck(ArrayList.java:653)<o:p></o:p> at java.util.ArrayList.get(ArrayList.java:429)<o:p></o:p> at enmasse.controller.standard.StandardController.ingressToEndpoint(StandardController.java:191)<o:p></o:p> at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)<o:p></o:p> at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)<o:p></o:p> at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)<o:p></o:p> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)<o:p></o:p> at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)<o:p></o:p> at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)<o:p></o:p> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)<o:p></o:p> at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)<o:p></o:p> at enmasse.controller.standard.StandardController.updateEndpoints(StandardController.java:158)<o:p></o:p> at enmasse.controller.standard.StandardController.resourcesUpdated(StandardController.java:93)<o:p></o:p> at enmasse.controller.common.WatcherVerticle.run(WatcherVerticle.java:89)<o:p></o:p> at java.lang.Thread.run(Thread.java:748) <o:p></o:p> ---<o:p></o:p> <o:p> </o:p> Looking at StandardController.ingressToEndpoint(), line 191 is:<o:p></o:p> builder.setHost(ingress.getSpec().getTls().get(0).getHosts().get(0));<o:p></o:p> I guess the ingress here is the "messaging" ingress (it has a "io.enmasse.certSecret" annotation set).<o:p></o:p> It looks like this:<o:p></o:p> ---<o:p></o:p> {<o:p></o:p> "apiVersion": "extensions/v1beta1",<o:p></o:p> "kind": "Ingress",<o:p></o:p> "metadata": {<o:p></o:p> "annotations": {<o:p></o:p> "addressSpace": "default",<o:p></o:p> "io.enmasse.certSecret": "external-certs-messaging"<o:p></o:p> },<o:p></o:p> "creationTimestamp": "2017-08-07T14:53:31Z",<o:p></o:p> "generation": 1,<o:p></o:p> "name": "messaging",<o:p></o:p> "namespace": "hono",<o:p></o:p> "resourceVersion": "509798",<o:p></o:p> "selfLink": "/apis/extensions/v1beta1/namespaces/hono/ingresses/messaging",<o:p></o:p> "uid": "2de5c2fb-7b80-11e7-adee-080027c98449"<o:p></o:p> },<o:p></o:p> "spec": {<o:p></o:p> "backend": {<o:p></o:p> "serviceName": "messaging",<o:p></o:p> "servicePort": "amqps"<o:p></o:p> }<o:p></o:p> },<o:p></o:p> "status": {<o:p></o:p> "loadBalancer": {}<o:p></o:p> }<o:p></o:p> }<o:p></o:p> ---<o:p></o:p> So, no "tls" below "spec".<o:p></o:p> <o:p> </o:p> Here is some more log output:<o:p></o:p> <o:p> </o:p> ragent:<o:p></o:p> ----<o:p></o:p> Mon, 07 Aug 2017 14:55:47 GMT ragent info Router connected from Router.qdrouterd-1128330706-25p37<o:p></o:p> Mon, 07 Aug 2017 14:55:47 GMT ragent info EXTRACT, results: , attributes: name,identity,type,prefix,distribution,waypoint,ingressPhase,egressPhase<o:p></o:p> Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved addresses for Router.qdrouterd-1128330706-25p37: {}<o:p></o:p> Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved link routes for Router.qdrouterd-1128330706-25p37: raw-> [{"name":"override.lwt_in","identity":"1","type":"org.apache.qpid.dispatch.router.config.linkRoute","prefix":"$lwt","distribution":"linkBalanced","connection":null,"containerId":"lwt-service","dir":"in","operStatus":"active"},{"name":"override.lwt_out","identity":"2","type":"org.apache.qpid.dispatch.router.config.linkRoute","prefix":"$lwt","distribution":"linkBalanced","connection":null,"containerId":"lwt-service","dir":"out","operStatus":"active"},{"name":"override.locate","identity":"3","type":"org.apache.qpid.dispatch.router.config.linkRoute","prefix":"locate","distribution":"linkBalanced","connection":"subscription-service","containerId":null,"dir":"out","operStatus":"active"}]<o:p></o:p> Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved link routes for Router.qdrouterd-1128330706-25p37: {}<o:p></o:p> Mon, 07 Aug 2017 14:55:47 GMT ragent info updating addresses for Router.qdrouterd-1128330706-25p37<o:p></o:p> Mon, 07 Aug 2017 14:55:47 GMT ragent info defining address telemetry/DEFAULT_TENANT on router Router.qdrouterd-1128330706-25p37<o:p></o:p> Mon, 07 Aug 2017 14:55:47 GMT ragent info defining in link route event/DEFAULT_TENANT on router Router.qdrouterd-1128330706-25p37<o:p></o:p> Mon, 07 Aug 2017 14:55:47 GMT ragent info defining out link route event/DEFAULT_TENANT on router Router.qdrouterd-1128330706-25p37<o:p></o:p> Mon, 07 Aug 2017 14:55:47 GMT ragent info addresses updated for Router.qdrouterd-1128330706-25p37<o:p></o:p> Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved listeners for Router.qdrouterd-1128330706-25p37: 172.17.0.10:55672<o:p></o:p> Mon, 07 Aug 2017 14:55:47 GMT ragent info Router.qdrouterd-1128330706-25p37 not ready for connectivity check: false false<o:p></o:p> Mon, 07 Aug 2017 14:55:47 GMT ragent info EXTRACT, results:<o:p></o:p> [..]<o:p></o:p> Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved addresses for Router.qdrouterd-1128330706-25p37: <o:p></o:p> [..]<o:p></o:p> Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved link routes for Router.qdrouterd-1128330706-25p37: raw-> [..]<o:p></o:p> Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved link routes for Router.qdrouterd-1128330706-25p37: {"event/DEFAULT_TENANT_in":{"name":"event/<o:p></o:p> [..]<o:p></o:p> Mon, 07 Aug 2017 14:55:47 GMT ragent info updating addresses for Router.qdrouterd-1128330706-25p37<o:p></o:p> Mon, 07 Aug 2017 14:55:47 GMT ragent info Router.qdrouterd-1128330706-25p37 not ready for connectivity check: true false<o:p></o:p> Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved connectors for Router.qdrouterd-1128330706-25p37: <o:p></o:p> Mon, 07 Aug 2017 14:55:47 GMT ragent info checking connectivity for Router.qdrouterd-1128330706-25p37<o:p></o:p> Mon, 07 Aug 2017 14:55:47 GMT ragent info checking connectors on router Router.qdrouterd-1128330706-25p37, missing=, stale=<o:p></o:p> ----<o:p></o:p> <o:p> </o:p> queue-scheduler:<o:p></o:p> ----<o:p></o:p> 2017-08-07 14:53:39 WARN MacAddressUtil:136 - Failed to find a usable hardware address from the network interfaces; using random bytes: e0:37:7a:1d:7d:a2:98:48<o:p></o:p> 2017-08-07 14:53:40 INFO QueueScheduler:95 - QueueScheduler is up and running<o:p></o:p> 2017-08-07 14:53:40 ERROR ConfigServiceClient:65 - Error connecting to configuration service<o:p></o:p> io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5672<o:p></o:p> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)<o:p></o:p> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)<o:p></o:p> at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:347)<o:p></o:p> at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)<o:p></o:p> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:630)<o:p></o:p> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:565)<o:p></o:p> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:479)<o:p></o:p> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441)<o:p></o:p> at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)<o:p></o:p> at java.lang.Thread.run(Thread.java:748)<o:p></o:p> 2017-08-07 14:53:50 INFO ConfigServiceClient:45 - Connected to the configuration service<o:p></o:p> ----<o:p></o:p> <o:p> </o:p> Any idea on what the reason could be or how to investigate further?<o:p></o:p> <o:p> </o:p> Best regards Carsten Lohmann (INST/ECS4) Bosch Software Innovations GmbH | Sch�neberger Ufer 89-91 | 10785 Berlin | GERMANY | <a href="http://www.bosch-si.com">www.bosch-si.com</a> Tel. +49 30 726112-130 | Fax +49 30 726112-100 | <a href="mailto:carsten.lohmann@bosch-si.com">carsten.lohmann@bosch-si.com</a> Sitz: Berlin, Registergericht: Amtsgericht Charlottenburg; HRB 148411 B Aufsichtsratsvorsitzender: Dr.-Ing. Thorsten L�cke; Gesch�ftsf�hrung: Dr.-Ing. Rainer Kallenbach, Michael Hahn <o:p></o:p> </div> </body> </html>