[EnMasse] EnMasse 0.12.0: Addresses not getting ready
Lohmann Carsten (INST/ECS4)
Carsten.Lohmann at bosch-si.com
Tue Aug 8 08:20:27 UTC 2017
Hi,
we've done an upgrade to EnMasse 0.12.0 and are now having the issue, that addresses registered via the address controller REST interface are not getting ready ("isReady" stays false).
Test was done on a local minikube cluster, using the kubernetes/enmasse.yaml deployment file.
When looking at the log of the address-controller, there is this exception (occurring repeatedly):
---
2017-08-07 14:53:32 WARN WatcherVerticle:92 - Exception doing resource update
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(ArrayList.java:653)
at java.util.ArrayList.get(ArrayList.java:429)
at enmasse.controller.standard.StandardController.ingressToEndpoint(StandardController.java:191)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at enmasse.controller.standard.StandardController.updateEndpoints(StandardController.java:158)
at enmasse.controller.standard.StandardController.resourcesUpdated(StandardController.java:93)
at enmasse.controller.common.WatcherVerticle.run(WatcherVerticle.java:89)
at java.lang.Thread.run(Thread.java:748)
---
Looking at StandardController.ingressToEndpoint(), line 191 is:
builder.setHost(ingress.getSpec().getTls().get(0).getHosts().get(0));
I guess the ingress here is the "messaging" ingress (it has a "io.enmasse.certSecret" annotation set).
It looks like this:
---
{
"apiVersion": "extensions/v1beta1",
"kind": "Ingress",
"metadata": {
"annotations": {
"addressSpace": "default",
"io.enmasse.certSecret": "external-certs-messaging"
},
"creationTimestamp": "2017-08-07T14:53:31Z",
"generation": 1,
"name": "messaging",
"namespace": "hono",
"resourceVersion": "509798",
"selfLink": "/apis/extensions/v1beta1/namespaces/hono/ingresses/messaging",
"uid": "2de5c2fb-7b80-11e7-adee-080027c98449"
},
"spec": {
"backend": {
"serviceName": "messaging",
"servicePort": "amqps"
}
},
"status": {
"loadBalancer": {}
}
}
---
So, no "tls" below "spec".
Here is some more log output:
ragent:
----
Mon, 07 Aug 2017 14:55:47 GMT ragent info Router connected from Router.qdrouterd-1128330706-25p37
Mon, 07 Aug 2017 14:55:47 GMT ragent info EXTRACT, results: , attributes: name,identity,type,prefix,distribution,waypoint,ingressPhase,egressPhase
Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved addresses for Router.qdrouterd-1128330706-25p37: {}
Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved link routes for Router.qdrouterd-1128330706-25p37: raw-> [{"name":"override.lwt_in","identity":"1","type":"org.apache.qpid.dispatch.router.config.linkRoute","prefix":"$lwt","distribution":"linkBalanced","connection":null,"containerId":"lwt-service","dir":"in","operStatus":"active"},{"name":"override.lwt_out","identity":"2","type":"org.apache.qpid.dispatch.router.config.linkRoute","prefix":"$lwt","distribution":"linkBalanced","connection":null,"containerId":"lwt-service","dir":"out","operStatus":"active"},{"name":"override.locate","identity":"3","type":"org.apache.qpid.dispatch.router.config.linkRoute","prefix":"locate","distribution":"linkBalanced","connection":"subscription-service","containerId":null,"dir":"out","operStatus":"active"}]
Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved link routes for Router.qdrouterd-1128330706-25p37: {}
Mon, 07 Aug 2017 14:55:47 GMT ragent info updating addresses for Router.qdrouterd-1128330706-25p37
Mon, 07 Aug 2017 14:55:47 GMT ragent info defining address telemetry/DEFAULT_TENANT on router Router.qdrouterd-1128330706-25p37
Mon, 07 Aug 2017 14:55:47 GMT ragent info defining in link route event/DEFAULT_TENANT on router Router.qdrouterd-1128330706-25p37
Mon, 07 Aug 2017 14:55:47 GMT ragent info defining out link route event/DEFAULT_TENANT on router Router.qdrouterd-1128330706-25p37
Mon, 07 Aug 2017 14:55:47 GMT ragent info addresses updated for Router.qdrouterd-1128330706-25p37
Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved listeners for Router.qdrouterd-1128330706-25p37: 172.17.0.10:55672
Mon, 07 Aug 2017 14:55:47 GMT ragent info Router.qdrouterd-1128330706-25p37 not ready for connectivity check: false false
Mon, 07 Aug 2017 14:55:47 GMT ragent info EXTRACT, results:
[..]
Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved addresses for Router.qdrouterd-1128330706-25p37:
[..]
Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved link routes for Router.qdrouterd-1128330706-25p37: raw-> [..]
Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved link routes for Router.qdrouterd-1128330706-25p37: {"event/DEFAULT_TENANT_in":{"name":"event/
[..]
Mon, 07 Aug 2017 14:55:47 GMT ragent info updating addresses for Router.qdrouterd-1128330706-25p37
Mon, 07 Aug 2017 14:55:47 GMT ragent info Router.qdrouterd-1128330706-25p37 not ready for connectivity check: true false
Mon, 07 Aug 2017 14:55:47 GMT ragent info retrieved connectors for Router.qdrouterd-1128330706-25p37:
Mon, 07 Aug 2017 14:55:47 GMT ragent info checking connectivity for Router.qdrouterd-1128330706-25p37
Mon, 07 Aug 2017 14:55:47 GMT ragent info checking connectors on router Router.qdrouterd-1128330706-25p37, missing=, stale=
----
queue-scheduler:
----
2017-08-07 14:53:39 WARN MacAddressUtil:136 - Failed to find a usable hardware address from the network interfaces; using random bytes: e0:37:7a:1d:7d:a2:98:48
2017-08-07 14:53:40 INFO QueueScheduler:95 - QueueScheduler is up and running
2017-08-07 14:53:40 ERROR ConfigServiceClient:65 - Error connecting to configuration service
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5672
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:347)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:630)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:565)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:479)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at java.lang.Thread.run(Thread.java:748)
2017-08-07 14:53:50 INFO ConfigServiceClient:45 - Connected to the configuration service
----
Any idea on what the reason could be or how to investigate further?
Best regards
Carsten Lohmann
(INST/ECS4)
Bosch Software Innovations GmbH | Schöneberger Ufer 89-91 | 10785 Berlin | GERMANY | www.bosch-si.com<http://www.bosch-si.com>
Tel. +49 30 726112-130 | Fax +49 30 726112-100 | carsten.lohmann at bosch-si.com<mailto:carsten.lohmann at bosch-si.com>
Sitz: Berlin, Registergericht: Amtsgericht Charlottenburg; HRB 148411 B
Aufsichtsratsvorsitzender: Dr.-Ing. Thorsten Lücke; Geschäftsführung: Dr.-Ing. Rainer Kallenbach, Michael Hahn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/enmasse/attachments/20170808/458d3df8/attachment.htm>
More information about the enmasse
mailing list