[Strimzi] Cluster-Operator 0.9.0 does not start due to error 404 on websocket connection
Schwarz, Markus
M.Schwarz at prosoz.de
Mon Jan 7 14:15:21 UTC 2019
Hi Jakub,
here is the complete log from the co:
+ JAR=/cluster-operator-0.9.0.jar
+ shift
+ . /bin/dynamic_resources.sh
++ get_heap_size
+++ cat /sys/fs/cgroup/memory/memory.limit_in_bytes
++ CONTAINER_MEMORY_IN_BYTES=268435456
++ DEFAULT_MEMORY_CEILING=1152921504606846975
++ '[' 268435456 -lt 1152921504606846975 ']'
++ '[' -z ']'
++ CONTAINER_HEAP_PERCENT=0.50
++ CONTAINER_MEMORY_IN_MB=256
+++ echo '256 0.50'
+++ awk '{ printf "%d", $1 * $2 }'
++ CONTAINER_HEAP_MAX=128
++ echo 128
+ MAX_HEAP=128
+ '[' -n 128 ']'
+ JAVA_OPTS='-Xms128m -Xmx128m '
+ export MALLOC_ARENA_MAX=2
+ MALLOC_ARENA_MAX=2
+ JAVA_OPTS='-Xms128m -Xmx128m -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom'
+ JAVA_OPTS='-Xms128m -Xmx128m -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom -XX:NativeMemoryTracking=summary -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps'
+ exec java -Xms128m -Xmx128m -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom -XX:NativeMemoryTracking=summary -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -jar /cluster-operator-0.9.0.jar -Xms128m -Xmx128m -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom -XX:NativeMemoryTracking=summary -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps
2019-01-07 14:02:15 INFO Main:70 - ClusterOperator 0.9.0 is starting
2019-01-07T14:02:16.434+0000: [GC (Allocation Failure) 2019-01-07T14:02:16.434+0000: [DefNew: 34944K->4352K(39296K), 0.0154766 secs] 34944K->8729K(126720K), 0.0155884 secs] [Times: user=0.01 sys=0.01, real=0.01 secs]
2019-01-07T14:02:17.040+0000: [Full GC (Metadata GC Threshold) 2019-01-07T14:02:17.040+0000: [Tenured: 4377K->7631K(87424K), 0.0301695 secs] 24907K->7631K(126720K), [Metaspace: 20706K->20706K(1069056K)], 0.0303281 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
2019-01-07 14:02:17 INFO Main:262 - Using config:
PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
KUBERNETES_PORT_53_UDP_PROTO: udp
STRIMZI_DEFAULT_KAFKA_MIRRORMAKER_IMAGE: strimzi/kafka-mirror-maker:0.9.0
PROMETHEUS_PORT_9090_TCP: tcp://172.30.38.237:9090
STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 120000
STRIMZI_DEFAULT_KAFKA_CONNECT_IMAGE: strimzi/kafka-connect:0.9.0
KUBERNETES_PORT_443_TCP_PROTO: tcp
KUBERNETES_PORT_53_TCP_PORT: 53
PROMETHEUS_PORT_9090_TCP_ADDR: 172.30.38.237
KAFKA_METRICS_PORT_9404_TCP_PROTO: tcp
STRIMZI_VERSION: 0.9.0
PROMETHEUS_PORT_9090_TCP_PROTO: tcp
KUBERNETES_PORT_53_TCP_PROTO: tcp
KUBERNETES_PORT_53_TCP_ADDR: 172.30.0.1
HOSTNAME: strimzi-cluster-operator-5d5445c55d-9lb8v
HOME: /
MALLOC_ARENA_MAX: 2
STRIMZI_DEFAULT_ZOOKEEPER_IMAGE: strimzi/zookeeper:0.9.0
STRIMZI_NAMESPACE: msw
KUBERNETES_SERVICE_PORT_HTTPS: 443
SHLVL: 1
JAVA_HOME: /usr/lib/jvm/java
KAFKA_METRICS_PORT_9404_TCP_ADDR: 172.30.193.60
STRIMZI_DEFAULT_KAFKA_INIT_IMAGE: strimzi/kafka-init:0.9.0
KAFKA_METRICS_SERVICE_HOST: 172.30.193.60
KUBERNETES_PORT_443_TCP: tcp://172.30.0.1:443
PROMETHEUS_PORT_9090_TCP_PORT: 9090
STRIMZI_OPERATION_TIMEOUT_MS: 300000
PROMETHEUS_SERVICE_PORT: 9090
KAFKA_METRICS_SERVICE_PORT_METRICS: 9404
STRIMZI_DEFAULT_KAFKA_IMAGE: strimzi/kafka:0.9.0
STRIMZI_DEFAULT_TLS_SIDECAR_KAFKA_IMAGE: strimzi/kafka-stunnel:0.9.0
PROMETHEUS_PORT: tcp://172.30.38.237:9090
STRIMZI_LOG_LEVEL: INFO
KUBERNETES_PORT: tcp://172.30.0.1:443
PROMETHEUS_SERVICE_HOST: 172.30.38.237
STRIMZI_DEFAULT_KAFKA_CONNECT_S2I_IMAGE: strimzi/kafka-connect-s2i:0.9.0
KAFKA_METRICS_PORT: tcp://172.30.193.60:9404
KUBERNETES_PORT_53_TCP: tcp://172.30.0.1:53
KUBERNETES_PORT_53_UDP: udp://172.30.0.1:53
KUBERNETES_SERVICE_PORT: 443
KAFKA_METRICS_SERVICE_PORT: 9404
KUBERNETES_PORT_53_UDP_ADDR: 172.30.0.1
STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE: strimzi/topic-operator:0.9.0
PWD: /
PROMETHEUS_SERVICE_PORT_PROMETHEUS: 9090
KUBERNETES_PORT_443_TCP_ADDR: 172.30.0.1
STRIMZI_DEFAULT_USER_OPERATOR_IMAGE: strimzi/user-operator:0.9.0
KUBERNETES_SERVICE_PORT_DNS_TCP: 53
STRIMZI_DEFAULT_TLS_SIDECAR_ENTITY_OPERATOR_IMAGE: strimzi/entity-operator-stunnel:0.9.0
KUBERNETES_PORT_53_UDP_PORT: 53
KAFKA_METRICS_PORT_9404_TCP: tcp://172.30.193.60:9404
KUBERNETES_SERVICE_HOST: 172.30.0.1
KUBERNETES_SERVICE_PORT_DNS: 53
KUBERNETES_PORT_443_TCP_PORT: 443
STRIMZI_DEFAULT_TLS_SIDECAR_ZOOKEEPER_IMAGE: strimzi/zookeeper-stunnel:0.9.0
2019-01-07 14:02:17 INFO ClusterOperator:58 - Creating ClusterOperator for namespace msw
2019-01-07 14:02:17 INFO ClusterOperator:86 - Starting ClusterOperator for namespace msw
2019-01-07 14:02:17 INFO ClusterOperator:93 - Started operator for Kafka kind
2019-01-07 14:02:17 WARN WatchConnectionManager:185 - Exec Failure: HTTP 404, Status: 404 - 404 page not found
java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found'
at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:219) [cluster-operator-0.9.0.jar:0.9.0]
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:186) [cluster-operator-0.9.0.jar:0.9.0]
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153) [cluster-operator-0.9.0.jar:0.9.0]
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [cluster-operator-0.9.0.jar:0.9.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
2019-01-07 14:02:17 INFO WatchConnectionManager:379 - Current reconnect backoff is 1000 milliseconds (T0)
2019-01-07 14:02:17 ERROR Main:141 - Cluster Operator verticle in namespace msw failed to start
io.fabric8.kubernetes.client.KubernetesClientException: 404 page not found
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$2.onFailure(WatchConnectionManager.java:189) ~[cluster-operator-0.9.0.jar:0.9.0]
at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:546) ~[cluster-operator-0.9.0.jar:0.9.0]
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:188) ~[cluster-operator-0.9.0.jar:0.9.0]
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153) ~[cluster-operator-0.9.0.jar:0.9.0]
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) ~[cluster-operator-0.9.0.jar:0.9.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
Heap
def new generation total 39296K, used 16240K [0x00000000f8000000, 0x00000000faaa0000, 0x00000000faaa0000)
eden space 34944K, 46% used [0x00000000f8000000, 0x00000000f8fdc0d8, 0x00000000fa220000)
from space 4352K, 0% used [0x00000000fa660000, 0x00000000fa660000, 0x00000000faaa0000)
to space 4352K, 0% used [0x00000000fa220000, 0x00000000fa220000, 0x00000000fa660000)
tenured generation total 87424K, used 7631K [0x00000000faaa0000, 0x0000000100000000, 0x0000000100000000)
the space 87424K, 8% used [0x00000000faaa0000, 0x00000000fb213ed8, 0x00000000fb214000, 0x0000000100000000)
Metaspace used 23303K, capacity 23692K, committed 24064K, reserved 1071104K
class space used 2656K, capacity 2770K, committed 2816K, reserved 1048576K
Thanks & Regards
Markus
Von: Jakub Scholz [mailto:jakub at scholz.cz]
Gesendet: Montag, 7. Januar 2019 15:04
An: Schwarz, Markus <M.Schwarz at prosoz.de>
Cc: strimzi at redhat.com
Betreff: Re: [Strimzi] Cluster-Operator 0.9.0 does not start due to error 404 on websocket connection
Hi Markus,
0.9.0 should work fine with with Kubernetes 1.9 / OpenShift 3.9. Could you share the complete log from the CO? The Kubernetes client normally takes the address of the Kubernetes APi from the Kubernetes environment variables and connects there. Maybe in your case there is something strange / wrong with your cluster configuration.
Thanks & Regards
Jakub
On Mon, Jan 7, 2019 at 2:40 PM Schwarz, Markus <M.Schwarz at prosoz.de<mailto:M.Schwarz at prosoz.de>> wrote:
Hi,
We are currently running strimzi 0.4.0 ( I know, it’s old) on our Openshift Origin 3.9 cluster and everything is working about fine.
I know try to update to 0.9.0 to catch up to things and implement some security. So I took all the yaml-files from the cluster-operator install folder, made the necessary namespace amendments and gave it a try. The strimzi-cluster-operator pod tries to start but then dies with the following error message:
---
2019-01-07 12:49:29 WARN WatchConnectionManager:185 - Exec Failure: HTTP 404, Status: 404 - 404 page not found
java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found'
at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:219) [cluster-operator-0.9.0.jar:0.9.0]
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:186) [cluster-operator-0.9.0.jar:0.9.0]
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153) [cluster-operator-0.9.0.jar:0.9.0]
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [cluster-operator-0.9.0.jar:0.9.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
2019-01-07 12:49:29 INFO WatchConnectionManager:379 - Current reconnect backoff is 1000 milliseconds (T0)
2019-01-07 12:49:29 ERROR Main:141 - Cluster Operator verticle in namespace msw failed to start
io.fabric8.kubernetes.client.KubernetesClientException: 404 page not found
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$2.onFailure(WatchConnectionManager.java:189) ~[cluster-operator-0.9.0.jar:0.9.0]
at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:546) ~[cluster-operator-0.9.0.jar:0.9.0]
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:188) ~[cluster-operator-0.9.0.jar:0.9.0]
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153) ~[cluster-operator-0.9.0.jar:0.9.0]
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) ~[cluster-operator-0.9.0.jar:0.9.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
---
It seems to try to open a websocket connection on a url which does not exists. I don’t know if this is connected to the outdated version of kubernetes (1.9.1) or if this might be a configuration error of some sort, any hint will be appreciated. I could not find any kubernetes/openshift version requirements for strimzi.
Thanks!
Markus
[cid:image001.jpg at 01D4A69A.F3D64FB0] <http://www.prosoz.de>
prosoz-herten-footer
_______________________________________________
Strimzi mailing list
Strimzi at redhat.com<mailto:Strimzi at redhat.com>
https://www.redhat.com/mailman/listinfo/strimzi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/strimzi/attachments/20190107/95344595/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 71629 bytes
Desc: image001.jpg
URL: <http://listman.redhat.com/archives/strimzi/attachments/20190107/95344595/attachment.jpg>
More information about the Strimzi
mailing list