[EnMasse] Redeploying new certificates to Openshift results in Enmasse not being reachable anymore
Bob Claerhout
bob.claerhout at aloxy.io
Wed Jul 10 07:33:53 UTC 2019
Hi Jens,
Thanks for your response.
In an attempt to fix the issues, I rebooted the server yesterday which didn't work. So unless this is different from killing the pods, this isn't working.
However, I scaled down the adapters and back up and the AMQP adapter (from Hono) seems to be able to connect to Enmasse although not passing the readiness check:
07:16:08.983 [vert.x-eventloop-thread-0] INFO o.e.h.a.a.i.VertxBasedAmqpProtocolAdapter - secure AMQP server listening on 0.0.0.0:5671
07:16:08.983 [vert.x-eventloop-thread-0] INFO o.e.h.a.a.i.VertxBasedAmqpProtocolAdapter - insecure AMQP server listening on [0.0.0.0:5672]
07:16:08.993 [vert.x-eventloop-thread-1] INFO o.e.h.s.VertxBasedHealthCheckServer - registering additional resource: Prometheus Registry Scraper [endpoint: /prometheus]
07:16:09.001 [vert.x-eventloop-thread-1] INFO o.e.h.s.VertxBasedHealthCheckServer - readiness probe available at http://0.0.0.0:8088/readiness
07:16:09.001 [vert.x-eventloop-thread-1] INFO o.e.h.s.VertxBasedHealthCheckServer - liveness probe available at http://0.0.0.0:8088/liveness
07:16:09.004 [main] INFO o.e.h.adapter.amqp.impl.Application - Started Application in 2.914 seconds (JVM running for 3.44)
07:16:09.355 [vert.x-eventloop-thread-0] INFO o.e.h.a.a.i.VertxBasedAmqpProtocolAdapter - connected to AMQP Messaging Network
Best regards,
Bob
On 7/10/19 9:08 AM, Jens Reimann wrote:
Hi Bob,
I think (and this is just a guess) that redeploying the certificates, also redeploys the internal service CA. So that would mean that both your TLS server and clients need to pick up those new keys/certs/CAbundle from the file system. OKD injects those into the file system during runtime. However, most Java applications (and I think Hono is not different here) read in the cert material on startup, and then stick to that.
So it might help to simply kill all the pods (clients + servers) and let them come up again.
If that fixes the your situation, then I think that this is indeed the issue. And we should talk about an alternate way to handle it.
Otherwise we would need to keep looking for the cause.
Cheers
Jens
On Tue, Jul 9, 2019 at 5:31 PM Bob Claerhout <bob.claerhout at aloxy.io<mailto:bob.claerhout at aloxy.io>> wrote:
Hi all,
Today I have redeployed new certificates on our OKD cluster.
I've done this by retrieving new certifcates with certbot. The certificates were renewed perfectly after which I redeployed the certificates using following ansible-playbook command:
ansible-playbook -i inventory.ini playbooks/redeploy-certificates.yml
This succeeds perfectly apart from redeploying the certificates to the service_catalog because (apparently) I do not have a kube-service-catalog (I'm just giving all information I have to present a complete story). When I disable updating the certificates of this service-catalog, the script succeeds. After that, my frontend application and the openshift console are using the new certificate.
However, Eclipse hono is no longer able to connect to Enmasse.
The hono adapters are getting following exceptions:
14:49:14.276 [vert.x-eventloop-thread-0] DEBUG o.e.h.client.impl.HonoConnectionImpl - connection attempt failed
javax.net.ssl.SSLHandshakeException: Failed to create SSL connection
at io.vertx.core.net.impl.ChannelProvider$1.userEventTriggered(ChannelProvider.java:111)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)
at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)
at io.netty.handler.ssl.SslHandler.handleUnwrapThrowable(SslHandler.java:1224)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1205)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:259)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:642)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:461)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:361)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:448)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1065)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1052)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:999)
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1457)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1365)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199)
... 19 common frames omitted
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:290)
at java.base/sun.security.validator.Validator.validate(Validator.java:264)
at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:321)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:279)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:620)
... 30 common frames omitted
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:380)
... 36 common frames omitted
When I have a look at Enmasse logging, I'm getting following logging in one of the qdrouters:
2019-07-09 14:52:07.293404 +0000 ROUTER (info) [C326] Connection Opened: dir=in host=10.128.1.175:53936<http://10.128.1.175:53936> vhost= encrypted=TLSv1.2 auth=EXTERNAL user=CN=admin.aloxy,O=io.enmasse container_id=standard-controller props={:product="vertx-proton", :version="3.6.3"}
2019-07-09 14:52:07.293478 +0000 ROUTER (info) [C326][L536] Link attached: dir=in source={<none> expire:sess outcomes:@PN_SYMBOL[:"amqp:accepted:list", :"amqp:rejected:list", :"amqp:released:list", :"amqp:modified:list"]} target={$management expire:sess}
2019-07-09 14:52:07.293919 +0000 ROUTER (info) [C326][L537] Link attached: dir=out source={<dynamic> expire:sess} target={<none> expire:sess}
2019-07-09 14:52:07.299307 +0000 ROUTER (info) [C326][L536] Link closed due to connection loss: del=4 presett=0 psdrop=0 acc=4 rej=0 rel=0 mod=0 delay1=0 delay10=0
2019-07-09 14:52:07.299336 +0000 ROUTER (info) [C326][L537] Link closed due to connection loss: del=4 presett=4 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0
2019-07-09 14:52:07.299346 +0000 ROUTER (info) [C326] Connection Closed
Also I'm unable to browse the console:
[cid:part3.427DA5AA.F998ADF6 at aloxy.io]
I've tested with all other routes as well, none of them are available:
[cid:part4.A2E9A208.1FDC66FC at aloxy.io]
I've tried to get rid of the custom certificates attached to the routes, but this didn't work.
In our installation we also have Eclipse ditto running. Ditto is also not able to connect to Enmasse.
Does anyone of you have a clue of what's going on? Have I missed something?
Kind regards,
--
[cid:part5.7F62261F.561E432E at aloxy.io] Bob Claerhout
Software developer
M +32 479 34 84 92
The Beacon, Sint-Pietersvliet 7, 2000
Antwerp
bob.claerhout at aloxy.io<mailto:bob.claerhout at aloxy.io> | www.aloxy.io<http://www.aloxy.io>
_______________________________________________
enmasse mailing list
enmasse at redhat.com<mailto:enmasse at redhat.com>
https://www.redhat.com/mailman/listinfo/enmasse
--
Jens Reimann
Principal Software Engineer / EMEA ENG Middleware
Werner-von-Siemens-Ring 14
85630 Grasbrunn
Germany
phone: +49 89 2050 71286
_____________________________________________________________________________
Red Hat GmbH, www.de.redhat.com<http://www.de.redhat.com>,
Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Paul Argiry, Charles Cachera, Tom Savage, Michael O'Neill
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/enmasse/attachments/20190710/4d6987dd/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: icmkeokoohinpphk.png
Type: image/png
Size: 42252 bytes
Desc: icmkeokoohinpphk.png
URL: <http://listman.redhat.com/archives/enmasse/attachments/20190710/4d6987dd/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dngdinjkdhlmpihp.png
Type: image/png
Size: 49739 bytes
Desc: dngdinjkdhlmpihp.png
URL: <http://listman.redhat.com/archives/enmasse/attachments/20190710/4d6987dd/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: koilllhhakloomkn.png
Type: image/png
Size: 6330 bytes
Desc: koilllhhakloomkn.png
URL: <http://listman.redhat.com/archives/enmasse/attachments/20190710/4d6987dd/attachment-0002.png>
More information about the enmasse
mailing list