[EnMasse] Redeploying new certificates to Openshift results in Enmasse not being reachable anymore

Tue Jul 9 14:56:47 UTC 2019

Hi all,

Today I have redeployed new certificates on our OKD cluster.
I've done this by retrieving new certifcates with certbot. The certificates were renewed perfectly after which I redeployed the certificates using following ansible-playbook command:
     ansible-playbook -i inventory.ini playbooks/redeploy-certificates.yml

This succeeds perfectly apart from redeploying the certificates to the service_catalog because (apparently) I do not have a kube-service-catalog (I'm just giving all information I have to present a complete story). When I disable updating the certificates of this service-catalog, the script succeeds. After that, my frontend application and the openshift console are using the new certificate.

However, Eclipse hono is no longer able to connect to Enmasse.

The hono adapters are getting following exceptions:
14:49:14.276 [vert.x-eventloop-thread-0] DEBUG o.e.h.client.impl.HonoConnectionImpl - connection attempt failed
javax.net.ssl.SSLHandshakeException: Failed to create SSL connection
    at io.vertx.core.net.impl.ChannelProvider$1.userEventTriggered(ChannelProvider.java:111)
    at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)
    at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)
    at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)
    at io.netty.handler.ssl.SslHandler.handleUnwrapThrowable(SslHandler.java:1224)
    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1205)
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243)
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264)
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:259)
    at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:642)
    at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:461)
    at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:361)
    at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
    at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:448)
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1065)
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1052)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:999)
    at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1457)
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1365)
    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199)
    ... 19 common frames omitted
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
    at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:290)
    at java.base/sun.security.validator.Validator.validate(Validator.java:264)
    at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:321)
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:279)
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)
    at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:620)
    ... 30 common frames omitted
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
    at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
    at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
    at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:380)
    ... 36 common frames omitted

When I have a look at Enmasse logging, I'm getting following logging in one of the qdrouters:

2019-07-09 14:52:07.293404 +0000 ROUTER (info) [C326] Connection Opened: dir=in host=10.128.1.175:53936 vhost= encrypted=TLSv1.2 auth=EXTERNAL user=CN=admin.aloxy,O=io.enmasse container_id=standard-controller props={:product="vertx-proton", :version="3.6.3"}
2019-07-09 14:52:07.293478 +0000 ROUTER (info) [C326][L536] Link attached: dir=in source={<none> expire:sess outcomes:@PN_SYMBOL[:"amqp:accepted:list", :"amqp:rejected:list", :"amqp:released:list", :"amqp:modified:list"]} target={$management expire:sess}
2019-07-09 14:52:07.293919 +0000 ROUTER (info) [C326][L537] Link attached: dir=out source={<dynamic> expire:sess} target={<none> expire:sess}
2019-07-09 14:52:07.299307 +0000 ROUTER (info) [C326][L536] Link closed due to connection loss: del=4 presett=0 psdrop=0 acc=4 rej=0 rel=0 mod=0 delay1=0 delay10=0
2019-07-09 14:52:07.299336 +0000 ROUTER (info) [C326][L537] Link closed due to connection loss: del=4 presett=4 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0
2019-07-09 14:52:07.299346 +0000 ROUTER (info) [C326] Connection Closed

Also I'm unable to browse the console:
[cid:part1.B276F940.155F7047 at aloxy.io]
I've tested with all other routes as well, none of them are available:
[cid:part2.A4B51044.0C11036F at aloxy.io]

I've tried to get rid of the custom certificates attached to the routes, but this didn't work.

In our installation we also have Eclipse ditto running. Ditto is also not able to connect to Enmasse.

Does anyone of you have a clue of what's going on? Have I missed something?

Kind regards,

--

[cid:part3.123D80AC.53E5549A at aloxy.io]  Bob Claerhout
Software developer
M +32 479 34 84 92
The Beacon, Sint-Pietersvliet 7, 2000
Antwerp
bob.claerhout at aloxy.io<mailto:bob.claerhout at aloxy.io> | www.aloxy.io<http://www.aloxy.io>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/enmasse/attachments/20190709/4881b21b/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: icmkeokoohinpphk.png
Type: image/png
Size: 42252 bytes
Desc: icmkeokoohinpphk.png
URL: <http://listman.redhat.com/archives/enmasse/attachments/20190709/4881b21b/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dngdinjkdhlmpihp.png
Type: image/png
Size: 49739 bytes
Desc: dngdinjkdhlmpihp.png
URL: <http://listman.redhat.com/archives/enmasse/attachments/20190709/4881b21b/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: koilllhhakloomkn.png
Type: image/png
Size: 6330 bytes
Desc: koilllhhakloomkn.png
URL: <http://listman.redhat.com/archives/enmasse/attachments/20190709/4881b21b/attachment-0002.png>