[EnMasse] Redeploying new certificates to Openshift results in Enmasse not being reachable anymore
Jens Reimann
jreimann at redhat.com
Wed Jul 10 07:08:14 UTC 2019
Hi Bob,
I think (and this is just a guess) that redeploying the certificates, also
redeploys the internal service CA. So that would mean that both your TLS
server and clients need to pick up those new keys/certs/CAbundle from the
file system. OKD injects those into the file system during runtime.
However, most Java applications (and I think Hono is not different here)
read in the cert material on startup, and then stick to that.
So it might help to simply kill all the pods (clients + servers) and let
them come up again.
If that fixes the your situation, then I think that this is indeed the
issue. And we should talk about an alternate way to handle it.
Otherwise we would need to keep looking for the cause.
Cheers
Jens
On Tue, Jul 9, 2019 at 5:31 PM Bob Claerhout <bob.claerhout at aloxy.io> wrote:
> Hi all,
>
> Today I have redeployed new certificates on our OKD cluster.
> I've done this by retrieving new certifcates with certbot. The
> certificates were renewed perfectly after which I redeployed the
> certificates using following ansible-playbook command:
> ansible-playbook -i inventory.ini playbooks/redeploy-certificates.yml
>
> This succeeds perfectly apart from redeploying the certificates to the
> service_catalog because (apparently) I do not have a kube-service-catalog
> (I'm just giving all information I have to present a complete story). When
> I disable updating the certificates of this service-catalog, the script
> succeeds. After that, my frontend application and the openshift console are
> using the new certificate.
>
> However, Eclipse hono is no longer able to connect to Enmasse.
>
> The hono adapters are getting following exceptions:
> 14:49:14.276 [vert.x-eventloop-thread-0] DEBUG
> o.e.h.client.impl.HonoConnectionImpl - connection attempt failed
> javax.net.ssl.SSLHandshakeException: Failed to create SSL connection
> at
> io.vertx.core.net.impl.ChannelProvider$1.userEventTriggered(ChannelProvider.java:111)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)
> at
> io.netty.handler.ssl.SslHandler.handleUnwrapThrowable(SslHandler.java:1224)
> at
> io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1205)
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243)
> at
> io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
> at
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed:
> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
> valid certification path to requested target
> at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
> at
> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
> at
> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264)
> at
> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:259)
> at
> java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:642)
> at
> java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:461)
> at
> java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:361)
> at
> java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
> at
> java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:448)
> at
> java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1065)
> at
> java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1052)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at
> java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:999)
> at
> io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1457)
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1365)
> at
> io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199)
> ... 19 common frames omitted
> Caused by: sun.security.validator.ValidatorException: PKIX path building
> failed: sun.security.provider.certpath.SunCertPathBuilderException: unable
> to find valid certification path to requested target
> at
> java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
> at
> java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:290)
> at
> java.base/sun.security.validator.Validator.validate(Validator.java:264)
> at
> java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:321)
> at
> java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:279)
> at
> java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)
> at
> java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:620)
> ... 30 common frames omitted
> Caused by: sun.security.provider.certpath.SunCertPathBuilderException:
> unable to find valid certification path to requested target
> at
> java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
> at
> java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
> at
> java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
> at
> java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:380)
> ... 36 common frames omitted
>
> When I have a look at Enmasse logging, I'm getting following logging in
> one of the qdrouters:
>
> 2019-07-09 14:52:07.293404 +0000 ROUTER (info) [C326] Connection Opened:
> dir=in host=10.128.1.175:53936 vhost= encrypted=TLSv1.2 auth=EXTERNAL
> user=CN=admin.aloxy,O=io.enmasse container_id=standard-controller
> props={:product="vertx-proton", :version="3.6.3"}
> 2019-07-09 14:52:07.293478 +0000 ROUTER (info) [C326][L536] Link attached:
> dir=in source={<none> expire:sess outcomes:@PN_SYMBOL[:"amqp:accepted:list",
> :"amqp:rejected:list", :"amqp:released:list", :"amqp:modified:list"]}
> target={$management expire:sess}
> 2019-07-09 14:52:07.293919 +0000 ROUTER (info) [C326][L537] Link attached:
> dir=out source={<dynamic> expire:sess} target={<none> expire:sess}
> 2019-07-09 14:52:07.299307 +0000 ROUTER (info) [C326][L536] Link closed
> due to connection loss: del=4 presett=0 psdrop=0 acc=4 rej=0 rel=0 mod=0
> delay1=0 delay10=0
> 2019-07-09 14:52:07.299336 +0000 ROUTER (info) [C326][L537] Link closed
> due to connection loss: del=4 presett=4 psdrop=0 acc=0 rej=0 rel=0 mod=0
> delay1=0 delay10=0
> 2019-07-09 14:52:07.299346 +0000 ROUTER (info) [C326] Connection Closed
>
>
> Also I'm unable to browse the console:
>
> I've tested with all other routes as well, none of them are available:
>
>
> I've tried to get rid of the custom certificates attached to the routes,
> but this didn't work.
>
> In our installation we also have Eclipse ditto running. Ditto is also not
> able to connect to Enmasse.
>
> Does anyone of you have a clue of what's going on? Have I missed something?
>
> Kind regards,
>
> --
>
> *Bob Claerhout*
> Software developer
> M +32 479 34 84 92
> The Beacon, Sint-Pietersvliet 7, 2000
> Antwerp
> bob.claerhout at aloxy.io | www.aloxy.io
> _______________________________________________
> enmasse mailing list
> enmasse at redhat.com
> https://www.redhat.com/mailman/listinfo/enmasse
>
--
Jens Reimann
Principal Software Engineer / EMEA ENG Middleware
Werner-von-Siemens-Ring 14
85630 Grasbrunn
Germany
phone: +49 89 2050 71286
_____________________________________________________________________________
Red Hat GmbH, www.de.redhat.com,
Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB
153243,
Managing Directors: Paul Argiry, Charles Cachera, Tom Savage, Michael
O'Neill
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/enmasse/attachments/20190710/73d649b8/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: icmkeokoohinpphk.png
Type: image/png
Size: 42252 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/enmasse/attachments/20190710/73d649b8/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dngdinjkdhlmpihp.png
Type: image/png
Size: 49739 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/enmasse/attachments/20190710/73d649b8/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: koilllhhakloomkn.png
Type: image/png
Size: 6330 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/enmasse/attachments/20190710/73d649b8/attachment-0002.png>
More information about the enmasse
mailing list