[EnMasse] Redeploying new certificates to Openshift results in Enmasse not being reachable anymore
Jens Reimann
jreimann at redhat.com
Wed Jul 10 09:54:35 UTC 2019
Hi Bob,
no, that should have the same effect. As the applications got restarted and
had to chance to reload their files.
One other thing that might be, you could manually delete the secrets
containing the inter service certificates. You still need to restart the
pods after that. If the service CA was recreated, but the certificates did
not get renewed, then this would be an issue as well. Showing the same
behavior as if the pods did not pick up the new certificates.
You should be able to find the corresponding secrets by looking for an
annotation of `service.alpha.openshift.io/originating-service-name`.
Cheers
Jens
On Wed, Jul 10, 2019 at 10:08 AM Bob Claerhout <bob.claerhout at aloxy.io>
wrote:
> Hi Jens,
>
> Thanks for your response.
> In an attempt to fix the issues, I rebooted the server yesterday which
> didn't work. So unless this is different from killing the pods, this isn't
> working.
>
> However, I scaled down the adapters and back up and the AMQP adapter (from
> Hono) seems to be able to connect to Enmasse although not passing the
> readiness check:
>
> 07:16:08.983 [vert.x-eventloop-thread-0] INFO
> o.e.h.a.a.i.VertxBasedAmqpProtocolAdapter - secure AMQP server listening on
> 0.0.0.0:5671
> 07:16:08.983 [vert.x-eventloop-thread-0] INFO
> o.e.h.a.a.i.VertxBasedAmqpProtocolAdapter - insecure AMQP server listening
> on [0.0.0.0:5672]
> 07:16:08.993 [vert.x-eventloop-thread-1] INFO
> o.e.h.s.VertxBasedHealthCheckServer - registering additional resource:
> Prometheus Registry Scraper [endpoint: /prometheus]
> 07:16:09.001 [vert.x-eventloop-thread-1] INFO
> o.e.h.s.VertxBasedHealthCheckServer - readiness probe available at
> http://0.0.0.0:8088/readiness
> 07:16:09.001 [vert.x-eventloop-thread-1] INFO
> o.e.h.s.VertxBasedHealthCheckServer - liveness probe available at
> http://0.0.0.0:8088/liveness
> 07:16:09.004 [main] INFO o.e.h.adapter.amqp.impl.Application - Started
> Application in 2.914 seconds (JVM running for 3.44)
> 07:16:09.355 [vert.x-eventloop-thread-0] INFO
> o.e.h.a.a.i.VertxBasedAmqpProtocolAdapter - connected to AMQP Messaging
> Network
>
> Best regards,
> Bob
>
> On 7/10/19 9:08 AM, Jens Reimann wrote:
>
> Hi Bob,
>
> I think (and this is just a guess) that redeploying the certificates, also
> redeploys the internal service CA. So that would mean that both your TLS
> server and clients need to pick up those new keys/certs/CAbundle from the
> file system. OKD injects those into the file system during runtime.
> However, most Java applications (and I think Hono is not different here)
> read in the cert material on startup, and then stick to that.
>
> So it might help to simply kill all the pods (clients + servers) and let
> them come up again.
>
> If that fixes the your situation, then I think that this is indeed the
> issue. And we should talk about an alternate way to handle it.
>
> Otherwise we would need to keep looking for the cause.
>
> Cheers
>
> Jens
>
>
> On Tue, Jul 9, 2019 at 5:31 PM Bob Claerhout <bob.claerhout at aloxy.io>
> wrote:
>
>> Hi all,
>>
>> Today I have redeployed new certificates on our OKD cluster.
>> I've done this by retrieving new certifcates with certbot. The
>> certificates were renewed perfectly after which I redeployed the
>> certificates using following ansible-playbook command:
>> ansible-playbook -i inventory.ini playbooks/redeploy-certificates.yml
>>
>> This succeeds perfectly apart from redeploying the certificates to the
>> service_catalog because (apparently) I do not have a kube-service-catalog
>> (I'm just giving all information I have to present a complete story). When
>> I disable updating the certificates of this service-catalog, the script
>> succeeds. After that, my frontend application and the openshift console are
>> using the new certificate.
>>
>> However, Eclipse hono is no longer able to connect to Enmasse.
>>
>> The hono adapters are getting following exceptions:
>> 14:49:14.276 [vert.x-eventloop-thread-0] DEBUG
>> o.e.h.client.impl.HonoConnectionImpl - connection attempt failed
>> javax.net.ssl.SSLHandshakeException: Failed to create SSL connection
>> at
>> io.vertx.core.net.impl.ChannelProvider$1.userEventTriggered(ChannelProvider.java:111)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)
>> at
>> io.netty.handler.ssl.SslHandler.handleUnwrapThrowable(SslHandler.java:1224)
>> at
>> io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1205)
>> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243)
>> at
>> io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
>> at
>> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
>> at
>> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
>> at
>> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
>> at
>> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
>> at
>> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
>> at
>> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
>> at
>> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
>> at
>> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
>> at
>> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
>> at
>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>> at java.base/java.lang.Thread.run(Thread.java:834)
>> Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building
>> failed: sun.security.provider.certpath.SunCertPathBuilderException: unable
>> to find valid certification path to requested target
>> at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
>> at
>> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
>> at
>> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264)
>> at
>> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:259)
>> at
>> java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:642)
>> at
>> java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:461)
>> at
>> java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:361)
>> at
>> java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
>> at
>> java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:448)
>> at
>> java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1065)
>> at
>> java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1052)
>> at java.base/java.security.AccessController.doPrivileged(Native
>> Method)
>> at
>> java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:999)
>> at
>> io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1457)
>> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1365)
>> at
>> io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199)
>> ... 19 common frames omitted
>> Caused by: sun.security.validator.ValidatorException: PKIX path building
>> failed: sun.security.provider.certpath.SunCertPathBuilderException: unable
>> to find valid certification path to requested target
>> at
>> java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
>> at
>> java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:290)
>> at
>> java.base/sun.security.validator.Validator.validate(Validator.java:264)
>> at
>> java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:321)
>> at
>> java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:279)
>> at
>> java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)
>> at
>> java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:620)
>> ... 30 common frames omitted
>> Caused by: sun.security.provider.certpath.SunCertPathBuilderException:
>> unable to find valid certification path to requested target
>> at
>> java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
>> at
>> java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
>> at
>> java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
>> at
>> java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:380)
>> ... 36 common frames omitted
>>
>> When I have a look at Enmasse logging, I'm getting following logging in
>> one of the qdrouters:
>>
>> 2019-07-09 14:52:07.293404 +0000 ROUTER (info) [C326] Connection Opened:
>> dir=in host=10.128.1.175:53936 vhost= encrypted=TLSv1.2 auth=EXTERNAL
>> user=CN=admin.aloxy,O=io.enmasse container_id=standard-controller
>> props={:product="vertx-proton", :version="3.6.3"}
>> 2019-07-09 14:52:07.293478 +0000 ROUTER (info) [C326][L536] Link
>> attached: dir=in source={<none> expire:sess outcomes:@PN_SYMBOL[:"amqp:accepted:list",
>> :"amqp:rejected:list", :"amqp:released:list", :"amqp:modified:list"]}
>> target={$management expire:sess}
>> 2019-07-09 14:52:07.293919 +0000 ROUTER (info) [C326][L537] Link
>> attached: dir=out source={<dynamic> expire:sess} target={<none> expire:sess}
>> 2019-07-09 14:52:07.299307 +0000 ROUTER (info) [C326][L536] Link closed
>> due to connection loss: del=4 presett=0 psdrop=0 acc=4 rej=0 rel=0 mod=0
>> delay1=0 delay10=0
>> 2019-07-09 14:52:07.299336 +0000 ROUTER (info) [C326][L537] Link closed
>> due to connection loss: del=4 presett=4 psdrop=0 acc=0 rej=0 rel=0 mod=0
>> delay1=0 delay10=0
>> 2019-07-09 14:52:07.299346 +0000 ROUTER (info) [C326] Connection Closed
>>
>>
>> Also I'm unable to browse the console:
>>
>> I've tested with all other routes as well, none of them are available:
>>
>>
>> I've tried to get rid of the custom certificates attached to the routes,
>> but this didn't work.
>>
>> In our installation we also have Eclipse ditto running. Ditto is also not
>> able to connect to Enmasse.
>>
>> Does anyone of you have a clue of what's going on? Have I missed
>> something?
>>
>> Kind regards,
>>
>> --
>>
>> *Bob Claerhout*
>> Software developer
>> M +32 479 34 84 92
>> The Beacon, Sint-Pietersvliet 7, 2000
>> Antwerp
>> bob.claerhout at aloxy.io | www.aloxy.io
>> _______________________________________________
>> enmasse mailing list
>> enmasse at redhat.com
>> https://www.redhat.com/mailman/listinfo/enmasse
>>
>
>
> --
> Jens Reimann
> Principal Software Engineer / EMEA ENG Middleware
> Werner-von-Siemens-Ring 14
> 85630 Grasbrunn
> Germany
> phone: +49 89 2050 71286
>
> _____________________________________________________________________________
>
> Red Hat GmbH, www.de.redhat.com,
> Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB
> 153243,
> Managing Directors: Paul Argiry, Charles Cachera, Tom Savage, Michael
> O'Neill
>
>
>
--
Jens Reimann
Principal Software Engineer / EMEA ENG Middleware
Werner-von-Siemens-Ring 14
85630 Grasbrunn
Germany
phone: +49 89 2050 71286
_____________________________________________________________________________
Red Hat GmbH, www.de.redhat.com,
Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB
153243,
Managing Directors: Paul Argiry, Charles Cachera, Tom Savage, Michael
O'Neill
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/enmasse/attachments/20190710/c680aa16/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: icmkeokoohinpphk.png
Type: image/png
Size: 42252 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/enmasse/attachments/20190710/c680aa16/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dngdinjkdhlmpihp.png
Type: image/png
Size: 49739 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/enmasse/attachments/20190710/c680aa16/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: koilllhhakloomkn.png
Type: image/png
Size: 6330 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/enmasse/attachments/20190710/c680aa16/attachment-0002.png>
More information about the enmasse
mailing list