Securing registry with Keycloak with untrusted certificate

[christofluethi]

Hi there

I'm currently using apicurio/apicurio-registry-sql:2.0.1.Final and try to secure the registry with keycloak.

As our keycloak does not have a trusted certificate we are running in the following stacktrace:

2021-07-21 07:07:04,618 ERROR [io.qua.ver.htt.run.QuarkusErrorHandler] (vert.x-eventloop-thread-0) HTTP Request to / failed, error id: 507116bd-2bbe-4bfc-9492-db15948896c9-2: java.lang.RuntimeException: Could not obtain configuration from server [https://<HOST>/auth/realms/<REALM>/.well-known/uma2-configuration].
        at org.keycloak.authorization.client.AuthzClient.<init>(AuthzClient.java:266)
        at org.keycloak.authorization.client.AuthzClient.create(AuthzClient.java:94)
        at io.apicurio.registry.services.auth.CustomAuthenticationMechanism$BearerTokenExtractor.<init>(CustomAuthenticationMechanism.java:126)
        at io.apicurio.registry.services.auth.CustomAuthenticationMechanism.authenticate(CustomAuthenticationMechanism.java:79)
        at io.apicurio.registry.services.auth.CustomAuthenticationMechanism_ClientProxy.authenticate(CustomAuthenticationMechanism_ClientProxy.zig:189)
        at io.quarkus.vertx.http.runtime.security.HttpAuthenticator.attemptAuthentication(HttpAuthenticator.java:100)
        at io.quarkus.vertx.http.runtime.security.HttpAuthenticator_ClientProxy.attemptAuthentication(HttpAuthenticator_ClientProxy.zig:157)
        at io.quarkus.vertx.http.runtime.security.HttpSecurityRecorder$2.handle(HttpSecurityRecorder.java:101)
        at io.quarkus.vertx.http.runtime.security.HttpSecurityRecorder$2.handle(HttpSecurityRecorder.java:51)
        at io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1038)
        at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:137)
        at io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:132)
        at io.quarkus.vertx.http.runtime.cors.CORSFilter.handle(CORSFilter.java:92)
        at io.quarkus.vertx.http.runtime.cors.CORSFilter.handle(CORSFilter.java:18)
        at io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1038)
        at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:137)
        at io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:132)
        at io.vertx.ext.web.impl.RouterImpl.handle(RouterImpl.java:54)
        at io.vertx.ext.web.impl.RouterImpl.handle(RouterImpl.java:36)
        at io.quarkus.vertx.http.runtime.VertxHttpRecorder$9.handle(VertxHttpRecorder.java:424)
        at io.quarkus.vertx.http.runtime.VertxHttpRecorder$9.handle(VertxHttpRecorder.java:421)
        at io.quarkus.vertx.http.runtime.VertxHttpRecorder$1.handle(VertxHttpRecorder.java:147)
        at io.quarkus.vertx.http.runtime.VertxHttpRecorder$1.handle(VertxHttpRecorder.java:129)
        at io.vertx.core.http.impl.WebSocketRequestHandler.handle(WebSocketRequestHandler.java:50)
        at io.vertx.core.http.impl.WebSocketRequestHandler.handle(WebSocketRequestHandler.java:32)
        at io.vertx.core.http.impl.Http1xServerConnection.handleMessage(Http1xServerConnection.java:136)
        at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
        at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:43)
        at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:229)
        at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:163)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
        at io.netty.handler.codec.http.websocketx.extensions.WebSocketServerExtensionHandler.channelRead(WebSocketServerExtensionHandler.java:101)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.vertx.core.http.impl.Http1xUpgradeToH2CHandler.channelRead(Http1xUpgradeToH2CHandler.java:109)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.vertx.core.http.impl.Http1xOrH2CHandler.end(Http1xOrH2CHandler.java:61)
        at io.vertx.core.http.impl.Http1xOrH2CHandler.channelRead(Http1xOrH2CHandler.java:38)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.RuntimeException: Error executing http method [GET]. Response : null
        at org.keycloak.authorization.client.util.HttpMethod.execute(HttpMethod.java:106)
        at org.keycloak.authorization.client.util.HttpMethodResponse$2.execute(HttpMethodResponse.java:50)
        at org.keycloak.authorization.client.AuthzClient.<init>(AuthzClient.java:264)
        ... 72 more
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:349)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:287)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
        at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:182)
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:171)
        at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1418)
        at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1324)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:411)
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436)
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.keycloak.authorization.client.util.HttpMethod.execute(HttpMethod.java:84)
        ... 74 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
        at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
        at java.base/sun.security.validator.Validator.validate(Validator.java:264)
        at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:222)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:638)
        ... 99 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
        at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
        ... 105 more

Used Environment Variables:

	env:
	- name: REGISTRY_DATASOURCE_URL
	  value: jdbc:postgresql://postgresql:5432/apicurio
	- name: REGISTRY_DATASOURCE_USERNAME
	  value: <USERNAME>
	- name: REGISTRY_DATASOURCE_PASSWORD
	  value: <PASSWORD>
	- name: LOG_LEVEL
	  value: INFO
	- name: QUARKUS_PROFILE
	  value: prod
	- name: AUTH_ENABLED
	  value: "true"
	- name: ROLES_ENABLED
	  value: "true"
	- name: KEYCLOAK_URL
	  value: https://<HOST>/auth
	- name: KEYCLOAK_REALM
	  value: <REALM>
	- name: KEYCLOAK_API_CLIENT_ID
	  value: dev-apicurio-registry-api
	- name: KEYCLOAK_UI_CLIENT_ID
	  value: dev-apicurio-registry

I tried the following without any luck:

• Mounting a configmap with quarkus.oidc.tls.verification=none (mountpath: /deployments/config/application.properties)
• Using a truststore and specifiying it with quarkus.http.ssl.certificate.trust-store-file=/deployments/tls/ca/truststore.jks
• Specifying the truststore as System property -Djavax.net.ssl.trustStore

Has anyone managed to get a running setup with an untrusted keycloak certificate?

Another question related to securing the registry with keycloak. Is it possible to change the role names by overriding registry.auth.roles.admin=some-other-admin-role?

[carlesarnal]

Hi, Quarkus properties will not work since this is something that is happening in a custom controller. You need to import the certificate into the keystore.

If you're running in docker, this is something you can done with something like this:

USER root
COPY servercert.cer $JAVA_HOME/jre/lib/security
RUN \
    cd $JAVA_HOME/jre/lib/security \
    && keytool -keystore cacerts -storepass yourstorepass -noprompt -trustcacerts -importcert -alias certificatealias -file servercert.cer

[carlesarnal]

Another question related to securing the registry with keycloak. Is it possible to change the role names by overriding registry.auth.roles.admin=some-other-admin-role?

If you're using the application without the UI yes, you can customize the roles. If you're using the UI then no, roles are fixed in the UI. We know this is a problem and is something that will be changed in future versions.

[christofluethi]

Hi, Quarkus properties will not work since this is something that is happening in a custom controller. You need to import the certificate into the keystore.

If you're running in docker, this is something you can done with something like this:

USER root COPY servercert.cer $JAVA_HOME/jre/lib/security RUN \ cd $JAVA_HOME/jre/lib/security \ && keytool -keystore cacerts -storepass yourstorepass -noprompt -trustcacerts -importcert -alias certificatealias -file servercert.cer

Since the certs are provided by the runtime-platform and not at buildtime I added our tuststore.pem to $JAVA_HOME/lib/security/cacerts in an initContainer (assuming thats the correct path as $JAVA_HOME/jre does not exist in apicurio-registry-sql:2.0.1.Final). However it fails with the same error.

Using curl --cacert truststore.pem https://... works fine and keytool -v -list -cacerts lists the added custom certs. So certificates seem to be the right ones.

[EricWittmann]

The bug where the roles are not customizable is fixed and will be available in a 2.1.x release. Maybe we need to backport the fix and release 2.0.2.Final if that bug fix is needed sooner.

[EricWittmann]

As for the root problem, I would have expected the -Djavax.net.ssl.trustStore approach to work. Are you 100% sure the truststore is mounted and the system property is set?

We would probably need to reproduce this locally to dig into it any further, wdyt @carlesarnal ?

[christofluethi]

The bug where the roles are not customizable is fixed and will be available in a 2.1.x release. Maybe we need to backport the fix and release 2.0.2.Final if that bug fix is needed sooner.

thats good news. is there a release date for the 2.1.x release? For us the renaming of roles is required. Depending on the release date a backport to 2.0.2.Final would be appreciated.

i'll recheck the certificate issue as i would also expect the -Djavax.net.ssl.trustStore to work. will let you know.

[christofluethi]

I was able to sort out the certificate issue. root cause was that I directly added the pem to the cacerts which did not add it in a correct way. using keytool -importkeystore using a full keystore did the trick.

a backport of the role-name changing feature to 2.0.2 would still be highly appreciated.

[EricWittmann]

Release date for 2.1.0 is "soon". :) I'm out on PTO for 2 weeks starting next week - we probably won't get a release done before that, and not sure if we would do one while I'm out. Maybe.