IPv6 Service Address TLS Hostname Verification Fails

[lyind]

Kubernetes 1.19.1 (for example when setup using kubeadm) does not - by default - sign the IPv6 short-form of the api-server address.

Unfortunately kubernetes-client/java (version 10.0.0) uses the short-form string representation of the kubernetes service ClusterIp in the URL, which leads to the following exception:

2020-09-23 10:35:05.162 [] [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-0] ReflectorRunnable.defaultWatchErrorHandler() ERROR: class io.kubernetes.client.openapi.models.V1Pod#Reflector loop failed unexpectedly
io.kubernetes.client.openapi.ApiException: javax.net.ssl.SSLPeerUnverifiedException: Hostname 2a01:4f8:d0:1601::1 not verified:
    certificate: sha256/Xv8eDm3+ZqXDNTXy0friFPgGYo258qvYjPsc1XNSNbQ=
    DN: CN=kube-apiserver
    subjectAltNames: [2a01:4f8:d0:1601:0:0:0:1, 2a01:4f8:d0:1600:0:0:0:4, 2a01:4f8:d0:1600:ffff:ffff:fffe:0, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, node-d0-50-99-f1-8e-cb]
	at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:886)
	at io.kubernetes.client.informer.SharedInformerFactory$1.list(SharedInformerFactory.java:182)
	at io.kubernetes.client.informer.cache.ReflectorRunnable.run(ReflectorRunnable.java:74)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: javax.net.ssl.SSLPeerUnverifiedException: Hostname 2a01:4f8:d0:1601::1 not verified:
    certificate: sha256/Xv8eDm3+ZqXDNTXy0friFPgGYo258qvYjPsc1XNSNbQ=
    DN: CN=kube-apiserver
    subjectAltNames: [2a01:4f8:d0:1601:0:0:0:1, 2a01:4f8:d0:1600:0:0:0:4, 2a01:4f8:d0:1600:ffff:ffff:fffe:0, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, node-d0-50-99-f1-8e-cb]
	at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:350)
	at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:300)
	at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:185)
	at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224)
	at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108)
	at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88)
	at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
	at io.kubernetes.client.util.credentials.TokenFileAuthentication.intercept(TokenFileAuthentication.java:72)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229)
	at okhttp3.RealCall.execute(RealCall.java:81)
	at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:882)
	... 8 common frames omitted

Valid alternatives could be to either use the full (un-shortened) IPv6 address (2a01:4f8:d0:1601:0:0:0:1 in my case) or just use the default DNS name kubernetes. People looking to avoid the service name lookup, ie. for performance/reliability reasons, could instantiate the client separately.

This issue is somewhat related to the (fixed): #839

[brendandburns]

Thanks for reporting. We'd be happy to take a PR to address this.

[lyind]

OkHttps HttpUrl.canonicalizeHost() method always "canonicalizes" IPv6 addresses in request URLs into the compressed/short representation (ie. 0:0:0:0:0:0:0:1 -> ::1).

OkHostnameVerifier.verifyIpAddress() checks for matching SANs using String.equalsIgnoreCase().

Unfortunately IPv6 addresses in TLS certificate SAN fields are stored in uncompressed format (ie. 0:0:0:0:0:0:0:1), which is correct:

When the subjectAltName extension contains an iPAddress, the address
MUST be stored in the octet string in "network byte order", as
specified in [RFC791]. The least significant bit (LSB) of each octet
is the LSB of the corresponding byte in the network address. For IP
version 4, as specified in [RFC791], the octet string MUST contain
exactly four octets. For IP version 6, as specified in
[RFC2460], the octet string MUST contain exactly sixteen octets.

https://tools.ietf.org/html/rfc5280#section-4.2.1.6
Kudos to user @aojea for pointing it out in kubernetes/kubernetes#95007.

OkHostnameVerifier's deviation from the standard causes issues in the field.

Fortunately this is fixed starting from OkHttp 4.5.0. Issue leading to the fix: square/okhttp#5885

It seems possible to workaround this by overriding the OkHttp dependencies in build.gradle (or pom.xml):

    compile 'com.squareup.okhttp3:logging-interceptor:4.9.0'
    compile 'com.squareup.okhttp3:okhttp:4.9.0'

Please update the OkHttp library.

[brendandburns]

We'll look into updating. 3.x to 4.x is a big jump in versioning, so we'll need to validate that it works and/or do it across a major version update. (which fortunately we're about to do)

[lyind]

@brendanburns I updated to 4.9.0 for testing purposes. The method to make the kubernetes CA certificate known to the HTTP client doesn't work anymore. The CA certificate from the crafted TrustManager is not accepted and one gets a "No certification path" exception:

2020-09-25 12:18:42.087 [] [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-0] ReflectorRunnable.defaultWatchErrorHandler() ERROR: class io.kubernetes.client.openapi.models.V1Pod#Reflector loop failed unexpectedly
io.kubernetes.client.openapi.ApiException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at io.kubernetes.client.util.Watch.createWatch(Watch.java:109)
	at io.kubernetes.client.informer.SharedInformerFactory$1.watch(SharedInformerFactory.java:190)
	at io.kubernetes.client.informer.SharedInformerFactory$1.watch(SharedInformerFactory.java:178)
	at io.kubernetes.client.informer.cache.ReflectorRunnable.run(ReflectorRunnable.java:103)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
	at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
	at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
	at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(Unknown Source)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(Unknown Source)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(Unknown Source)
	at java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
	at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)
	at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
	at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source)
	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
	at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.kt:379)
	at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.kt:337)
	at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:209)
	at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226)
	at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106)
	at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74)
	at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
	at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
	at io.kubernetes.client.util.Watch.createWatch(Watch.java:93)
	... 9 common frames omitted
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.validator.PKIXValidator.doBuild(Unknown Source)
	at java.base/sun.security.validator.PKIXValidator.engineValidate(Unknown Source)
	at java.base/sun.security.validator.Validator.validate(Unknown Source)
	at java.base/sun.security.ssl.X509TrustManagerImpl.validate(Unknown Source)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)
	... 39 common frames omitted
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(Unknown Source)
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown Source)
	at java.base/java.security.cert.CertPathBuilder.build(Unknown Source)
	... 45 common frames omitted

[lyind]

After some debugging I found that my dependency injection framework injected a No-Arg constructor created ApiClient instance in my SharedInformerFactory instances.

Sorry for the inconvenience.

I wish the No-Arg constructors for ApiClient, CoreV1Api, SharedInformerFactory and similar could be replaced by static factory methods like ApiClient.defaultClient() or similar. All the No-Arg constructors for CoreV1Api, Exec and such are easy to miss. Explicit is better than implicit.

[lyind]

Seems like the original issue is still valid, though.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[liyuntian]

I use client-java:16.0.0. my kubernetes version is 1.21.
I have the same problem,which version this bug is fixed?

[njucjc]

After some debugging I found that my dependency injection framework injected a No-Arg constructor created ApiClient instance in my SharedInformerFactory instances.

Sorry for the inconvenience.

I wish the No-Arg constructors for ApiClient, CoreV1Api, SharedInformerFactory and similar could be replaced by static factory methods like ApiClient.defaultClient() or similar. All the No-Arg constructors for CoreV1Api, Exec and such are easy to miss. Explicit is better than implicit.

@lyind How can I fix this issue

[lyind]

@njucjc It's long fixed (by OkHttp client upgrade and later switch to Vert.x).

Also make sure you setup a provider for a configured ApiClient via your dependency injection framework which returns the result of ClientBuilder.cluster().build(), for example.

You may want to do similar for CoreV1Api and SharedInformerFactory.