Kubeclient does not support certificate authorities with intermediate certificate

[jperville]

The issue

We tried to run a Ruby application (using the kubeclient gem) on a kubernetes cluster which uses a custom CA (the cluster's CA itself is signed by another custom CA, hence the need for intermediate certificates).

Kubeclient (initialized from the KUBECONFIG file) fails with SSL verify errors.

How to reproduce

First, we create a KUBECONFIG file in the container by executing the following script:

write_client_kubeconfig() {
  KUBECTL=${1:?please provide the KUBECTL environment variable}

  # only needed for writing a kubeconfig:
  master_url=${MASTER_URL:-https://kubernetes.default.svc.cluster.local:443}
  master_ca=${MASTER_CA:-/var/run/secrets/kubernetes.io/serviceaccount/ca.crt}
  token_file=${TOKEN_FILE:-/var/run/secrets/kubernetes.io/serviceaccount/token}

  # set up configuration for openshift client
  if [ -n "${WRITE_KUBECONFIG:-''}" ]; then
      # craft a kubeconfig, usually at $KUBECONFIG location
      ${KUBECTL} config set-cluster master \
            --certificate-authority="${master_ca}" \
            --server="${master_url}"
      ${KUBECTL} config set-credentials account \
            --token="$(cat ${token_file})"
      ${KUBECTL} config set-context current \
            --cluster=master \
            --user=account \
            --namespace="${infra_project}"
      ${KUBECTL} config use-context current
  fi
}

write_client_kubeconfig kubectl

Then we try listing services cluster-wide using the kubectl binary:

kubectl get services --all-namespaces

This should work, assuming that the current service account is allowed to list services cluster-wide.

Finally, we try listing the same services in Ruby using kubeclient:

mkdir -p /tmp/test
cd /tmp/test

cat <<EOF | tee Gemfile
source 'https://rubygems.org'
gem 'kubeclient', '~> 4.8'
EOF

bundle install --path .bundle

cat <<EOF | tee test.rb
require 'kubeclient'

config = Kubeclient::Config.read(ENV.fetch('KUBECONFIG'))
context = config.context
ssl_options = context.ssl_options
auth_options = context.auth_options

client = Kubeclient::Client.new(
    context.api_endpoint, 'v1',
    ssl_options: ssl_options, auth_options: auth_options
)

services_names = client.get_services.map { |svc| svc.metadata.name }
puts services_names.inspect
EOF

bundle exec ruby test.rb

On clusters where the kubernetes CA has been signed by an intermediate CA, kubeclient fails to verify the kubernetes API certificate, even if the cacert in /var/run/secrets/kubernetes.io/serviceaccount/ca.crt contains the intermediate certificates.

We see the following stacktrace:

 Kubeclient::HttpError: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get issuer certificate)
/opt/deploy/.bundle/ruby/2.6.0/gems/kubeclient-4.6.0/lib/kubeclient/common.rb:130:in `rescue in handle_exception'
/opt/deploy/.bundle/ruby/2.6.0/gems/kubeclient-4.6.0/lib/kubeclient/common.rb:120:in `handle_exception'
/opt/deploy/.bundle/ruby/2.6.0/gems/kubeclient-4.6.0/lib/kubeclient/common.rb:567:in `fetch_entities'
/opt/deploy/.bundle/ruby/2.6.0/gems/kubeclient-4.6.0/lib/kubeclient/common.rb:554:in `load_entities'
/opt/deploy/.bundle/ruby/2.6.0/gems/kubeclient-4.6.0/lib/kubeclient/common.rb:134:in `discover'

The explanation

The reason of this behavior is the use of OpenSSL::X509::Store#add_cert in https://github.com/abonas/kubeclient/blob/v4.9.1/lib/kubeclient/config.rb#L58 .

Per the documentation:

add_cert(cert)
Adds the OpenSSL::X509::Certificate cert to the certificate store.

If we had used the add_file method instead of add_cert every certificate included in the cacert file would have been loaded.

Documentation of add_file:

add_file(file) → self
Adds the certificates in file to the certificate store. file is the path to the file, and the file contains one or more certificates in PEM format concatenated together.

I will submit a PR next

[cben]

Closing, assuming #461 fixed this.

[DocX]

Hello, is there an ETA when this fix will be released in the client release? AFAIK it is not in the latest v4.9.2 cc @cben

[cben]

reopening:

• the fix only covered certificate-authority, but certificate-authority-data would still have the bug.
• according to new test I added for this in [v4.y] Load cluster ca certificates using OpenSSL::X509::Store#add_file #552, the fix doesn't seem to work on Mac?
  my mistake in test certs, now fixed 😌 and Mac passed.

[cben]

interesting, in lostisland/faraday#371 (opened on faraday but actaully not specific to faraday at all) people had possibly related findings 👀

• Also, in ca_file you have to have all certificates in the chain in correct order.

• The most important here is -subject_hash and -subject_hash_old - it wasn't working because of different OpenSSL versions used to prepare hashes and to compile Ruby etc.

• I also had this issue on OSX.
  Make sure your Ruby OpenSSL is a similar version as your system openssl!

EDIT: turns out the failure was my mistake, not Mac-specific 🎉. And those discussion of "hashes" IIUC are only interesting for OpenSSL::X509::Store#add_path, not relevant for #add_file.

[cben]

@DocX certificate-authority fix released in gem version 4.9.3 (together with other Config security fixes — see #554 and #555, recommend updating ASAP!)

Keeping issue open for certificate-authority-data.