TLS auth with EcdsaP521 works with openssl but not with rustls

[mcluseau]

Current and expected behavior

with a cluster definition like that in kubeconfig:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <REDACTED>
    server: https://10.1.1.1:6443
  name: test

and the following program:

use k8s_openapi::api::core::v1::Pod;
use kube::{
    api::{Api, ListParams},
    Client,
};

#[tokio::main]
async fn main() -> eyre::Result<()> {
    let cfg = kube::Config::infer().await?;
    let client: Client = cfg.try_into()?;

    let pods: Api<Pod> = Api::default_namespaced(client);
    println!(
        "found {} pods",
        pods.list(&ListParams::default()).await?.items.len()
    );

    Ok(())
}

(a) with the rustls feature:

$ grep ^k Cargo.toml  
k8s-openapi = { version = "0.22.0", features = ["v1_24"] }
kube = { version = "0.93.1", features = ["client", "config", "rustls-tls"], default-features = false }
$ cargo run
[...]
Error: ServiceError: client error (Connect)

Caused by:
   0: client error (Connect)
   1: invalid peer certificate: UnknownIssuer

Location:
    src/main.rs:16:9

(b) with the openssl feature:

$ grep ^k Cargo.toml  
k8s-openapi = { version = "0.22.0", features = ["v1_24"] }
kube = { version = "0.93.1", features = ["client", "config", "openssl-tls"], default-features = false }
$ cargo run
[...]
found 17 pods

Possible solution

Using openssl is a solution but from issue search I found it does not support tls-server-name.
I can live with it as I can setup the name resolution as I need it.
I can put a few hours on this and on #991 to see if I can find solutions.

Additional context

No response

Environment

$ kubectl version
Server Version: v1.28.6

Configuration and features

k8s-openapi = { version = "0.22.0", features = ["v1_24"] }
kube = { version = "0.93.1", features = ["client", "config", "rustls-tls"], default-features = false }

and

k8s-openapi = { version = "0.22.0", features = ["v1_24"] }
kube = { version = "0.93.1", features = ["client", "config", "openssl-tls"], default-features = false }

Affected crates

kube-client

Would you like to work on fixing this bug?

yes

[clux]

If you have any more specifics or ideas, then investigation is be appreciated.

i am comparing this to my own setup, and i am currently using rustls with certificate-authority-data on most my clusters, and this totally works for me™.

perhaps this is a trust issue with your ca setup / mac or something judging by the UnknownIssuer?

[mcluseau]

I've reproduced with pure tokio_rustls using the parsed kube::Config. An additional element: if using the IP as the server-name, I get UnknownIssuer; if using kubernetes (one of the names in the cert) I get HandshakeFailure.

Both ways are working with the openssl command, with verification OK. I wonder if I use an unsupported something. Here what openssl s_client -connect 10.1.1.1:6443 -servername kubernetes -CAfile cert.pem gives:

[...]
Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512
Peer signing digest: SHA512
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1339 bytes and written 410 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 521 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
[...]

I'll check further later, but I wanted to share tonight's results :)

[mcluseau]

it's now very likely that it's a compatibility issue:

• support for my algos is recent Support for ecdsaWithSHA512 signature algorithm rustls/rustls#1367 ;
• it's through the use of aws-lc-rs, with a big hint here: https://github.com/rustls/rustls/blob/839e95ad2ba755535161e71ce09e2dd36f1ded09/rustls/tests/common/mod.rs#L258
• enabling tokio-rustls feature aws-lc-rs fails at cargo level with error: failed to select a version for the requirement 'aws-lc-rs = "^1.8.1"', you have to add it manually in addition to enabling the feature
• then the program panics with no process-level CryptoProvider available -- call CryptoProvider::install_default() before this point
• to solve it according to the documentation (https://docs.rs/rustls/latest/rustls/index.html#crate-features) you need to explicitly add the rustls dependency.
• annnnnd then it works!

Here's the fixed code:

use k8s_openapi::api::core::v1::Pod;
use kube::{
    api::{Api, ListParams},
    Client,
};

#[tokio::main]
async fn main() -> eyre::Result<()> {
    rustls::crypto::aws_lc_rs::default_provider()
        .install_default()
        .unwrap();

    let cfg = kube::Config::infer().await?;

    let client: Client = cfg.try_into()?;

    let pods: Api<Pod> = Api::default_namespaced(client);
    println!(
        "found {} pods",
        pods.list(&ListParams::default()).await?.items.len()
    );

    Ok(())
}

I think I can do some support work in kube-rs. My plan would be to add a aws_lc_rs feature and if enabled, add the support code somewhere so callers don't have to enable all the dependencies etc to get this working. Does that SGTyou?

[mcluseau]

TDLR; many dependencies issues where specific to me wandering around and the Cargo.toml that works with the above code is actually simple:

[package]
name = "test-kube"
version = "0.1.0"
edition = "2021"

[dependencies]
eyre = "0.6.12"
k8s-openapi = { version = "0.22.0", features = ["v1_24"] }
kube = "0.93.1"
rustls = "0.23.12"
tokio = { version = "1.40.0", features = ["macros", "rt-multi-thread"] }

The need for installing the aws_lc_rs CryptoProvider still holds though.

[clux]

Thanks for investigating this!
so it looks like some rustls pathways does not work anymore without their new aws crypto lib :/

We have originally kept that feature disabled on our end, as not to force the dependency on people who can't build it. Maybe this should be changed if the need is widespread. At the very least we should documented that people might need to opt into the rustls feature. I am not sure which one is best yet. Open to ideas.

[mcluseau]

I still think it's the way to go:

My plan would be to add a aws_lc_rs feature and if enabled, add the support code somewhere so callers don't have to enable all the dependencies etc to get this working. Does that SGTyou?

[mcluseau]

PR #1568 created for review of this change proposal.

[clux]

This is now released in kube 0.94. Have edited the PR title here a bit to make the backlink from the release a bit better.
Please correct if it's wrong, but AFAIKT the original auth issue above is specific to the algorithm - and support for aws-lc-rs incidentally fixes it.

[mcluseau]

AFAIKT the original auth issue above is specific to the algorithm - and support for aws-lc-rs incidentally fixes it.

Yes, this PR fixes the issue :)