ssl.ca.location and related settings do not affect libcurl requests for OIDC token endpoints

[lpsinger]

libcurl is used to request the OIDC token endpoint, but libcurl's SSL settings are not configured by librdkafka.

Description

The ssl.ca.location setting in particular is important because the system on which librdkafka is run may have CA certificates in a different location than the system on which librdkafka was built. However the SSL settings only affect the SSL context used for the broker connection, and do not affect the SSL context used by libcurl for HTTPS requests.

How to reproduce

N/A

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• librdkafka version (release number or git tag): c481ad3
• Apache Kafka version: 3.1.0
• librdkafka client configuration: <REPLACE with e.g., message.timeout.ms=123, auto.reset.offset=earliest, ..>
• Operating system: <REPLACE with e.g., Centos 5 (x64)>
• Provide logs (with debug=.. as necessary) from librdkafka
• Provide broker log excerpts
• Critical issue

[edenhill]

Right, I wonder though if they can always be considered to require the same CA chain, seeing how they could be run by different organizations.

Maybe we need a separate sasl.oauthbearer.oidc.ssl.ca.location property that defaults to ssl.ca.location ?

[lpsinger]

Indeed. I was hoping that there would be some guidance from the Kafka Java client config, but there is not.

But I think that the common case that both need to be able to find the OS CA bundle is the most important one.

[edenhill]

Okay, so I think we have three different scenarios we want to support:

1. inherit: Use same CA chain as ssl.ca.location. This should be the default.
2. system: Use the system default CA chain,.
3. /some/path: Use a specific CA chain

[lpsinger]

Unfortunately the new KIP-768 support in confluent-kafka-python does not work at all in the wheels because the ssl.ca.location = 'probe' behavior has no impact on curl.

I think that this is going to evolve into a major refactoring of the ssl context configuration, because almost every single setting under ssl.* will need to be replicated under sasl.oauthbearer.oidc.ssl.*. For instance, one may legitimately want to do mTLS with the IdP.

[RafalSkolasinski]

I don't see any activity here so just wanted to check - is there currently any workaround for situation where CAs are not located under /etc/ssl/certs/ca-certificates.crt?

[lpsinger]

I don't see any activity here so just wanted to check - is there currently any workaround for situation where CAs are not located under /etc/ssl/certs/ca-certificates.crt?

Our current workaround is to not use confluent-kafka-python's built-in KIP-768 support and instead to use a workalike oauth_cb written in pure Python. Here is an example in our project: https://github.com/nasa-gcn/gcn-kafka-python/blob/main/gcn_kafka/oidc.py

[RafalSkolasinski]

We are using golang clients in our context. If I understand you right @lpsinger, you are basically fetching JWT tokens on your own and pass it to Kafka client?

[lpsinger]

Yes. The oath callback mechanism exists in librdkafka too, so you could implement an analogous workaround in go.

[JohnPreston]

FYI I didn't face this issue with using Python images built based on Amazon Linux 2/2023, but indeed that's a problem it seems for all Debian-based builds. My workaround was to simply symlink the ca-bundle cert to the path the library complains it can't find it.

For example, see https://github.com/JohnPreston/kafka-overwatch/blob/main/Dockerfile#L21-L23

But that does warrant the idea of being able to provide the sasl.oauth.ca.location (or something along that name) as an option for the users to override the default.

[thomasnal]

@lpsinger This post is tremendously useful. I have implemented the callback, however, librdkafka seems to be bugged handling OAuth extensions together with the callback. Do you use extensions in your case?

The callback retrieves the token - thanks for the sample. But Kafka fails the whole authentication with error,

SASL authentication error: Authentication failed: 1 extensions are invalid! 
They are: logicalCluster: CLUSTER_ID_MISSING_OR_EMPTY (after 5268ms in state AUTH_REQ) (_AUTHENTICATION)

I verified the extensions attribute is provided the correct value 'sasl.oauthbearer.extensions': 'logicalCluster=***,identityPoolId=***',. Since I am connecting to confluent Kafka the extensions attribute is required.

[lpsinger]

Do you use extensions in your case?

No, we are not setting sasl.oauthbearer.extensions in our project.

[thomasnal]

Thanks. I have just been through Kafka source code. I figured out they expect the callback to return extensions as extra arguments in the tuple. For anyone else, it expects the extensions in form of a dict.

ext = {'logicalCluster':'***,'identityPoolId':'***'}
principal = ''  # No complaints when empty string
return token["access_token"], token["expires_at"], principal, ext