Using configProviders other than file fails validation

[swehner]

When using any configProvider with a name different from file, config validation fails because it receives the raw value and tries to validate the connection.

Example config:

connector.class=com.snowflake.kafka.connector.SnowflakeSinkConnector
config.providers=secretManager
config.providers.secretManager.class=com.github.jcustenborder.kafka.config.aws.SecretsManagerConfigProvider
snowflake.user.name=testuser
snowflake.private.key=${secretManager:data-platform-demo/msk_connect_credentials:private_key}

Error is

  | 2023-11-16T17:57:16.000+01:00 | [Worker-0a7081b95c2eab512] [2023-11-16 16:57:16,917] ERROR [SF_KAFKA_CONNECTOR] Validate: Error connecting to snowflake:[SF_KAFKA_CONNECTOR] Exception: Invalid private key
  | 2023-11-16T17:57:16.000+01:00 | [Worker-0a7081b95c2eab512] Error Code: 0002
  | 2023-11-16T17:57:16.000+01:00 | [Worker-0a7081b95c2eab512] Detail: private key should be a valid PEM RSA private key

Adding some more log statements it receives the uninterpolated string:

2023-11-16T17:57:16.000+01:00 | [Worker-0a7081b95c2eab512] [2023-11-16 16:57:16,903] ERROR [SF_KAFKA_CONNECTOR] %%%% got private key ${secretManager:data-platform-demo/msk_connect_credentials:private_key} (com.snowflake.kafka.connector.internal.InternalUtils:94)

Opened a pull request to fix this issue here: #747

[hitesh-yadav-x]

Assuming that you are trying to pull the credentials from AWS Secrets Manager. May I suggest use the default environment provider that kafka has and inject the variables in the container/run environment.

If using DOCKER set
ENV CONNECT_CONFIG_PROVIDERS="env"
ENV CONNECT_CONFIG_PROVIDERS_ENV_CLASS="org.apache.kafka.common.config.provider.EnvVarConfigProvider"

Then in config use below:

"snowflake.role.name": "${env:SNOWFLAKE_ROLE}",

[swehner]

Assuming that you are trying to pull the credentials from AWS Secrets Manager. May I suggest use the default environment provider that kafka has and inject the variables in the container/run environment.

If using DOCKER set ENV CONNECT_CONFIG_PROVIDERS="env" ENV CONNECT_CONFIG_PROVIDERS_ENV_CLASS="org.apache.kafka.common.config.provider.EnvVarConfigProvider"

Then in config use below:

"snowflake.role.name": "${env:SNOWFLAKE_ROLE}",

Thanks for the suggestions - I'm trying to pull them from property store (ssm), but secrets manager would be fine.
Problem is that i'm using msk connect as of now and I don't think I can set environment vars for the workers. I will keep it in mind once we switch over to a different cluster setup.

In any case isn't it a bit strange to restrict this specific validation behavior only to a configprovider named file?

[sfc-gh-xhuang]

Merged in #753

This will go out in the 2.2.0 release slated for January