How to connect to an externalDatabase with a password from Vault?

[marekargalas]

Checks

• I have checked for existing issues.
• I have looked through the FAQ.

Question

Hi,

I am injecting Vault secrets to Airflow (using annotations not backend engine). I would like to also use it for getting PG database password to connect metastore. I found how this could be done but I suppose just with running Vault backend. Is there a way to supply password from attached folder or environment variables? Thanks!

Edit: Based on this it seems there is no way to get password other way. Any chance to adjust it in future?

[thesuperzapper]

@marekargalas Right now the chart only supports getting the password from a Secret resource using externalDatabase.passwordSecret and externalDatabase.passwordSecretKey, there are a number of ways to automatically create Secrets from your Vault, for example:

• secrets-store-csi-driver
• external-secrets

While it would be possible to create a value like externalDatabase.passwordFile, that specifies the path of a file in the pod which contains the password, I'm not sure how important that feature is, given the secret approach already exists.

PS: we also plan to add externalDatabase.userSecret and externalDatabase.userSecretKey to support getting the username from a Secret also, which will probably make vault integration even easier.

[marekargalas]

@thesuperzapper thanks for clarification! So AIRFLOW__CORE__SQL_ALCHEMY_CONN_SECRET is not supported as well, do I understand it right? Our use case is not use k8s secrets at all

[thesuperzapper]

@thesuperzapper thanks for clarification! So AIRFLOW__CORE__SQL_ALCHEMY_CONN_SECRET is not supported as well, do I understand it right? Our use case is not use k8s secrets at all

@marekargalas You are correct that we don't support AIRFLOW__CORE__SQL_ALCHEMY_CONN_SECRET, as this would be overridden by the chart's generated AIRFLOW__CORE__SQL_ALCHEMY_CONN (which actually uses AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD internally to set it).

NOTE: we might want to add the _SECRET configs to _helpers/validate-values.tpl, so that the helm install will fail if users try and set AIRFLOW__CORE__SQL_ALCHEMY_CONN_SECRET. Do you want to raise a PR for that?