SQL Alchemy Conn Secret Fails to parse

[davido912]

Using the 1.10.12: Python 3.6 image and inserting
sql_alchemy_conn_secret = sql_alchemy_conn with the relevant AWS Secret ID, fails to parse and returns exception
airflow.exceptions.AirflowConfigException: error: cannot use sqlite with the LocalExecutor.

Anyone else encountering this?

[mik-laj]

CC: @kaxil

[kaxil]

Thanks @davido912 , can you confirm please what database you are using / passing in sql_alchemy_conn_secret?

[davido912]

@kaxil using an AWS RDS instance (Postgres). This instance works perfectly fine with the hardcoded string in the config, but when trying to use the _secret parameter in the config or as an env variable, it fails to detect it and fails the above mentioned error.
Any more details I can provide that might help with this?

[kaxil]

@davideanastasia Which Secrets backend are you using AWS Systems Manager or AWS Secrets Manager? Also, are you using backport-packages ?

[davido912]

@kaxil
backend = airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
package apache-airflow-backport-providers-amazon==2020.6.24 installed.

when ingesting the parameter with _secret in config, it doesn't seem to pick up that it is a secret

[kaxil]

@kaxil
backend = airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
package apache-airflow-backport-providers-amazon==2020.6.24 installed.

when ingesting the parameter with _secret in config, it doesn't seem to pick up that it is a secret

I will double check and fix it for Airflow 1.10.13 -- that will be released in couple of weeks

[kaxil]

I was not able to reproduce this.

I added the following key in Hashicorp Vault:

❯ vault kv put secret/config/sql_alchemy_conn value=sqlite:////Users/airflow/airflow/airflow.db
Key              Value
---              -----
created_time     2020-11-20T20:05:21.517705Z
deletion_time    n/a
destroyed        false
version          1

and set the following environment Variables:

❯ env | grep AIRFLOW
AIRFLOW__CORE__SQL_ALCHEMY_CONN_SECRET=sql_alchemy_conn
AIRFLOW__SECRETS__BACKEND=airflow.contrib.secrets.hashicorp_vault.VaultBackend
AIRFLOW__SECRETS__BACKEND_KWARGS='{"url":"http://127.0.0.1:8200","auth_type":"token", "token": "s.BddaHiiDdddpW6gNnqtd9lJ"}'
AIRFLOW__CORE__EXECUTOR=SequentialExecutor

Only if I try to use LocalExecutor, I get the following error:

airflow.exceptions.AirflowConfigException: error: cannot use sqlite with the LocalExecutor

which is a correct error since I am trying to use LocalExecutor with sqlite

[loyarahulDE]

@kaxil not sure if this has been solved but looks like we still encounter this error working with mysql database as local executor.
I get this error starting airflow service after making the below changes to the airflow.cfg file. My backend database is mysql and the mysql host info us stored as secret in hashicorp vault as variable

airflow.exceptions.AirflowConfigException: error: cannot use sqlite with the LocalExecutor

[secrets]
backend = airflow.providers.hashicorp.secrets.vault.VaultBackend
backend_kwargs = {xxxx..
}

[core]
executor = LocalExecutor
#sql_alchemy_conn = mysql://airflow_user:xxxx
sql_alchemy_conn_secret: "sql_alchemy_conn_value"

[potiuk]

@kaxil not sure if this has been solved but looks like we still encounter this error working with mysql database as local executor. I get this error starting airflow service after making the below changes to the airflow.cfg file. My backend database is mysql and the mysql host info us stored as secret in hashicorp vault as variable

This issuee has been solved for Airlfow 1.10 2 years ago. @loyarahulDE . It is solved. It is pointless to make a comment in issue that has been closed for (hopefully) different version of Airlfow without providinf details of your issue.

Please if you want help with your problem, open a discussion in Github discussions or slack with all details and logs you can find about your issue. But make sure to provide all the details that might help others to help you,

[potiuk]

Please watch this talk of mine from the summit to understand why this is unempathetic behaviour and try to adapt https://www.youtube.com/watch?v=G6VjYvKr2wQ&list=PLGudixcDaxY2LxjeHpZRtzq7miykjjFOn&index=54