Undefined conf when using AWS secrets manager backend and sql_alchemy_conn_secret

[vasyharan]

Apache Airflow version: 2.2.0

Kubernetes version (if you are using kubernetes) (use kubectl version): 1.19 (server), 1.21 (client)

Environment:

• Cloud provider or hardware configuration: AWS EKS
• OS (e.g. from /etc/os-release): Docker image (apache/airflow:2.0.2-python3.7)
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What happened:

airflow bin fails during configuration initialization (stack below). I see a similar issue reported here: #13254, but my error is slightly different.

  File "/home/airflow/.local/bin/airflow", line 5, in <module>
    from airflow.__main__ import main
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/__init__.py", line 34, in <module>
    from airflow import settings
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/settings.py", line 37, in <module>
    from airflow.configuration import AIRFLOW_HOME, WEBSERVER_CONFIG, conf  # NOQA F401
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py", line 1098, in <module>
    conf = initialize_config()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py", line 860, in initialize_config
    conf.validate()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py", line 199, in validate
    self._validate_config_dependencies()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py", line 227, in _validate_config_dependencies
    is_sqlite = "sqlite" in self.get('core', 'sql_alchemy_conn')
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py", line 328, in get
    option = self._get_environment_variables(deprecated_key, deprecated_section, key, section)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py", line 394, in _get_environment_variables
    option = self._get_env_var_option(section, key)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py", line 298, in _get_env_var_option
    return _get_config_value_from_secret_backend(os.environ[env_var_secret_path])
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py", line 83, in _get_config_value_from_secret_backend
    secrets_client = get_custom_secret_backend()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py", line 999, in get_custom_secret_backend
    secrets_backend_cls = conf.getimport(section='secrets', key='backend')
NameError: name 'conf' is not defined

What you expected to happen:

airflow to correctly initialize the configuration.

How to reproduce it:
airflow.cfg

[core]
# ...
sql_alchemy_conn_secret = some-key

# ...
[secrets]
backend = airflow.contrib.secrets.aws_secrets_manager.SecretsManagerBackend
backend_kwargs = {config_prefix: 'airflow/config', connections_prefix: 'airflow/connections', variables_prefix: 'airflow/variables'}

Anything else we need to know:

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[uranusjr]

There's a circular reference happening here. The secret backend needs to read configs, but the config wants to read the secret backend during initialisation. Not sure how best to fix this...

[potiuk]

Yeah. We do have a problem with 'conf' causing circular references (not regular circular imports, just references because we have circular dependencies between modules in our code which IMHO should be unentangled and make the dependencies straightforward. It's pretty deeply embedded in the way how configuration is done and the fact that we 'plug-in' the code (like secret backends) to conf which then through chain of dependencies goes back and requires conf (but there are other, similar places with conf being in the middle of circular dependencies).

We have some hacks in the code I solved already quite a few of those in the past (for example here: #13260), but it was more of the band-aid rather than final solution.

Happy to brainstorm virtually and make some attempt to do it @uranusjr if you would like to (but likely something that we can attempt after 2.1 is released). That might also help in finally getting rid of the last pylint_todo.txt entries - which are there mainly because of the conf becoming the reason for circular dependencies (just to be clear - those are circular dependencies, not imports, those are pretty different problems).

[krnhotwings]

Recently ran into this issue. Adding an additional FYI that this does not occur in 2.0.1

The error seems to only occur when sql_alchemy_conn_secret is set. Does not occur when the following are set: core.fernet_key_secret, webserver.secret_key_secret