Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NameError: name conf is not defined in configuration.py after upgraded to 2.1.0 #16079

Closed
doowhtron opened this issue May 26, 2021 · 3 comments · Fixed by #16088
Closed

NameError: name conf is not defined in configuration.py after upgraded to 2.1.0 #16079

doowhtron opened this issue May 26, 2021 · 3 comments · Fixed by #16088
Labels
kind:bug This is a clearly a bug

Comments

@doowhtron
Copy link

doowhtron commented May 26, 2021 •
edited
Loading

Apache Airflow version:

Kubernetes version (if you are using kubernetes) (use kubectl version):

Environment:

  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):centos7
  • Kernel (e.g. uname -a): 3.10.0
  • Install tools:
  • Others:

What happened:

After upgraded from 2.0.1 to 2.1.0, airflow fails with the error:

Traceback (most recent call last):
  File "/data/apps/pyenv/versions/airflow-py381/bin/airflow", line 6, in <module>
    from airflow.__main__ import main
  File "/data/apps/pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/airflow/__init__.py", line 34, in <module>
    from airflow import settings
  File "/data/apps/pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/airflow/settings.py", line 35, in <module>
    from airflow.configuration import AIRFLOW_HOME, WEBSERVER_CONFIG, conf  # NOQA F401
  File "/data/apps/pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/airflow/configuration.py", line 1117, in <module>
    conf = initialize_config()
  File "/data/apps/pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/airflow/configuration.py", line 879, in initialize_config
    conf.validate()
  File "/data/apps/pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/airflow/configuration.py", line 204, in validate
    self._validate_config_dependencies()
  File "/data/apps/pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/airflow/configuration.py", line 232, in _validate_config_dependencies
    is_sqlite = "sqlite" in self.get('core', 'sql_alchemy_conn')
  File "/data/apps/pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/airflow/configuration.py", line 344, in get
    option = self._get_environment_variables(deprecated_key, deprecated_section, key, section)
  File "/data/apps/pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/airflow/configuration.py", line 410, in _get_environment_variables
    option = self._get_env_var_option(section, key)
  File "/data/apps/pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/airflow/configuration.py", line 314, in _get_env_var_option
    return _get_config_value_from_secret_backend(os.environ[env_var_secret_path])
  File "/data/apps/pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/airflow/configuration.py", line 83, in _get_config_value_from_secret_backend
    secrets_client = get_custom_secret_backend()
  File "/data/apps/pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/airflow/configuration.py", line 1018, in get_custom_secret_backend
    secrets_backend_cls = conf.getimport(section='secrets', key='backend')
NameError: name 'conf' is not defined

I have mask the password in airflow.cfg using the following env vars and define my own secrets backend in airflow.cfg

export AIRFLOW__CORE__SQL_ALCHEMY_CONN_SECRET=AIRFLOW__CORE__SQL_ALCHEMY_CONN_ENC
export AIRFLOW__CELERY__BROKER_URL_SECRET=AIRFLOW__CELERY__BROKER_URL_ENC
export AIRFLOW__CELERY__RESULT_BACKEND_SECRET=AIRFLOW__CELERY__RESULT_BACKEND_ENC

And I fixed this by moving "conf.validate()":

    if not os.path.isfile(WEBSERVER_CONFIG):
        import shutil

        log.info('Creating new FAB webserver config file in: %s', WEBSERVER_CONFIG)
        shutil.copy(_default_config_file_path('default_webserver_config.py'), WEBSERVER_CONFIG)

    # conf.validate()

    return conf
...

conf = initialize_config()
secrets_backend_list = initialize_secrets_backends()
conf.validate()

What you expected to happen:

The upgradation should be compatible.

How to reproduce it:

Use a self-define secret backend

Anything else we need to know:
@doowhtron doowhtron added the kind:bug This is a clearly a bug label May 26, 2021
@potiuk potiuk mentioned this issue May 26, 2021
Merged
@theis188
Copy link

I noticed this issue when using both a secret for sqlalchemy conn, and a custom secrets backend. It existed in Airflow 2.1.0 when using these two (sqlalchemy conn secret, and a custom secrets backend), but not in 2.0.0, and not in 2.1.0 when using a static sqlalchemy conn string.

@potiuk
Copy link
Member

potiuk commented May 27, 2021

I noticed this issue when using both a secret for sqlalchemy conn, and a custom secrets backend. It existed in Airflow 2.1.0 when using these two (sqlalchemy conn secret, and a custom secrets backend), but not in 2.0.0, and not in 2.1.0 when using a static sqlalchemy conn string.

@Theis @doowhtron -> would it be possible that you double-check the fix I proposed in: potiuk@19742c1 - it's a simple change in "airflow/configuration.py" - just moving the validation a little later in the config.

Would be great to verify that the fix works :)

@potiuk
Copy link
Member

potiuk commented May 27, 2021 •
edited
Loading

Ah. I see you updated the description @doowhtron and you'v come to the same fix I proposed :). Good!. Great minds think alike!

potiuk added a commit to potiuk/airflow that referenced this issue May 27, 2021
@potiuk
Fixes problem where conf variable was used before initialization
9fb9101 
There was a problem that when we initialized configuration, we've run
validate() which - among others - checkd if the connection is an `sqlite`
but when the SQLAlchemy connection was not configured via variable but
via secret manager, it has fallen back to secret_backend, which should
be configured via conf and initialized.
The problem is that the "conf" object is not yet created, because
the "validate()" method has not finished yet and
"initialize_configuration" has not yet returned.
This led to snake eating its own tail.

This PR defers the validate() method to after secret backends have
been initialized. The effect of it is that secret backends might
be initialized with configuration that is not valid, but there are
no real negative consequences of this.

Fixes: apache#16079
Fixes: apache#15685

starting
potiuk added a commit that referenced this issue May 27, 2021
@potiuk
Fixes problem where conf variable was used before initialization (#16088
65519ab 
)

There was a problem that when we initialized configuration, we've run
validate() which - among others - checkd if the connection is an `sqlite`
but when the SQLAlchemy connection was not configured via variable but
via secret manager, it has fallen back to secret_backend, which should
be configured via conf and initialized.
The problem is that the "conf" object is not yet created, because
the "validate()" method has not finished yet and
"initialize_configuration" has not yet returned.
This led to snake eating its own tail.

This PR defers the validate() method to after secret backends have
been initialized. The effect of it is that secret backends might
be initialized with configuration that is not valid, but there are
no real negative consequences of this.

Fixes: #16079
Fixes: #15685

starting
jhtimmins pushed a commit to astronomer/airflow that referenced this issue Jun 3, 2021
@potiuk @jhtimmins
Fixes problem where conf variable was used before initialization (apa…
194762f 
…che#16088)

There was a problem that when we initialized configuration, we've run
validate() which - among others - checkd if the connection is an `sqlite`
but when the SQLAlchemy connection was not configured via variable but
via secret manager, it has fallen back to secret_backend, which should
be configured via conf and initialized.
The problem is that the "conf" object is not yet created, because
the "validate()" method has not finished yet and
"initialize_configuration" has not yet returned.
This led to snake eating its own tail.

This PR defers the validate() method to after secret backends have
been initialized. The effect of it is that secret backends might
be initialized with configuration that is not valid, but there are
no real negative consequences of this.

Fixes: apache#16079
Fixes: apache#15685

starting

(cherry picked from commit 65519ab)
ashb pushed a commit that referenced this issue Jun 22, 2021
@potiuk @ashb
Fixes problem where conf variable was used before initialization (#16088
9a3fb62 
)

There was a problem that when we initialized configuration, we've run
validate() which - among others - checkd if the connection is an `sqlite`
but when the SQLAlchemy connection was not configured via variable but
via secret manager, it has fallen back to secret_backend, which should
be configured via conf and initialized.
The problem is that the "conf" object is not yet created, because
the "validate()" method has not finished yet and
"initialize_configuration" has not yet returned.
This led to snake eating its own tail.

This PR defers the validate() method to after secret backends have
been initialized. The effect of it is that secret backends might
be initialized with configuration that is not valid, but there are
no real negative consequences of this.

Fixes: #16079
Fixes: #15685

starting

(cherry picked from commit 65519ab)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind:bug This is a clearly a bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fixes problem where conf variable was used before initialization potiuk/airflow
3 participants
@potiuk @doowhtron @theis188