Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Viewing Dag Code for Stateless Webserver #8159

Merged
merged 3 commits into from
Apr 6, 2020

Conversation

kaxil
Copy link
Member

@kaxil kaxil commented Apr 6, 2020

This bug was found by @KostyaEsmukov

Details in #8151 (comment)

Basically when DagCode was instantiated it read the source code from Dag File, hence when the Webserver does not have access to the Dag File, it used to fail with FileNotFoundError:

>>> DagCode(dag_orm.fileloc)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 4, in __init__
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 433, in _initialize_instance
    manager.dispatch.init_failure(self, args, kwargs)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
    exc_value, with_traceback=exc_tb,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 430, in _initialize_instance
    return manager.original_init(*mixed[1:], **kwargs)
  File "/usr/local/lib/python3.7/site-packages/airflow/models/dagcode.py", line 59, in __init__
    self.source_code = DagCode._read_code(self.fileloc)
  File "/usr/local/lib/python3.7/site-packages/airflow/models/dagcode.py", line 63, in _read_code
    with open_maybe_zipped(fileloc, 'r') as source:
  File "/usr/local/lib/python3.7/site-packages/airflow/utils/file.py", line 102, in open_maybe_zipped
    return io.open(fileloc, mode=mode)
FileNotFoundError: [Errno 2] No such file or directory: '<...my_dag...>'

This PR fixes this and adds additional tests to validate this behavior


Make sure to mark the boxes below before creating PR: [x]


In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

@kaxil kaxil requested review from ashb and potiuk April 6, 2020 12:24
@kaxil
Copy link
Member Author

kaxil commented Apr 6, 2020

Note: This PR is for v1-10-test branch to fix 1.10.10rc3 . I will port this to Master branch once we release 1.10.10

airflow/models/dagcode.py Outdated Show resolved Hide resolved
@kaxil
Copy link
Member Author

kaxil commented Apr 6, 2020

cc @anitakar


def code(self):
@classmethod
def code(cls, fileloc):
Copy link
Member

@mik-laj mik-laj Apr 6, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like this method name. It describes what he does. It does not contain any verb. However, this is not related to this PR. It should be get_code_by_fileloc or something similar.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a separate PR to improve the code in this file, that I have targetted against Master but yeah that is not the priority for now.

@kaxil kaxil merged this pull request into apache:v1-10-test Apr 6, 2020
@kaxil kaxil deleted the fix-dag-code-with-tests branch April 6, 2020 13:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants