Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AIRFLOW-5398] Update contrib example DAGs to context manager #5998

Merged
merged 25 commits into from
Sep 24, 2019
Merged

[AIRFLOW-5398] Update contrib example DAGs to context manager #5998

merged 25 commits into from
Sep 24, 2019

Conversation

eladkal
Copy link
Contributor

@eladkal eladkal commented Sep 4, 2019

Make sure you have checked all steps below.

Jira

  • My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
    • https://issues.apache.org/jira/browse/AIRFLOW-5398
    • In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.
    • In case you are proposing a fundamental code change, you need to create an Airflow Improvement Proposal (AIP).
    • In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

Description

Update example DAGs of EMR to context manager and bitshift composition.

Tests

  • My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

  • My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters (not including Jira issue reference)
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

Documentation

  • In case of new functionality, my PR adds documentation that describes how to use it.
    • All the public functions and the classes in the PR contain docstrings that explain what it does
    • If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

Code Quality

  • Passes flake8

@ashb
Copy link
Member

ashb commented Sep 4, 2019

This change looks fine, but it's there a reason why just these example dags are updated rather than all? I feel there was an existing issue to update all dags to use the context manager.

@eladkal
Copy link
Contributor Author

eladkal commented Sep 4, 2019

@ashb I worked with these specific examples didn't check the other ones
I'll make the changes for all examples in contrib

@eladkal eladkal changed the title [AIRFLOW-5398] Update EMR example DAGs to context manager [AIRFLOW-5398] Update contrib example DAGs to context manager Sep 4, 2019
@eladkal
Copy link
Contributor Author

eladkal commented Sep 11, 2019

@ashb Rebased again to solve conflicts

@codecov-io
Copy link

codecov-io commented Sep 16, 2019

Codecov Report

Merging #5998 into master will decrease coverage by 0.03%.
The diff coverage is 0%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #5998      +/-   ##
==========================================
- Coverage   80.09%   80.06%   -0.04%     
==========================================
  Files         596      609      +13     
  Lines       34872    35024     +152     
==========================================
+ Hits        27932    28042     +110     
- Misses       6940     6982      +42
Impacted Files Coverage Δ
...ontrib/example_dags/example_databricks_operator.py 0% <0%> (ø) ⬆️
...dags/example_azure_container_instances_operator.py 0% <0%> (ø) ⬆️
...example_dags/example_kubernetes_executor_config.py 0% <0%> (ø) ⬆️
...ow/contrib/example_dags/example_qubole_operator.py 0% <0%> (ø) ⬆️
...ample_dags/example_emr_job_flow_automatic_steps.py 0% <0%> (ø) ⬆️
...trib/example_dags/example_azure_cosmosdb_sensor.py 0% <0%> (ø) ⬆️
...flow/contrib/example_dags/example_qubole_sensor.py 0% <0%> (ø) ⬆️
.../contrib/example_dags/example_dingding_operator.py 0% <0%> (ø) ⬆️
...contrib/example_dags/example_papermill_operator.py 0% <0%> (ø)
...low/contrib/example_dags/example_winrm_operator.py 0% <0%> (ø) ⬆️
... and 109 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 25c53b0...a9b8393. Read the comment docs.

@mik-laj
Copy link
Member

mik-laj commented Sep 17, 2019

@eladkal Can you extract an example from docs/howto/operator/papermill.rst file and improve it also?

@eladkal
Copy link
Contributor Author

eladkal commented Sep 18, 2019

@mik-laj Done

@feluelle feluelle merged commit 9c52848 into apache:master Sep 24, 2019
ashb pushed a commit to ashb/airflow that referenced this pull request Oct 11, 2019
…#5998)

- replaces the explicit assignment of a dag to a task to be implicit via context manager
- changes task dependencies to be defined via bit shifting operator instead of set_upstream/set_downstream
- update docs for papermill to extract the code example from the code

(cherry picked from commit 9c52848)
adityav pushed a commit to adityav/airflow that referenced this pull request Oct 14, 2019
…#5998)

- replaces the explicit assignment of a dag to a task to be implicit via context manager
- changes task dependencies to be defined via bit shifting operator instead of set_upstream/set_downstream
- update docs for papermill to extract the code example from the code

(cherry picked from commit 9c52848)
@eladkal eladkal deleted the 5398 branch November 6, 2019 18:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants