Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[AIRFLOW-XXX] Documents about task_concurrency and pool #5262

Merged
merged 1 commit into from
May 9, 2019
Merged

[AIRFLOW-XXX] Documents about task_concurrency and pool #5262

merged 1 commit into from
May 9, 2019

Conversation

milton0825
Copy link
Contributor

@milton0825 milton0825 commented May 9, 2019 •
edited
Loading

Make sure you have checked all steps below.

Jira

  • My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
    • https://issues.apache.org/jira/browse/AIRFLOW-XXX
    • In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.
    • In case you are proposing a fundamental code change, you need to create an Airflow Improvement Proposal (AIP).
    • In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

Description

  • Here are some details about my PR, including screenshots of any UI changes:

Tests

  • My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

  • My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters (not including Jira issue reference)
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

Documentation

  • In case of new functionality, my PR adds documentation that describes how to use it.
    • All the public functions and the classes in the PR contain docstrings that explain what it does
    • If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

Code Quality

  • Passes flake8

@milton0825 milton0825 mentioned this pull request May 9, 2019
Merged
6 tasks
@codecov-io
Copy link

Codecov Report

Merging #5262 into master will decrease coverage by <.01%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #5262      +/-   ##
==========================================
- Coverage   78.66%   78.66%   -0.01%     
==========================================
  Files         470      470              
  Lines       30013    30013              
==========================================
- Hits        23610    23609       -1     
- Misses       6403     6404       +1
Impacted Files Coverage Δ
airflow/models/taskinstance.py 92.42% <0%> (-0.18%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 66b3054...66e2487. Read the comment docs.

@codecov-io
Copy link

codecov-io commented May 9, 2019 •
edited
Loading

Codecov Report

Merging #5262 into master will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master    #5262   +/-   ##
=======================================
  Coverage   78.66%   78.66%           
=======================================
  Files         470      470           
  Lines       30013    30013           
=======================================
  Hits        23610    23610           
  Misses       6403     6403

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 66b3054...d72b210. Read the comment docs.

XD-DENG
XD-DENG reviewed May 9, 2019
View reviewed changes
docs/faq.rst
- ``max_active_runs``: the Airflow scheduler will run no more than ``max_active_runs`` DagRuns of your DAG at a given time. If you do not set the ``max_active_runs`` in your DAG, the scheduler will use the default value from the ``max_active_runs_per_dag`` entry in your ``airflow.cfg``.

- ``pool``: This variable controls the number of concurrent running task instances assigned to the pool.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Personally I think pool is too specific to be put in this section.

On the other hand, if folks agree it's a good idea to add this here, you need to update line 175 ;-)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will change it to a few so that this documentation is more scalable in some senseXD

feng-tao
feng-tao reviewed May 9, 2019
View reviewed changes
docs/faq.rst Outdated
- ``concurrency``: The Airflow scheduler will run no more than ``$concurrency`` task instances for your DAG at any given time. Concurrency is defined in your Airflow DAG. If you do not set the concurrency on your DAG, the scheduler will use the default value from the ``dag_concurrency`` entry in your ``airflow.cfg``.
- ``parallelism``: This variable controls the number of task instances that runs simultaneously across the whole Airflow cluster. User could increase the parallelism variable in the ``airflow.cfg``.
- ``concurrency``: The Airflow scheduler will run no more than ``concurrency`` task instances for your DAG at any given time. Concurrency is defined in your Airflow DAG. If you do not set the concurrency on your DAG, the scheduler will use the default value from the ``dag_concurrency`` entry in your ``airflow.cfg``.
- ``task_concurrency``: This variable controls the number of concurrent running task instances across ``execution_dates`` per task.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

across all the dag runs or across execution dates?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dag_runs may be more accurate

@feng-tao feng-tao merged commit d04d7bc into apache:master May 9, 2019
JCoder01 pushed a commit to JCoder01/airflow that referenced this pull request May 14, 2019
@milton0825
[AIRFLOW-XXX] Documents about task_concurrency and pool (apache#5262)
5e9e94a
@milton0825 milton0825 deleted the update-concurrency-doc branch May 18, 2019 07:24
andriisoldatenko pushed a commit to andriisoldatenko/airflow that referenced this pull request Jul 26, 2019
@milton0825 @andriisoldatenko
[AIRFLOW-XXX] Documents about task_concurrency and pool (apache#5262)
f547dfe
wmorris75 pushed a commit to modmed/incubator-airflow that referenced this pull request Jul 29, 2019
@milton0825
[AIRFLOW-XXX] Documents about task_concurrency and pool (apache#5262)
7e5da60
kaxil pushed a commit that referenced this pull request Dec 17, 2019
@milton0825 @kaxil
[AIRFLOW-XXX] Documents about task_concurrency and pool (#5262)
6c3d7f9 
(cherry picked from commit d04d7bc)
ashb pushed a commit that referenced this pull request Dec 19, 2019
@milton0825 @ashb
[AIRFLOW-XXX] Documents about task_concurrency and pool (#5262)
f96f973 
(cherry picked from commit d04d7bc)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@XD-DENG XD-DENG XD-DENG left review comments

@feng-tao feng-tao feng-tao approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@milton0825 @codecov-io @feng-tao @XD-DENG