-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optimize CI builds for unimportant pyproject.toml changes #37305
Merged
potiuk
merged 1 commit into
apache:main
from
potiuk:optimize-ci-builds-for-unimportant-pyproject-toml-changes
Feb 10, 2024
Merged
Optimize CI builds for unimportant pyproject.toml changes #37305
potiuk
merged 1 commit into
apache:main
from
potiuk:optimize-ci-builds-for-unimportant-pyproject-toml-changes
Feb 10, 2024
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This one should help with significantly speeding D401-only PRS. |
potiuk
force-pushed
the
optimize-ci-builds-for-unimportant-pyproject-toml-changes
branch
2 times, most recently
from
February 10, 2024 14:45
ea010a3
to
654d70f
Compare
When dependencies change in pyproject.toml, we should run build with `upgrade-to-newer-dependencies`, however we should not run it when dependencies in pyproject.toml do not change. That saves about 30 minutes of elapsed time of the build and heavily limits the number of tests executed. It takes about 30 minutes now to build the image that has "upgrade-to-newer-dependencies", we only usually run default Python image in such case and we do not run many tests that are not needed (for example K8S tests). This PR optimizes out the case where non-dependency changes only are done in pyproject.toml. This happens for example when you only change docstrings and remove ruff rules in [[tools.ruff]] section. We compare the dependencies and optional dependencies coming from the change and only when there is a change in those, we set the `upgrade-to-newer-dependencies` flag. We also print what changed. Similarly full-tests-needed is only set when build-system changes or when dependencies change, because then we want to make sure new dependencies are working on all Python versions.
potiuk
force-pushed
the
optimize-ci-builds-for-unimportant-pyproject-toml-changes
branch
from
February 10, 2024 15:33
654d70f
to
60767ce
Compare
Taragolis
approved these changes
Feb 10, 2024
potiuk
deleted the
optimize-ci-builds-for-unimportant-pyproject-toml-changes
branch
February 10, 2024 16:08
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Feb 10, 2024
The apache#37305 optimized CI jobs to limit tests run when pyproject.toml is changed without impacting dependencies, optional_dependencies and build-system. However it impacts results of other unit tests as they were using `HEAD` as commit in selective checks, Instead of using HEAD, we are using neutral commit (one only changing INTHEWILD) for all such tests, so that we do not introduce side effects coming from the incoming PR.
This was referenced Feb 10, 2024
potiuk
added a commit
that referenced
this pull request
Feb 10, 2024
The #37305 optimized CI jobs to limit tests run when pyproject.toml is changed without impacting dependencies, optional_dependencies and build-system. However it impacts results of other unit tests as they were using `HEAD` as commit in selective checks, Instead of using HEAD, we are using neutral commit (one only changing INTHEWILD) for all such tests, so that we do not introduce side effects coming from the incoming PR.
potiuk
added a commit
that referenced
this pull request
Feb 12, 2024
When dependencies change in pyproject.toml, we should run build with `upgrade-to-newer-dependencies`, however we should not run it when dependencies in pyproject.toml do not change. That saves about 30 minutes of elapsed time of the build and heavily limits the number of tests executed. It takes about 30 minutes now to build the image that has "upgrade-to-newer-dependencies", we only usually run default Python image in such case and we do not run many tests that are not needed (for example K8S tests). This PR optimizes out the case where non-dependency changes only are done in pyproject.toml. This happens for example when you only change docstrings and remove ruff rules in [[tools.ruff]] section. We compare the dependencies and optional dependencies coming from the change and only when there is a change in those, we set the `upgrade-to-newer-dependencies` flag. We also print what changed. Similarly full-tests-needed is only set when build-system changes or when dependencies change, because then we want to make sure new dependencies are working on all Python versions. (cherry picked from commit 6ce5225)
potiuk
added a commit
that referenced
this pull request
Feb 12, 2024
The #37305 optimized CI jobs to limit tests run when pyproject.toml is changed without impacting dependencies, optional_dependencies and build-system. However it impacts results of other unit tests as they were using `HEAD` as commit in selective checks, Instead of using HEAD, we are using neutral commit (one only changing INTHEWILD) for all such tests, so that we do not introduce side effects coming from the incoming PR. (cherry picked from commit 08e655a)
potiuk
added
the
changelog:skip
Changes that should be skipped from the changelog (CI, tests, etc..)
label
Feb 12, 2024
potiuk
added a commit
that referenced
this pull request
Feb 13, 2024
When dependencies change in pyproject.toml, we should run build with `upgrade-to-newer-dependencies`, however we should not run it when dependencies in pyproject.toml do not change. That saves about 30 minutes of elapsed time of the build and heavily limits the number of tests executed. It takes about 30 minutes now to build the image that has "upgrade-to-newer-dependencies", we only usually run default Python image in such case and we do not run many tests that are not needed (for example K8S tests). This PR optimizes out the case where non-dependency changes only are done in pyproject.toml. This happens for example when you only change docstrings and remove ruff rules in [[tools.ruff]] section. We compare the dependencies and optional dependencies coming from the change and only when there is a change in those, we set the `upgrade-to-newer-dependencies` flag. We also print what changed. Similarly full-tests-needed is only set when build-system changes or when dependencies change, because then we want to make sure new dependencies are working on all Python versions. (cherry picked from commit 6ce5225)
potiuk
added a commit
that referenced
this pull request
Feb 13, 2024
The #37305 optimized CI jobs to limit tests run when pyproject.toml is changed without impacting dependencies, optional_dependencies and build-system. However it impacts results of other unit tests as they were using `HEAD` as commit in selective checks, Instead of using HEAD, we are using neutral commit (one only changing INTHEWILD) for all such tests, so that we do not introduce side effects coming from the incoming PR. (cherry picked from commit 08e655a)
sunank200
pushed a commit
to astronomer/airflow
that referenced
this pull request
Feb 21, 2024
When dependencies change in pyproject.toml, we should run build with `upgrade-to-newer-dependencies`, however we should not run it when dependencies in pyproject.toml do not change. That saves about 30 minutes of elapsed time of the build and heavily limits the number of tests executed. It takes about 30 minutes now to build the image that has "upgrade-to-newer-dependencies", we only usually run default Python image in such case and we do not run many tests that are not needed (for example K8S tests). This PR optimizes out the case where non-dependency changes only are done in pyproject.toml. This happens for example when you only change docstrings and remove ruff rules in [[tools.ruff]] section. We compare the dependencies and optional dependencies coming from the change and only when there is a change in those, we set the `upgrade-to-newer-dependencies` flag. We also print what changed. Similarly full-tests-needed is only set when build-system changes or when dependencies change, because then we want to make sure new dependencies are working on all Python versions.
sunank200
pushed a commit
to astronomer/airflow
that referenced
this pull request
Feb 21, 2024
The apache#37305 optimized CI jobs to limit tests run when pyproject.toml is changed without impacting dependencies, optional_dependencies and build-system. However it impacts results of other unit tests as they were using `HEAD` as commit in selective checks, Instead of using HEAD, we are using neutral commit (one only changing INTHEWILD) for all such tests, so that we do not introduce side effects coming from the incoming PR.
ephraimbuddy
pushed a commit
that referenced
this pull request
Feb 22, 2024
When dependencies change in pyproject.toml, we should run build with `upgrade-to-newer-dependencies`, however we should not run it when dependencies in pyproject.toml do not change. That saves about 30 minutes of elapsed time of the build and heavily limits the number of tests executed. It takes about 30 minutes now to build the image that has "upgrade-to-newer-dependencies", we only usually run default Python image in such case and we do not run many tests that are not needed (for example K8S tests). This PR optimizes out the case where non-dependency changes only are done in pyproject.toml. This happens for example when you only change docstrings and remove ruff rules in [[tools.ruff]] section. We compare the dependencies and optional dependencies coming from the change and only when there is a change in those, we set the `upgrade-to-newer-dependencies` flag. We also print what changed. Similarly full-tests-needed is only set when build-system changes or when dependencies change, because then we want to make sure new dependencies are working on all Python versions. (cherry picked from commit 6ce5225)
ephraimbuddy
pushed a commit
that referenced
this pull request
Feb 22, 2024
The #37305 optimized CI jobs to limit tests run when pyproject.toml is changed without impacting dependencies, optional_dependencies and build-system. However it impacts results of other unit tests as they were using `HEAD` as commit in selective checks, Instead of using HEAD, we are using neutral commit (one only changing INTHEWILD) for all such tests, so that we do not introduce side effects coming from the incoming PR. (cherry picked from commit 08e655a)
abhishekbhakat
pushed a commit
to abhishekbhakat/my_airflow
that referenced
this pull request
Mar 5, 2024
When dependencies change in pyproject.toml, we should run build with `upgrade-to-newer-dependencies`, however we should not run it when dependencies in pyproject.toml do not change. That saves about 30 minutes of elapsed time of the build and heavily limits the number of tests executed. It takes about 30 minutes now to build the image that has "upgrade-to-newer-dependencies", we only usually run default Python image in such case and we do not run many tests that are not needed (for example K8S tests). This PR optimizes out the case where non-dependency changes only are done in pyproject.toml. This happens for example when you only change docstrings and remove ruff rules in [[tools.ruff]] section. We compare the dependencies and optional dependencies coming from the change and only when there is a change in those, we set the `upgrade-to-newer-dependencies` flag. We also print what changed. Similarly full-tests-needed is only set when build-system changes or when dependencies change, because then we want to make sure new dependencies are working on all Python versions.
abhishekbhakat
pushed a commit
to abhishekbhakat/my_airflow
that referenced
this pull request
Mar 5, 2024
The apache#37305 optimized CI jobs to limit tests run when pyproject.toml is changed without impacting dependencies, optional_dependencies and build-system. However it impacts results of other unit tests as they were using `HEAD` as commit in selective checks, Instead of using HEAD, we are using neutral commit (one only changing INTHEWILD) for all such tests, so that we do not introduce side effects coming from the incoming PR.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When dependencies change in pyproject.toml, we should run build
with
upgrade-to-newer-dependencies
, however we should not runit when dependencies in pyproject.toml do not change.
That saves about 30 minutes of elapsed time of the build and heavily
limits the number of tests executed. It takes about 30 minutes now
to build the image that has "upgrade-to-newer-dependencies", we only
usually run default Python image in such case and we do not run many
tests that are not needed (for example K8S tests).
This PR optimizes out the case where non-dependency changes only are
done in pyproject.toml. This happens for example when you only change
docstrings and remove ruff rules in [[tools.ruff]] section.
We compare the dependencies and optional dependencies coming from
the change and only when there is a change in those, we set the
upgrade-to-newer-dependencies
flag. We also print what changed.Similarly full-tests-needed is only set when build-system changes
or when dependencies change, because then we want to make sure new
dependencies are working on all Python versions.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.