PULL_REQUEST_WORKFLOW.rst

• Why non-standard pull request workflow?
• Selective CI Checks
• Approval Workflow and Matrix tests
• Next steps

Why non-standard pull request workflow?

This document describes the Pull Request Workflow we've implemented in Airflow. The workflow is slightly more complex than regular workflow you might encounter in most of the projects because after experiencing some huge delays in processing queues in October 2020 with GitHub Actions, we've decided to optimize the workflow to minimize the use of GitHub Actions build time by utilising selective approach on which tests and checks in the CI system are run depending on analysis of which files changed in the incoming PR and allowing the Committers to control the scope of the tests during the approval/review process.

Just to give a bit of context, we started off with the approach that we always run all tests for all the incoming PRs, however due to our matrix of tests growing, this approach did not scale with the increasing number of PRs and when we had to compete with other Apache Software Foundation projects for the 180 slots that are available for the whole organization. More Apache Software Foundation projects started to use GitHub Actions and we've started to experience long queues when our jobs waited for free slots.

We approached the problem by:

1. Improving mechanism of cancelling duplicate workflow runs more efficiently in case of queue conditions (duplicate workflow runs are generated when someone pushes a fixup quickly - leading to running both out-dated and current run to completion, taking precious slots. This has been implemented by improving cancel-workflow-run action we are using. In version 4.1 it got a new feature of cancelling all duplicates even if there is a long queue of builds.

2. Heavily decreasing strain on the GitHub Actions jobs by introducing selective checks - mechanism to control which parts of the tests are run during the tests. This is implemented by the scripts/ci/selective_ci_checks.sh script in our repository. This script analyses which part of the code has changed and based on that it sets the right outputs that control which tests are executed in the Tests workflow, and whether we need to build CI images necessary to run those steps. This allowed to heavily decrease the strain especially for the Pull Requests that were not touching code (in which case the builds can complete in < 2 minutes) but also by limiting the number of tests executed in PRs that do not touch the "core" of Airflow, or only touching some - standalone - parts of Airflow such as "Providers", "WWW" or "CLI". This solution is not yet perfect as there are likely some edge cases but it is easy to maintain and we have an escape-hatch - all the tests are always executed in main pushes, so contributors can easily spot if there is a "missed" case and fix it - both by fixing the problem and adding those exceptions to the code. More about it can be found in the Selective CI checks chapter.

3. Even more optimisation came from limiting the scope of tests to only "default" matrix parameters. So far in Airflow we always run all tests for all matrix combinations. The primary matrix components are:

   • Python versions (currently 3.6, 3.7, 3.8, 3.9)
   • Backend types (currently MySQL/Postgres)
   • Backed version (currently MySQL 5.7, MySQL 8, Postgres 9.6, Postgres 13

   We've decided that instead of running all the combinations of parameters for all matrix component we will only run default values (Python 3.6, Mysql 5.7, Postgres 9.6) for all PRs which are not approved yet by the committers. This has a nice effect, that full set of tests (though with limited combinations of the matrix) are still run in the CI for every Pull Request that needs tests at all - allowing the contributors to make sure that their PR is "good enough" to be reviewed.

   Even after approval, the automated workflows we've implemented, check if the PR seems to need "full test matrix" and provide helpful information to both contributors and committers in the form of explanatory comments and labels set automatically showing the status of the PR. Committers have still control whether they want to merge such requests automatically or ask for rebase or re-run the tests and run "full tests" by applying the "full tests needed" label and re-running such request. The "full tests needed" label is also applied automatically after approval when the change touches the "core" of Airflow - also a separate check is added to the PR so that the "merge" button status will indicate to the committer that full tests are still needed. The committer might still decide, whether to merge such PR without the "full matrix". The "escape hatch" we have - i.e. running the full matrix of tests in the "merge push" will enable committers to catch and fix such problems quickly. More about it can be found in Approval workflow and Matrix tests chapter.

4. We've also applied (and received) funds to run self-hosted runners. This is not yet implemented, due to discussions about security of self-hosted runners for public repositories. Running self-hosted runners by public repositories is currently (as of end of October 2020) Discouraged by GitHub and we are working on solving the problem - also involving Apache Software Foundation infrastructure team. This document does not describe this part of the approach. Most likely we will add soon a document describing details of the approach taken there.

Selective CI Checks

In order to optimise our CI jobs, we've implemented optimisations to only run selected checks for some kind of changes. The logic implemented reflects the internal architecture of Airflow 2.0 packages and it helps to keep down both the usage of jobs in GitHub Actions as well as CI feedback time to contributors in case of simpler changes.

We have the following test types (separated by packages in which they are):

• Always - those are tests that should be always executed (always folder)
• Core - for the core Airflow functionality (core folder)
• API - Tests for the Airflow API (api and api_connexion folders)
• CLI - Tests for the Airflow CLI (cli folder)
• WWW - Tests for the Airflow webserver (www folder)
• Providers - Tests for all Providers of Airflow (providers folder)
• Other - all other tests (all other folders that are not part of any of the above)

We also have several special kinds of tests that are not separated by packages but they are marked with pytest markers. They can be found in any of those packages and they can be selected by the appropriate pytest custom command line options. See TESTING.rst for details but those are:

• Integration - tests that require external integration images running in docker-compose
• Quarantined - tests that are flaky and need to be fixed
• Postgres - tests that require Postgres database. They are only run when backend is Postgres
• MySQL - tests that require MySQL database. They are only run when backend is MySQL

Even if the types are separated, In case they share the same backend version/python version, they are run sequentially in the same job, on the same CI machine. Each of them in a separate docker run command and with additional docker cleaning between the steps to not fall into the trap of exceeding resource usage in one big test run, but also not to increase the number of jobs per each Pull Request.

The logic implemented for the changes works as follows:

1. In case of direct push (so when PR gets merged) or scheduled run, we always run all tests and checks. This is in order to make sure that the merge did not miss anything important. The remainder of the logic is executed only in case of Pull Requests. We do not add providers tests in case DEFAULT_BRANCH is different than main, because providers are only important in main branch and PRs to main branch.
2. We retrieve which files have changed in the incoming Merge Commit (github.sha is a merge commit automatically prepared by GitHub in case of Pull Request, so we can retrieve the list of changed files from that commit directly).
3. If any of the important, environment files changed (Dockerfile, ci scripts, setup.py, GitHub workflow files), then we again run all tests and checks. Those are cases where the logic of the checks changed or the environment for the checks changed so we want to make sure to check everything. We do not add providers tests in case DEFAULT_BRANCH is different than main, because providers are only important in main branch and PRs to main branch.
4. If any of py files changed: we need to have CI image and run full static checks so we enable image building
5. If any of docs changed: we need to have CI image so we enable image building
6. If any of chart files changed, we need to run helm tests so we enable helm unit tests
7. If any of API files changed, we need to run API tests so we enable them
8. If any of the relevant source files that trigger the tests have changed at all. Those are airflow sources, chart, tests and kubernetes_tests. If any of those files changed, we enable tests and we enable image building, because the CI images are needed to run tests.
9. Then we determine which types of the tests should be run. We count all the changed files in the relevant airflow sources (airflow, chart, tests, kubernetes_tests) first and then we count how many files changed in different packages:
   1. in any case tests in Always folder are run. Those are special tests that should be run any time modifications to any Python code occurs. Example test of this type is verifying proper structure of the project including proper naming of all files.
   2. if any of the Airflow API files changed we enable API test type
   3. if any of the Airflow CLI files changed we enable CLI test type and Kubernetes tests (the K8S tests depend on CLI changes as helm chart uses CLI to run Airflow).
   4. if this is a main branch and if any of the Provider files changed we enable Providers test type
   5. if any of the WWW files changed we enable WWW test type
   6. if any of the Kubernetes files changed we enable Kubernetes test type
   7. Then we subtract count of all the specific above per-type changed files from the count of all changed files. In case there are any files changed, then we assume that some unknown files changed (likely from the core of airflow) and in this case we enable all test types above and the Core test types - simply because we do not want to risk to miss anything.
   8. In all cases where tests are enabled we also add Integration and - depending on the backend used = Postgres or MySQL types of tests.
10. Quarantined tests are always run when tests are run - we need to run them often to observe how often they fail so that we can decide to move them out of quarantine. Details about the Quarantined tests are described in TESTING.rst
11. There is a special case of static checks. In case the above logic determines that the CI image needs to be build, we run long and more comprehensive version of static checks - including Mypy, Flake8. And those tests are run on all files, no matter how many files changed. In case the image is not built, we run only simpler set of changes - the longer static checks that require CI image are skipped, and we only run the tests on the files that changed in the incoming commit - unlike flake8/mypy, those static checks are per-file based and they should not miss any important change.

Similarly to selective tests we also run selective security scans. In Pull requests, the Python scan will only run when there is a python code change and JavaScript scan will only run if there is a JavaScript or yarn.lock file change. For main builds, all scans are always executed.

The selective check algorithm is shown here:

Approval Workflow and Matrix tests

As explained above the approval and matrix tests workflow works according to the algorithm below:

1. In case of "no-code" changes - so changes that do not change any of the code or environment of the application, no test are run (this is done via selective checks above). Also no CI/PROD images are build saving extra minutes. Such build takes less than 2 minutes currently and only few jobs are run which is a very small fraction of the "full build" time.
2. When new PR is created, only a "default set" of matrix test are running. Only default values for each of the parameters are used effectively limiting it to running matrix builds for only one python version and one version of each of the backends. In this case only one CI and one PROD image is built, saving precious job slots. This build takes around 50% less time than the "full matrix" build.
3. When such PR gets approved, the system further analyses the files changed in this PR and further decision is made that should be communicated to both Committer and Reviewer.

3a) In case of "no-code" builds, a message is communicated that the PR is ready to be merged and

no tests are needed.

3b) In case of "non-core" builds a message is communicated that such PR is likely OK to be merged as is with

limited set of tests, but that the committer might decide to re-run the PR after applying "full tests needed" label, which will trigger full matrix build for tests for this PR. The committer might make further decision on what to do with this PR.

3c) In case of "core" builds (i. e. when the PR touches some "core" part of Airflow) a message is

communicated that this PR needs "full test matrix", the "full tests needed" label is applied automatically and either the contributor might rebase the request to trigger full test build or the committer might re-run the build manually to trigger such full test rebuild. Also a check "in-progress" is added, so that the committer realises that the PR is not yet "green to merge". Pull requests with "full tests needed" label always trigger the full matrix build when rebased or re-run so if the PR gets rebased, it will continue triggering full matrix build.

4. If this or another committer "request changes" in a previously approved PR with "full tests needed" label, the bot automatically removes the label, moving it back to "run only default set of parameters" mode. For PRs touching core of airflow once the PR gets approved back, the label will be restored. If it was manually set by the committer, it has to be restored manually.

Note

Note that setting the labels and adding comments might be delayed, due to limitation of GitHub Actions, in case of queues, processing of Pull Request reviews might take some time, so it is advised not to merge PR immediately after approval. Luckily, the comments describing the status of the PR trigger notifications for the PRs and they provide good "notification" for the committer to act on a PR that was recently approved.

The PR approval workflow is possible thanks to two custom GitHub Actions we've developed:

• Get workflow origin
• Label when approved

Next steps

We are planning to also propose the approach to other projects from Apache Software Foundation to make it a common approach, so that our effort is not limited only to one project.

Discussion about it in this discussion