-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Moving CI to GitHub actions #3669
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kwigley
force-pushed
the
github-actions
branch
9 times, most recently
from
August 2, 2021 14:21
b7728ae
to
431fc8d
Compare
kwigley
force-pushed
the
github-actions
branch
18 times, most recently
from
August 2, 2021 20:40
28885e1
to
7bd8b60
Compare
* Test failures file * Add testing branch * Adding upload steps * Adding date to name * Adding to integration * Always upload artifacts * Adding adapter type * Always publish unit test results * Adding comments
kwigley
force-pushed
the
github-actions
branch
from
August 24, 2021 19:44
03c17e6
to
bebeb4d
Compare
kwigley
force-pushed
the
github-actions
branch
from
August 24, 2021 19:49
bebeb4d
to
1c1b7e2
Compare
kwigley
pushed a commit
that referenced
this pull request
Aug 24, 2021
* test * test test * try this again * test actions in same repo * nvm revert * formatting * fix sh script for building dists * fix windows build * add concurrency * fix random 'Cannot track experimental parser info when active user is None' error * fix build workflow * test slim ci * has changes * set up postgres for other OS * update descriptions * turn off python3.9 unit tests * add changelog * clean up todo * Update .github/workflows/main.yml * create actions for common code * temp commit to test * cosmetic updates * dev review feedback * updates * fix build checks * rm auto formatting changes * review feeback: update order of script for setting up postgres on macos runner * review feedback: add reasoning for not using secrets in workflow * review feedback: rm unnecessary changes * more review feedback * test pull_request_target action * fix path to cli tool * split up lint and unit workflows for clear resposibilites * rm `branches-ignore` filter from pull request trigger * testing push event * test * try this again * test actions in same repo * nvm revert * formatting * fix windows build * add concurrency * fix build workflow * test slim ci * has changes * set up postgres for other OS * update descriptions * turn off python3.9 unit tests * add changelog * clean up todo * Update .github/workflows/main.yml * create actions for common code * cosmetic updates * dev review feedback * updates * fix build checks * rm auto formatting changes * review feedback: add reasoning for not using secrets in workflow * review feedback: rm unnecessary changes * more review feedback * test pull_request_target action * fix path to cli tool * split up lint and unit workflows for clear resposibilites * rm `branches-ignore` filter from pull request trigger * test dynamic matrix generation * update label logic * finishing touches * align naming * pass opts to pytest * slim down push matrix, there are a lot of jobs * test bump num of proc * update matrix for all event triggers * handle case when no changes require integration tests * dev review feedback * clean up and add branch name for testing * Add test results publishing as artifact (#3794) * Test failures file * Add testing branch * Adding upload steps * Adding date to name * Adding to integration * Always upload artifacts * Adding adapter type * Always publish unit test results * Adding comments * rm unecessary env var * fix changelog * update job name * clean up python deps Co-authored-by: leahwicz <[email protected]>
kwigley
pushed a commit
that referenced
this pull request
Aug 24, 2021
* test * test test * try this again * test actions in same repo * nvm revert * formatting * fix sh script for building dists * fix windows build * add concurrency * fix random 'Cannot track experimental parser info when active user is None' error * fix build workflow * test slim ci * has changes * set up postgres for other OS * update descriptions * turn off python3.9 unit tests * add changelog * clean up todo * Update .github/workflows/main.yml * create actions for common code * temp commit to test * cosmetic updates * dev review feedback * updates * fix build checks * rm auto formatting changes * review feeback: update order of script for setting up postgres on macos runner * review feedback: add reasoning for not using secrets in workflow * review feedback: rm unnecessary changes * more review feedback * test pull_request_target action * fix path to cli tool * split up lint and unit workflows for clear resposibilites * rm `branches-ignore` filter from pull request trigger * testing push event * test * try this again * test actions in same repo * nvm revert * formatting * fix windows build * add concurrency * fix build workflow * test slim ci * has changes * set up postgres for other OS * update descriptions * turn off python3.9 unit tests * add changelog * clean up todo * Update .github/workflows/main.yml * create actions for common code * cosmetic updates * dev review feedback * updates * fix build checks * rm auto formatting changes * review feedback: add reasoning for not using secrets in workflow * review feedback: rm unnecessary changes * more review feedback * test pull_request_target action * fix path to cli tool * split up lint and unit workflows for clear resposibilites * rm `branches-ignore` filter from pull request trigger * test dynamic matrix generation * update label logic * finishing touches * align naming * pass opts to pytest * slim down push matrix, there are a lot of jobs * test bump num of proc * update matrix for all event triggers * handle case when no changes require integration tests * dev review feedback * clean up and add branch name for testing * Add test results publishing as artifact (#3794) * Test failures file * Add testing branch * Adding upload steps * Adding date to name * Adding to integration * Always upload artifacts * Adding adapter type * Always publish unit test results * Adding comments * rm unecessary env var * fix changelog * update job name * clean up python deps Co-authored-by: leahwicz <[email protected]>
kwigley
pushed a commit
that referenced
this pull request
Aug 25, 2021
* test * test test * try this again * test actions in same repo * nvm revert * formatting * fix sh script for building dists * fix windows build * add concurrency * fix random 'Cannot track experimental parser info when active user is None' error * fix build workflow * test slim ci * has changes * set up postgres for other OS * update descriptions * turn off python3.9 unit tests * add changelog * clean up todo * Update .github/workflows/main.yml * create actions for common code * temp commit to test * cosmetic updates * dev review feedback * updates * fix build checks * rm auto formatting changes * review feeback: update order of script for setting up postgres on macos runner * review feedback: add reasoning for not using secrets in workflow * review feedback: rm unnecessary changes * more review feedback * test pull_request_target action * fix path to cli tool * split up lint and unit workflows for clear resposibilites * rm `branches-ignore` filter from pull request trigger * testing push event * test * try this again * test actions in same repo * nvm revert * formatting * fix windows build * add concurrency * fix build workflow * test slim ci * has changes * set up postgres for other OS * update descriptions * turn off python3.9 unit tests * add changelog * clean up todo * Update .github/workflows/main.yml * create actions for common code * cosmetic updates * dev review feedback * updates * fix build checks * rm auto formatting changes * review feedback: add reasoning for not using secrets in workflow * review feedback: rm unnecessary changes * more review feedback * test pull_request_target action * fix path to cli tool * split up lint and unit workflows for clear resposibilites * rm `branches-ignore` filter from pull request trigger * test dynamic matrix generation * update label logic * finishing touches * align naming * pass opts to pytest * slim down push matrix, there are a lot of jobs * test bump num of proc * update matrix for all event triggers * handle case when no changes require integration tests * dev review feedback * clean up and add branch name for testing * Add test results publishing as artifact (#3794) * Test failures file * Add testing branch * Adding upload steps * Adding date to name * Adding to integration * Always upload artifacts * Adding adapter type * Always publish unit test results * Adding comments * rm unecessary env var * fix changelog * update job name * clean up python deps Co-authored-by: leahwicz <[email protected]> Co-authored-by: leahwicz <[email protected]>
kwigley
pushed a commit
that referenced
this pull request
Aug 25, 2021
* test * test test * try this again * test actions in same repo * nvm revert * formatting * fix sh script for building dists * fix windows build * add concurrency * fix random 'Cannot track experimental parser info when active user is None' error * fix build workflow * test slim ci * has changes * set up postgres for other OS * update descriptions * turn off python3.9 unit tests * add changelog * clean up todo * Update .github/workflows/main.yml * create actions for common code * temp commit to test * cosmetic updates * dev review feedback * updates * fix build checks * rm auto formatting changes * review feeback: update order of script for setting up postgres on macos runner * review feedback: add reasoning for not using secrets in workflow * review feedback: rm unnecessary changes * more review feedback * test pull_request_target action * fix path to cli tool * split up lint and unit workflows for clear resposibilites * rm `branches-ignore` filter from pull request trigger * testing push event * test * try this again * test actions in same repo * nvm revert * formatting * fix windows build * add concurrency * fix build workflow * test slim ci * has changes * set up postgres for other OS * update descriptions * turn off python3.9 unit tests * add changelog * clean up todo * Update .github/workflows/main.yml * create actions for common code * cosmetic updates * dev review feedback * updates * fix build checks * rm auto formatting changes * review feedback: add reasoning for not using secrets in workflow * review feedback: rm unnecessary changes * more review feedback * test pull_request_target action * fix path to cli tool * split up lint and unit workflows for clear resposibilites * rm `branches-ignore` filter from pull request trigger * test dynamic matrix generation * update label logic * finishing touches * align naming * pass opts to pytest * slim down push matrix, there are a lot of jobs * test bump num of proc * update matrix for all event triggers * handle case when no changes require integration tests * dev review feedback * clean up and add branch name for testing * Add test results publishing as artifact (#3794) * Test failures file * Add testing branch * Adding upload steps * Adding date to name * Adding to integration * Always upload artifacts * Adding adapter type * Always publish unit test results * Adding comments * rm unecessary env var * fix changelog * update job name * clean up python deps Co-authored-by: leahwicz <[email protected]> Co-authored-by: leahwicz <[email protected]>
IS-Josh
pushed a commit
to IS-Josh/dbt
that referenced
this pull request
Sep 4, 2021
* test * test test * try this again * test actions in same repo * nvm revert * formatting * fix sh script for building dists * fix windows build * add concurrency * fix random 'Cannot track experimental parser info when active user is None' error * fix build workflow * test slim ci * has changes * set up postgres for other OS * update descriptions * turn off python3.9 unit tests * add changelog * clean up todo * Update .github/workflows/main.yml * create actions for common code * temp commit to test * cosmetic updates * dev review feedback * updates * fix build checks * rm auto formatting changes * review feeback: update order of script for setting up postgres on macos runner * review feedback: add reasoning for not using secrets in workflow * review feedback: rm unnecessary changes * more review feedback * test pull_request_target action * fix path to cli tool * split up lint and unit workflows for clear resposibilites * rm `branches-ignore` filter from pull request trigger * testing push event * test * try this again * test actions in same repo * nvm revert * formatting * fix windows build * add concurrency * fix build workflow * test slim ci * has changes * set up postgres for other OS * update descriptions * turn off python3.9 unit tests * add changelog * clean up todo * Update .github/workflows/main.yml * create actions for common code * cosmetic updates * dev review feedback * updates * fix build checks * rm auto formatting changes * review feedback: add reasoning for not using secrets in workflow * review feedback: rm unnecessary changes * more review feedback * test pull_request_target action * fix path to cli tool * split up lint and unit workflows for clear resposibilites * rm `branches-ignore` filter from pull request trigger * test dynamic matrix generation * update label logic * finishing touches * align naming * pass opts to pytest * slim down push matrix, there are a lot of jobs * test bump num of proc * update matrix for all event triggers * handle case when no changes require integration tests * dev review feedback * clean up and add branch name for testing * Add test results publishing as artifact (dbt-labs#3794) * Test failures file * Add testing branch * Adding upload steps * Adding date to name * Adding to integration * Always upload artifacts * Adding adapter type * Always publish unit test results * Adding comments * rm unecessary env var * fix changelog * update job name * clean up python deps Co-authored-by: leahwicz <[email protected]>
TeddyCr
pushed a commit
to TeddyCr/dbt
that referenced
this pull request
Sep 9, 2021
* test * test test * try this again * test actions in same repo * nvm revert * formatting * fix sh script for building dists * fix windows build * add concurrency * fix random 'Cannot track experimental parser info when active user is None' error * fix build workflow * test slim ci * has changes * set up postgres for other OS * update descriptions * turn off python3.9 unit tests * add changelog * clean up todo * Update .github/workflows/main.yml * create actions for common code * temp commit to test * cosmetic updates * dev review feedback * updates * fix build checks * rm auto formatting changes * review feeback: update order of script for setting up postgres on macos runner * review feedback: add reasoning for not using secrets in workflow * review feedback: rm unnecessary changes * more review feedback * test pull_request_target action * fix path to cli tool * split up lint and unit workflows for clear resposibilites * rm `branches-ignore` filter from pull request trigger * testing push event * test * try this again * test actions in same repo * nvm revert * formatting * fix windows build * add concurrency * fix build workflow * test slim ci * has changes * set up postgres for other OS * update descriptions * turn off python3.9 unit tests * add changelog * clean up todo * Update .github/workflows/main.yml * create actions for common code * cosmetic updates * dev review feedback * updates * fix build checks * rm auto formatting changes * review feedback: add reasoning for not using secrets in workflow * review feedback: rm unnecessary changes * more review feedback * test pull_request_target action * fix path to cli tool * split up lint and unit workflows for clear resposibilites * rm `branches-ignore` filter from pull request trigger * test dynamic matrix generation * update label logic * finishing touches * align naming * pass opts to pytest * slim down push matrix, there are a lot of jobs * test bump num of proc * update matrix for all event triggers * handle case when no changes require integration tests * dev review feedback * clean up and add branch name for testing * Add test results publishing as artifact (dbt-labs#3794) * Test failures file * Add testing branch * Adding upload steps * Adding date to name * Adding to integration * Always upload artifacts * Adding adapter type * Always publish unit test results * Adding comments * rm unecessary env var * fix changelog * update job name * clean up python deps Co-authored-by: leahwicz <[email protected]>
iknox-fa
pushed a commit
that referenced
this pull request
Feb 8, 2022
* test * test test * try this again * test actions in same repo * nvm revert * formatting * fix sh script for building dists * fix windows build * add concurrency * fix random 'Cannot track experimental parser info when active user is None' error * fix build workflow * test slim ci * has changes * set up postgres for other OS * update descriptions * turn off python3.9 unit tests * add changelog * clean up todo * Update .github/workflows/main.yml * create actions for common code * temp commit to test * cosmetic updates * dev review feedback * updates * fix build checks * rm auto formatting changes * review feeback: update order of script for setting up postgres on macos runner * review feedback: add reasoning for not using secrets in workflow * review feedback: rm unnecessary changes * more review feedback * test pull_request_target action * fix path to cli tool * split up lint and unit workflows for clear resposibilites * rm `branches-ignore` filter from pull request trigger * testing push event * test * try this again * test actions in same repo * nvm revert * formatting * fix windows build * add concurrency * fix build workflow * test slim ci * has changes * set up postgres for other OS * update descriptions * turn off python3.9 unit tests * add changelog * clean up todo * Update .github/workflows/main.yml * create actions for common code * cosmetic updates * dev review feedback * updates * fix build checks * rm auto formatting changes * review feedback: add reasoning for not using secrets in workflow * review feedback: rm unnecessary changes * more review feedback * test pull_request_target action * fix path to cli tool * split up lint and unit workflows for clear resposibilites * rm `branches-ignore` filter from pull request trigger * test dynamic matrix generation * update label logic * finishing touches * align naming * pass opts to pytest * slim down push matrix, there are a lot of jobs * test bump num of proc * update matrix for all event triggers * handle case when no changes require integration tests * dev review feedback * clean up and add branch name for testing * Add test results publishing as artifact (#3794) * Test failures file * Add testing branch * Adding upload steps * Adding date to name * Adding to integration * Always upload artifacts * Adding adapter type * Always publish unit test results * Adding comments * rm unecessary env var * fix changelog * update job name * clean up python deps Co-authored-by: leahwicz <[email protected]> automatic commit by git-black, original commits: ab06149
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
resolves #3688
This PR moves our CI to GitHub actions. I created the following 2 workflows.
main.yml
This workflow is the "main" CI workflow. This will run on all PRs, pushes to mainline branches ('develop', '*.latest`, etc.), and when manually dispatched in the GitHub UI. Today this workflow run static code checks (mypy, flake8) and unit tests. This could also include other lint checks, formatting checks, etc. This also builds, tests, and stores python distributions (wheel and source) as well, this can be used in other workflows (build docker image, publishing, etc.). This workflow runs for PRs from forked repositories and GitHub Actions does not pass secrets to forks, so this workflow would fail if it tried to access them.
integration.yml
This workflow can be triggers by untrusted actors.
Be default, only integration test affected by the code changes in the PRs will be run. The mapping of file changes and tests run can be found here: https://github.com/dbt-labs/dbt/blob/0d556de6994891561b408fe53ba8c521bc165730/.github/workflows/integration-pr.yml#L43-L55
If someone would like to run all tests or tests for a specific adapter, you can add a label to the pr to do so.
test all
-> run all integration teststest ${adapter}
-> run tests for that specific ${adapter}It also requires an additional check when the PR is from a forked repository. Right now, that additional check is a PR label
ok to test
. This extra check is needed to ensure that outside contributors do not exploit any secrets or compute. Since this action runs on thepull_request_target
event, it is important to note that the action runs in the context of the "target" of a PR. So, for a PR pointing at the develop branch, the action will run on the develop branch, but also include all secrets and even a github token with write permissions. We explicitly checkout the head of the branch that the PR is coming from in order to run checks against the code to be merged. Since only folks with write access to the repo can add labels to PRs, we can ensure that the PR is coming from a trusted actor. Please see https://securitylab.github.com/research/github-actions-preventing-pwn-requests/ for more info why we should do this.Additional notes
While testing these new workflows, I discovered an issue with python3.9 unit tests. I have turned them off for the time being and created #3689 to follow up with a fix. https://github.com/dbt-labs/dbt/blob/0d556de6994891561b408fe53ba8c521bc165730/.github/workflows/main.yml#L34
I think there is still work to be done to increase the reliability and trust in our test suite, but I believe that this PR will contribute towards that.
Checklist
CHANGELOG.md
and added information about my change to the "dbt next" section.