Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

separate tox testenvs to cache CRDS reference files retrieved by initial test #7323

Closed
wants to merge 35 commits into from
Closed

separate tox testenvs to cache CRDS reference files retrieved by initial test #7323

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
752620e
separate tox testenvs to cache CRDS reference files retrieved by init…
zacharyburnett Oct 28, 2022
1a230a9
simplify cron CI
zacharyburnett Oct 28, 2022
9bca5be
fix checks job
zacharyburnett Oct 28, 2022
9aaf5a0
checkout tags
zacharyburnett Oct 28, 2022
5e51b43
fix toxenv
zacharyburnett Oct 28, 2022
ed8aeee
rename toxenvs
zacharyburnett Oct 28, 2022
f30c7b7
consolidate jobs
zacharyburnett Oct 28, 2022
cee87d6
run full matrix
zacharyburnett Oct 28, 2022
0cccaaa
test pyargs once
zacharyburnett Oct 28, 2022
800ed16
fix CRDS path and disable fail-fast
zacharyburnett Oct 28, 2022
f660d92
use string concatenation with `join()`
zacharyburnett Oct 28, 2022
790e04c
use string concatenation with `format()`
zacharyburnett Oct 28, 2022
0d53fcf
rename jobs
zacharyburnett Oct 28, 2022
91d9fec
fill matrix
zacharyburnett Oct 28, 2022
1da95ad
print home dir
zacharyburnett Oct 28, 2022
4d13fb4
print CRDS dir
zacharyburnett Oct 28, 2022
310b9c0
print CRDS dir
zacharyburnett Oct 28, 2022
f8791fd
hash reference files
zacharyburnett Oct 28, 2022
a88e1ba
add change log entry
zacharyburnett Oct 28, 2022
26e0464
set desired CRDS context from input
zacharyburnett Oct 28, 2022
94d80d6
install CRDS
zacharyburnett Oct 28, 2022
dd54c22
attempt cache after first test (to construct key from reference file …
zacharyburnett Oct 31, 2022
63cdbb3
remove `-n auto` from default testenv
zacharyburnett Oct 31, 2022
8edcd82
use Python 3.10 for checks
zacharyburnett Oct 31, 2022
f1e5f98
remove manual usage of `crds sync`
zacharyburnett Oct 31, 2022
208dbe7
fix comma
zacharyburnett Oct 31, 2022
045e5bd
save reference files hash for future cache keys
zacharyburnett Oct 31, 2022
0380292
only hash files once
zacharyburnett Oct 31, 2022
4f6569f
use latest Python in checks and build
zacharyburnett Nov 1, 2022
dc8c61f
use `pytest-xdist` in bottleneck
zacharyburnett Nov 1, 2022
40b7843
rename jobs
zacharyburnett Nov 1, 2022
2f921f5
refactor matrices
zacharyburnett Nov 1, 2022
f77fc21
cache `.tox` environment
zacharyburnett Nov 1, 2022
bd23c6c
exclude `.eggs` and `build` directories from test discovery
zacharyburnett Nov 2, 2022
1b703b3
don't cache `tox` environment
zacharyburnett Nov 2, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
204 changes: 139 additions & 65 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,86 +19,160 @@ env:
CRDS_CLIENT_RETRY_DELAY_SECONDS: 20

jobs:
tox:
check:
name: ${{ matrix.name }}
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
toxenv: [ check-style, check-security, check-install ]
python-version: [ '3.x' ]
os: [ ubuntu-latest ]
include:
- name: Python 3.10 Tests
python-version: '3.10'
os: ubuntu-latest
toxenv: py310

- name: Latest dependency versions w/coverage
os: ubuntu-latest
python-version: 3.9
toxenv: py39-cov

- name: Oldest dependency versions
os: ubuntu-latest
python-version: 3.8
toxenv: py38-oldestdeps-cov

- name: SDP dependencies in requirements-sdp.txt
os: ubuntu-latest
python-version: 3.9
toxenv: sdpdeps

- name: Installed package with --pyargs
os: ubuntu-latest
python-version: 3.9
toxenv: pyargs

- name: Verify install_requires in setup.py
os: ubuntu-latest
python-version: 3.9
toxenv: verify-install-requires

- name: Build distribution
os: ubuntu-latest
python-version: 3.9
toxenv: twine

- name: Code style check
os: ubuntu-latest
python-version: 3.9
toxenv: style

toxenv: check-style
Comment on lines 32 to +33
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These and the other 2 are already in the matrix above. Probably no need to add them here.

- name: Security audit
os: ubuntu-latest
python-version: 3.9
toxenv: security

# MacOS job is flaky on Github actions and fails routinely.
# Re-enable when things get more reliable.
# - name: macOS
# os: macos-latest
# python-version: 3.9
# toxenv: py39
toxenv: check-security
- name: Verify install_requires in setup.py
toxenv: check-install
steps:
- name: Checkout code
uses: actions/checkout@v3
- uses: actions/checkout@v3
with:
fetch-depth: 0

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
- uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}

- name: Install tox
run: |
pip install tox

- name: Run tests
run: tox -e ${{ matrix.toxenv }}

- name: Upload coverage to codecov
if: ${{ contains(matrix.toxenv,'-cov') }}
- run: pip install tox
- run: tox -e ${{ matrix.toxenv }}
quick_test:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the sole purpose of this quick_test to generate a cache that the other tests can use?

name: ${{ matrix.toxenv }} (Python ${{ matrix.python-version }}, ${{ matrix.os }})
needs: [ check ]
runs-on: ${{ matrix.os }}
strategy:
matrix:
toxenv: [ test-xdist ]
python-version: [ '3.x' ]
os: [ ubuntu-latest ]
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- run: pip install tox
- run: tox -e ${{ matrix.toxenv }}
- run: echo "REFERENCE_FILES_HASH=${{ hashFiles(format('{0}/references/*', env.CRDS_PATH)) }}" >> $GITHUB_ENV
- uses: actions/cache@v3
with:
path: ${{ env.CRDS_PATH }}
key: crds-reference-files-${{ env.REFERENCE_FILES_HASH }}
- run: echo ${{ env.REFERENCE_FILES_HASH }} > reference_files_hash.txt
- uses: actions/upload-artifact@v3
with:
name: reference_files_hash
path: reference_files_hash.txt
test:
name: ${{ matrix.toxenv }} (Python ${{ matrix.python-version }}, ${{ matrix.os }})
needs: [ quick_test ]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since these tests don't even kick off until quick_test is finished, this is going to ~double the runtime of the test suite.

runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
toxenv: [ test-xdist, test-oldestdeps, test-sdpdeps-xdist ]
python-version: [ '3.8', '3.9', '3.10', '3.11' ]
os: [ ubuntu-latest, macos-latest ]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And I don't think we should be running on macos. The runners are too slow, and they don't test anything extra compared to the linux runners. If our test suite was 3 minutes, fine. But it is over an hour on the macos runners.

Likewise, oldestdeps probably only needs to run on the oldest supported version of python (and in fact probably only can), and the sdpdeps probably only should run on whatever version of python DMS/SDP is currently using, python 3.9 currently, i.e. whatever our JenkinsfileRT says.

steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- run: pip install tox
- uses: actions/download-artifact@v3
with:
name: reference_files_hash
- run: echo "REFERENCE_FILES_HASH=$(cat reference_files_hash.txt)" >> $GITHUB_ENV
- uses: actions/cache@v3
with:
path: ${{ env.CRDS_PATH }}
key: crds-reference-files-${{ env.REFERENCE_FILES_HASH }}
- run: tox -e ${{ matrix.toxenv }}
test_pyargs:
name: ${{ matrix.toxenv }} (Python ${{ matrix.python-version }}, ${{ matrix.os }})
needs: [ test ]
runs-on: ${{ matrix.os }}
strategy:
matrix:
toxenv: [ test-pyargs-xdist ]
python-version: [ '3.x' ]
os: [ ubuntu-latest ]
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- run: pip install tox
- uses: actions/download-artifact@v3
with:
name: reference_files_hash
- run: echo "REFERENCE_FILES_HASH=$(cat reference_files_hash.txt)" >> $GITHUB_ENV
- uses: actions/cache@v3
with:
path: ${{ env.CRDS_PATH }}
key: crds-reference-files-${{ env.REFERENCE_FILES_HASH }}
- run: tox -e ${{ matrix.toxenv }}
test_with_coverage:
name: ${{ matrix.toxenv }} (Python ${{ matrix.python-version }}, ${{ matrix.os }})
needs: [ test ]
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
toxenv: [ test-cov ]
python-version: [ '3.x' ]
os: [ ubuntu-latest ]
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- run: pip install tox
- uses: actions/download-artifact@v3
with:
name: reference_files_hash
- run: echo "REFERENCE_FILES_HASH=$(cat reference_files_hash.txt)" >> $GITHUB_ENV
- uses: actions/cache@v3
with:
path: ${{ env.CRDS_PATH }}
key: crds-reference-files-${{ env.REFERENCE_FILES_HASH }}
- run: tox -e ${{ matrix.toxenv }}
- if: ${{ contains(matrix.toxenv,'-cov') }}
uses: codecov/codecov-action@v3
with:
file: ./coverage.xml
flags: unit
fail_ci_if_error: true
build:
name: ${{ matrix.toxenv }} (Python ${{ matrix.python-version }}, ${{ matrix.os }})
needs: [ test ]
runs-on: ${{ matrix.os }}
strategy:
matrix:
toxenv: [ build-twine ]
python-version: [ '3.8', '3.9', '3.10', '3.11' ]
os: [ ubuntu-latest, macos-latest ]
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- run: pip install tox
- run: tox -e ${{ matrix.toxenv }}
35 changes: 8 additions & 27 deletions .github/workflows/ci_cron.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ on:

env:
CRDS_SERVER_URL: https://jwst-crds.stsci.edu
CRDS_PATH: ~/crds_cache
CRDS_PATH: $HOME/crds_cache
CRDS_CLIENT_RETRY_COUNT: 3
CRDS_CLIENT_RETRY_DELAY_SECONDS: 20

Expand All @@ -25,32 +25,13 @@ jobs:
python-version: 3.9
toxenv: devdeps
steps:
- name: Install system packages
if: ${{ contains(matrix.toxenv,'docs') }}
run: |
sudo apt-get install graphviz texlive-latex-extra dvipng

- name: Checkout the code
uses: actions/checkout@v3
with:
fetch-depth: 0

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}

- name: Install tox
run: |
pip install tox

- name: Run tests
run: tox -e ${{ matrix.toxenv }}

- name: Upload coverage to codecov
if: ${{ contains(matrix.toxenv,'-cov') }}
uses: codecov/codecov-action@v3
Comment on lines -50 to -52
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing coverage reporting here is probably a good idea since nothing is changing on master when this cron job runs.

- run: pip install tox
- uses: actions/cache@v3
with:
file: ./coverage.xml
flags: unit
fail_ci_if_error: true
path: ${{ env.CRDS_PATH }}
key: crds-reference-files-${{ hashFiles(format('{0}/references/*', env.CRDS_PATH)) }}
- run: tox -e ${{ matrix.toxenv }}
5 changes: 5 additions & 0 deletions CHANGES.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,11 @@ extract_2d

- Fix slice limits used in extraction of WFSS 2D cutouts. [#7312]

general
-------

- refactor ``tox`` environment factors and set dependent CI jobs [#7323]

lib
---

Expand Down
2 changes: 2 additions & 0 deletions setup.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,8 @@ norecursedirs =
jwst/associations/tests/data
scripts
.tox
.eggs
build
asdf_schema_tests_enabled = true
asdf_schema_validate_default = false
asdf_schema_root = jwst/transforms/resources/schemas jwst/datamodels/schemas
Expand Down
Loading