Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI and testing infrastructure improvements. #3896

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
+1,013 −2,486
Open

CI and testing infrastructure improvements. #3896

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
265bb77
Update micromamba docker image and setup-micromamba action
zaneselvans Oct 3, 2024
c20d4fe
Merge branch 'main' into update-micromamba-docker
zaneselvans Oct 4, 2024
4cc2001
Merge branch 'main' into update-micromamba-docker
zaneselvans Oct 4, 2024
b890d3b
Merge branch 'main' into update-micromamba-docker
zaneselvans Oct 4, 2024
9587805
Merge branch 'main' into update-micromamba-docker
zaneselvans Oct 4, 2024
1182084
Re-lock dependencies and update pre-commit hooks
zaneselvans Oct 5, 2024
e131261
Update to pre-commit 4.0.0
zaneselvans Oct 6, 2024
6d7b2ec
Only try to remove pudl-dev conda env if it exists.
zaneselvans Oct 6, 2024
272e94a
Use standard 4-core GHA runner
zaneselvans Oct 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions .github/actionlint.yaml
View file
Open in desktop
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Configuration no longer required because we don't need to use a special large runner.

This file was deleted.

12 changes: 5 additions & 7 deletions .github/workflows/pytest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ jobs:
fetch-depth: 2

- name: Install conda-lock environment with micromamba
uses: mamba-org/setup-micromamba@v1
uses: mamba-org/setup-micromamba@v2
with:
environment-file: environments/conda-lock.yml
environment-name: pudl-dev
Expand Down Expand Up @@ -78,7 +78,7 @@ jobs:
fetch-depth: 2

- name: Install conda-lock environment with micromamba
uses: mamba-org/setup-micromamba@v1
uses: mamba-org/setup-micromamba@v2
with:
environment-file: environments/conda-lock.yml
environment-name: pudl-dev
Expand Down Expand Up @@ -120,9 +120,7 @@ jobs:
path: .coverage

ci-integration:
runs-on:
group: large-runner-group
labels: ubuntu-latest-4core
runs-on: ubuntu-latest
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See this pytest action run for confirmation that the larger paid runner is no longer necessary.

if: github.event_name == 'workflow_dispatch' || github.event.merge_group
permissions:
contents: read
Expand All @@ -139,7 +137,7 @@ jobs:
fetch-depth: 2

- name: Install Conda environment using mamba
uses: mamba-org/setup-micromamba@v1
uses: mamba-org/setup-micromamba@v2
with:
environment-file: environments/conda-lock.yml
environment-name: pudl-dev
Expand Down Expand Up @@ -224,7 +222,7 @@ jobs:
run: |
find coverage -type f
- name: Install Micromamba
uses: mamba-org/setup-micromamba@v1
uses: mamba-org/setup-micromamba@v2
with:
init-shell: bash
environment-name: coverage
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/update-conda-lockfile.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,12 @@ jobs:
with:
token: ${{ secrets.PUDL_BOT_PAT }}
- name: Install Micromamba
uses: mamba-org/setup-micromamba@v1
uses: mamba-org/setup-micromamba@v2
with:
environment-name: conda-lock
create-args: >-
python=3.12
conda-lock>=2.5.2
conda-lock>=2.5.7
prettier
- name: Run conda-lock to recreate lockfile from scratch
run: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/zenodo-cache-sync.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ jobs:
uses: actions/checkout@v4

- name: Install Conda environment using mamba
uses: mamba-org/setup-micromamba@v1
uses: mamba-org/setup-micromamba@v2
with:
environment-file: environments/conda-lock.yml
environment-name: pudl-dev
Expand Down
12 changes: 6 additions & 6 deletions .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ repos:

# Other file formatting, plus common Git mistakes & text file standardization:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.6.0
rev: v5.0.0
hooks:
- id: check-added-large-files # Don't accidentally commit giant files.
- id: check-merge-conflict # Watch for lingering merge markers.
Expand All @@ -29,7 +29,7 @@ repos:
# Formatters: hooks that re-write Python & documentation files
####################################################################################
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.6.8
rev: v0.6.9
hooks:
- id: ruff
args: [--fix, --exit-non-zero-on-fix]
Expand Down Expand Up @@ -60,7 +60,7 @@ repos:

# Check Github Actions
- repo: https://github.com/rhysd/actionlint
rev: v1.7.2
rev: v1.7.3
hooks:
- id: actionlint

Expand All @@ -71,7 +71,7 @@ repos:
hooks:
- id: nb-output-clear
name: nb-output-clear
stages: [commit]
stages: [pre-commit]
language: system
verbose: false
files: ".*\\.ipynb"
Expand All @@ -81,7 +81,7 @@ repos:

- id: unit-tests
name: unit-tests
stages: [commit]
stages: [pre-commit]
language: system
verbose: false
pass_filenames: false
Expand All @@ -90,7 +90,7 @@ repos:

- id: conda-lock
name: conda-lock
stages: [commit]
stages: [pre-commit]
language: system
verbose: false
files: "pyproject.toml"
Expand Down
5 changes: 4 additions & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,10 @@ conda-lock.yml: pyproject.toml
.PHONY: pudl-dev
pudl-dev:
${mamba} run --name base ${mamba} install --quiet --yes "conda-lock>=2.5.7"
${mamba} run --name base ${mamba} env remove --yes --name pudl-dev
# Only attempt to remove the pudl-dev environment if it already exists.
if ${mamba} env list | grep -q pudl-dev; then \
${mamba} env remove --quiet --yes --name pudl-dev; \
fi
Comment on lines -55 to +58
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I discovered this was failing after removing my local pudl-dev environment, and attempting to recreate it with the make target. Oops.

${mamba} run --name base conda-lock install \
--name pudl-dev \
--${mamba} \
Expand Down
2 changes: 1 addition & 1 deletion docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
FROM mambaorg/micromamba:1.5.9
FROM mambaorg/micromamba:1.5.10
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As of yet there's no micromamba 2.0 image, but I imagine it'll come out shortly.


ENV PGDATA=${CONTAINER_HOME}/pgdata

Check warning on line 3 in docker/Dockerfile

View workflow job for this annotation

GitHub Actions / Test building the PUDL ETL Docker image

Variables should be defined before their use 

UndefinedVar: Usage of undefined variable '$CONTAINER_HOME'
More info: https://docs.docker.com/go/dockerfile/rule/undefined-var/

USER root

Expand Down
Loading
Loading