Skip to content
This repository has been archived by the owner on Dec 16, 2022. It is now read-only.

update Python environment setup in GitHub Actions #5272

Merged
merged 20 commits into from
Jun 21, 2021
Merged

update Python environment setup in GitHub Actions #5272

merged 20 commits into from
Jun 21, 2021

Conversation

epwalsh
Copy link
Member

@epwalsh epwalsh commented Jun 17, 2021

This PR makes some improvements to how we cache and initialize the Python environment within GitHub Actions jobs. In particular, we now

  • Create, use, and cache a virtualenv instead of the Python installation created by the setup-python action to avoid messing with system-wide files. Messing with system-wide files is okay for GitHub-hosted runners, but not for self-hosted runners.
  • Consolidates where the PyTorch/Torchvision versions are defined. They still have to be hard-coded in the GH Actions config for now, but just in a single spot (an environment variable at the top of the file).
  • Completely reset all caches weekly to keep them fresh.
  • Use a secondary cache restore key based on the torch installation to speed up builds when a non-torch requirement changes.

Another notable change proposed here is that we only run the "CPU Tests" job on the latest supported Python version. We still test to make sure we can properly install AllenNLP on all officially supported versions, but I don't think it's worth running all of our tests on different Python versions because it seems redundant and takes up build time and cache space. The only issues we've had between Python versions have had to do with either how the package is built or syntax, both of which are still checked by our "Test Install" job that runs on all supported Python versions. Cache space is limited to 5GiB I think, and any cache that includes a torch version with CUDA support takes up a sizeable portion of that (2.2GiB).

@epwalsh epwalsh changed the title update Python environment setup update Python environment setup in GitHub Actions Jun 18, 2021
@epwalsh epwalsh merged commit 6af9069 into main Jun 21, 2021
@epwalsh epwalsh deleted the ci-updates branch June 21, 2021 16:15
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants