Skip to content

Commit

Permalink
chore: Change [ALLGCP] tag to use a github label instead [FE-187] (#1033
Browse files Browse the repository at this point in the history 
)

* chore: Change [ALLGCP] tag to use a github label instead [FE-187]

* added the context 'github-write'

* added the context 'github-write'

* added the context 'github-write'

* Minor format change

* Testing adding 'github-write' to existing 'context' to see if Google environment variables show up

* Test: Move 'gcp' to the outer 'context:'

* Moved the matrix -> parameters -> context: ['gcp'] to the newly added outer context below the 'github-write'

* Rebased and added requires 'check-if-nightly-tests-should-run' to Slurm tests

* Added missing 'requires:' to 'build-react:'

* Updated the ALLGCP documentation

* Changed github-write to github-read, as per Danny's suggestion, as well as using the '--json' option to 'gh pr view'

* Dumping out the labels for debugging

* Switched back from github-read to github-write, otherwise 'gh pr view' says 'no pull requests found for branch FE-187'

* Danny thinks he may have fixed the github-read, so switching back from github-write to github-read to see what happens.
  • Loading branch information
@rcorujo @determined-ci
rcorujo authored and determined-ci committed Feb 2, 2024
1 parent a681070 commit e69d428
Show file tree
Hide file tree
Showing 2 changed files with 67 additions and 38 deletions.
94 changes: 64 additions & 30 deletions .circleci/real_config.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -200,27 +200,21 @@ commands:
skip-on-dev-branch:
steps:
# Install the GitHub CLI.
- gh/install
- run:
name: Check if extra GCP tests specfied
command: |
echo "Not on main, release, or rc branch. Checking for [ALLGCP] tag..."
if [[ "$COMMIT_MESSAGE" != *"[ALLGCP]"* ]] ; then
echo "[ALLGCP] tag not found. Skipping job."
circleci-agent step halt
# Check if the 'ci-run-allgcp' label was set on the pull request,
# which indicates that we want the GCP tests to run on demand.
if gh pr view --json labels -q '.labels.[] | .name' | grep -w ci-run-allgcp
then
echo "The 'ci-run-allgcp' label was found in github pull request. Proceeding to run test-e2e-slurm-*-gcp tests."
else
echo "Found [ALLGCP] tag. Proceeding to run test-e2e-slurm-*-gcp tests"
echo "The 'ci-run-allgcp' label was not found in github pull request. Skipping job."
circleci-agent step halt
fi
set-commit-message:
steps:
- run:
name: Set git commit message to environment variable
command: |
COMMIT_MESSAGE=$(git log --format=%B -n 1 $CIRCLE_SHA1)
echo "COMMIT_MESSAGE=$(printf "%q" "$COMMIT_MESSAGE")" >> $BASH_ENV
echo "Commit message: $COMMIT_MESSAGE"
set-slack-user-id:
steps:
- run:
Expand Down Expand Up @@ -2090,9 +2084,9 @@ jobs:
steps:
- run: echo "Test suite disabled."

# By default, this job only runs on the main branch unless
# oterhwise specified. To invoke this job on a developer branch, the
# "[ALLGCP]" keyword must be included in the commit message.
# By default, this job only runs on the main branch unless otherwise
# specified. To invoke this job on a developer branch, add the 'ci-run-allgcp'
# label to your pull request on github.
test-e2e-hpc-gcp:
parameters:
mark:
Expand Down Expand Up @@ -2158,7 +2152,6 @@ jobs:
- equal: [ "main", <<pipeline.git.branch>> ]
- matches: { pattern: "^release-.*$", value: <<pipeline.git.branch>> }
steps:
- set-commit-message
- skip-on-dev-branch

- skip-if-docs-only
Expand Down Expand Up @@ -3515,43 +3508,59 @@ workflows:

# Singularity over SLURM test on GCP
- test-e2e-hpc-gcp:
context:
# Provides the GITHUB_USERNAME and GITHUB_TOKEN enviroment variable
# that's required by the "gh" command for authentication.
- github-read
- gcp
matrix:
parameters:
name: [test-e2e-slurm-singularity-gcp]
context: ["gcp"]
always-run: [true]
mark: ["e2e_slurm and not parallel"]
requires:
- build-go

# Podman over SLURM test on GCP
- test-e2e-hpc-gcp:
context:
# Provides the GITHUB_USERNAME and GITHUB_TOKEN enviroment variable
# that's required by the "gh" command for authentication.
- github-read
- gcp
matrix:
parameters:
name: [test-e2e-slurm-podman-gcp]
container-run-type: ["podman"]
context: ["gcp"]
mark: ["e2e_slurm and not parallel"]
requires:
- build-go

# Enroot over SLURM test on GCP
- test-e2e-hpc-gcp:
context:
# Provides the GITHUB_USERNAME and GITHUB_TOKEN enviroment variable
# that's required by the "gh" command for authentication.
- github-read
- gcp
matrix:
parameters:
name: [test-e2e-slurm-enroot-gcp]
container-run-type: ["enroot"]
context: ["gcp"]
mark: ["e2e_slurm and not parallel"]
requires:
- build-go

# Singularity over PBS test on GCP
- test-e2e-hpc-gcp:
context:
# Provides the GITHUB_USERNAME and GITHUB_TOKEN enviroment variable
# that's required by the "gh" command for authentication.
- github-read
- gcp
matrix:
parameters:
name: [test-e2e-pbs-singularity-gcp]
context: ["gcp"]
workload-manager: ["pbs"]
always-run: [true]
mark: ["e2e_pbs and not parallel"]
Expand All @@ -3560,35 +3569,47 @@ workflows:

# Podman over PBS test on GCP
- test-e2e-hpc-gcp:
context:
# Provides the GITHUB_USERNAME and GITHUB_TOKEN enviroment variable
# that's required by the "gh" command for authentication.
- github-read
- gcp
matrix:
parameters:
name: [test-e2e-pbs-podman-gcp]
container-run-type: ["podman"]
workload-manager: ["pbs"]
context: ["gcp"]
mark: ["e2e_pbs and not parallel"]
extra-pytest-flags: ["-k 'not test_slurm_verify_home'"]
requires:
- build-go

# Enroot over PBS test on GCP
- test-e2e-hpc-gcp:
context:
# Provides the GITHUB_USERNAME and GITHUB_TOKEN enviroment variable
# that's required by the "gh" command for authentication.
- github-read
- gcp
matrix:
parameters:
name: [test-e2e-pbs-enroot-gcp]
container-run-type: ["enroot"]
workload-manager: ["pbs"]
context: ["gcp"]
mark: ["e2e_pbs and not parallel"]
requires:
- build-go

# Podman over SLURM test using Agent on GCP
- test-e2e-hpc-gcp:
context:
# Provides the GITHUB_USERNAME and GITHUB_TOKEN enviroment variable
# that's required by the "gh" command for authentication.
- github-read
- gcp
matrix:
parameters:
name: [test-e2e-slurm-agent-podman-gcp]
context: ["gcp"]
agent-use: ["-A"]
container-run-type: ["podman"]
mark: ["e2e_slurm and not parallel"]
Expand Down Expand Up @@ -4312,41 +4333,53 @@ workflows:
- check-if-nightly-tests-should-run


- build-proto
- build-helm
- build-proto:
requires:
- check-if-nightly-tests-should-run
- build-helm:
requires:
- check-if-nightly-tests-should-run
- build-docs:
requires:
- check-if-nightly-tests-should-run
- build-proto
- build-helm
- build-react
- build-react:
requires:
- check-if-nightly-tests-should-run
- package-and-push-system-local:
requires:
- check-if-nightly-tests-should-run
- build-docs
- build-react
- test-e2e-slurm:
name: test-e2e-slurm-restart
mark: "e2e_slurm_restart"
slack-mentions: "${SLACK_USER_ID}"
requires:
- check-if-nightly-tests-should-run
- package-and-push-system-local
extra-pytest-flags: "--no-compare-stats"
- test-e2e-slurm:
name: test-e2e-slurm-preemption
mark: "e2e_slurm_preemption"
slack-mentions: "${SLACK_USER_ID}"
requires:
- check-if-nightly-tests-should-run
- package-and-push-system-local
extra-pytest-flags: "--no-compare-stats"
- test-e2e-slurm:
name: test-e2e-slurm-znode
slack-mentions: "${SLACK_USER_ID}"
requires:
- check-if-nightly-tests-should-run
- package-and-push-system-local
extra-pytest-flags: "--no-compare-stats"
- test-e2e-slurm:
name: test-e2e-slurm-enroot-znode
slack-mentions: "${SLACK_USER_ID}"
requires:
- check-if-nightly-tests-should-run
- package-and-push-system-local
master_config: |
task_container_defaults:
Expand Down Expand Up @@ -4396,6 +4429,7 @@ workflows:
name: test-e2e-slurm-agent-singularity-znode
slack-mentions: "${SLACK_USER_ID}"
requires:
- check-if-nightly-tests-should-run
- package-and-push-system-local
agent-use: "-A"
extra-pytest-flags: "-k 'not node_not_available'"
Expand Down
11 changes: 3 additions & 8 deletions tools/slurm/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -134,14 +134,9 @@ This invocation specifies that all tests are to be run via the remote launcher r

# Notes on `make slurmcluster` tests on CircleCI

By default, the `test-e2e-*-gcp` jobs are not run within the `test-e2e` workflow on a **developer branch**. If you would like to invoke these jobs on a certain commit, you must add the "[ALLGCP]" keyword to the commit message. For example,
```
git add --all
git commit -m "[ALLGCP] This is my commit where all hpc-gcp jobs will run."
git push
```
will invoke the slurm-gcp jobs within the `test-e2e` workflow.
**On branch `main` and `release/rc` branches, these jobs always run, regardless of commit message.**
By default, the `test-e2e-*-gcp` jobs are not run within the `test-e2e` workflow on a **developer branch**. If you would like to invoke these jobs on a certain commit, you must add the `ci-run-allgcp` label to your pull request on github.

**On branch `main` and `release/rc` branches, these jobs always run without needing to set the `ci-run-allgcp` label.**

The following test suites currently run only on hardware. They do not run successfully with `make slurmcluster` and thus are not executed via GCP as part of the CI/CD gate:
- `test-e2e-slurm-gpu`: Test is skipped because the compute instance that the tests run on do not have any GPUs.
Expand Down

0 comments on commit e69d428

Please sign in to comment.