tests(ci): use BuildJet for ubuntu jobs #13943

connorjclark · 2022-04-28T21:38:36Z

Default github action runner for ubunutu has 7gb ram and 2 cpu cores. see https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources

These BuildJet runners have 4cpu and 16gb ram

I picked this tier because it matches the price GitHub gives for their machines (although I don't know if our GoogleChrome org even pays that, I think public repos may be free? not sure).

Need to test more, but first look says maybe 2-4 minutes faster?

with buildjet https://github.com/GoogleChrome/lighthouse/actions/runs/2241966717
with github https://github.com/GoogleChrome/lighthouse/actions/runs/2241937334

Also could be we have far less problems with provisioning (github queuing our jobs)

brendankenny · 2022-04-28T21:47:27Z

I picked this tier because it matches the price GitHub gives for their machines (although I don't know if our GoogleChrome org even pays that, I think public repos may be free? not sure).

we fall under the enterprise plan

connorjclark · 2022-04-28T22:02:52Z

package.json

@@ -59,7 +59,7 @@
    "watch": "yarn unit-core --watch",
    "unit:cicoverage": "yarn c8 --all yarn unit:ci",
    "coverage": "yarn unit:cicoverage && c8 report --reporter html",
-    "coverage:smoke": "yarn c8 yarn smoke -j=1 && c8 report --reporter html",
+    "coverage:smoke": "yarn c8 yarn smoke && c8 report --reporter html",


uh, we don't even use this.

uh, we don't even use this.

maybe it's for running locally?

connorjclark · 2022-04-28T22:03:27Z

.github/workflows/smoke.yml

@@ -53,7 +53,7 @@ jobs:
    - run: sudo apt-get install xvfb
    - name: Run smoke tests
      run: |
-        xvfb-run --auto-servernum yarn c8 yarn smoke --debug -j=1 --retries=2 --shard=${{ matrix.smoke-test-shard }}/$SHARD_TOTAL
+        xvfb-run --auto-servernum yarn c8 yarn smoke --debug --retries=2 --shard=${{ matrix.smoke-test-shard }}/$SHARD_TOTAL


we don't have as much cpu contention now that we went from 2->4 cpus, so let's try this

or maybe set it to 4?

(thanks to @adamshiervani for the idea!)

Happy to help! 🚀

we don't have as much cpu contention now that we went from 2->4 cpus, so let's try this

no one ever updated it for GHA either, that's a holdover from travis. It should be fine, because all the perf sensitive ones are set to run serially. The default is 5 jobs at once, I have no idea why. Maybe that was the number of manual batches we used before that refactor. Maybe it's the number that runs ok on a 4 core desktop machine 🤷

connorjclark · 2022-04-28T22:13:04Z

package.json

@@ -37,7 +37,7 @@
    "start": "yarn build-report --standalone && node ./lighthouse-cli/index.js",
    "jest": "node --experimental-vm-modules ./node_modules/jest/bin/jest.js",
    "test": "yarn diff:sample-json && yarn lint --quiet && yarn unit && yarn type-check",
-    "test-bundle": "yarn smoke --runner bundle -j=1 --retries=2",
+    "test-bundle": "yarn smoke --runner bundle --retries=2",


bundle smokes really didn't like this change

https://github.com/GoogleChrome/lighthouse/runs/6219410614?check_suite_focus=true

Yeah, doesn't seem to like it. Maybe you can try 4, like you previously suggested!

I get the same thing locally, so I guess we have to run these tests serially. others seemed fine though.

#9943 (comment)

-j=1 was added at inception of the smokehouse bin: https://github.com/GoogleChrome/lighthouse/pull/9943/files#diff-7ae45ad102eab3b6d7e7896acd08c427a9b25b346470d7bc6507b6481575d519R27

the failures are because we are clearing marky in every run of lighthouse, which is fine but the bundled smokehouse is the only one that actually runs lighthouse in the same process everytime

-j=1 was added at inception of the smokehouse bin: https://github.com/GoogleChrome/lighthouse/pull/9943/files#diff-7ae45ad102eab3b6d7e7896acd08c427a9b25b346470d7bc6507b6481575d519R27

that's the same PR?

the failures are because we are clearing marky in every run of lighthouse, which is fine but the bundled smokehouse is the only one that actually runs lighthouse in the same process everytime

Ah, I didn't look at the logs. That'll be two places it fails then :) I assume that's from the FR changes. Just say no to singletons

hah, I misread and thought that linked to your other comment above....

We could change lighthouse-runners/bundle.js to use a new process for each run (make a new file bundle-entry.js that just takes url, configJson, fraggleRock from argv) then we could remove this restriction. wdyt @brendankenny ? This runner is the odd one out, all the others make a new process for each run.

Yeah, it's kind of annoying in that we should be able to use performance marks (and debug) in a way that doesn't run afoul of sharing a process, but you're right this is the odd one out, so it's not a huge deal.

this might work with node workers? Maybe no real benefit but it could be a single file solution (run the test unless isMainThread, in which case run itself as a worker)

if we're really lazy, sharding works with the bundle runner as well, so can always shard these (and the existing smoke shard count can definitely be reduced when they run so fast on these runners :)

brendankenny · 2022-05-05T19:02:24Z

Should we land this before #13968? I assume we have all the approvals we need?

brendankenny · 2022-05-05T19:07:19Z

.github/workflows/smoke.yml

@@ -53,7 +53,7 @@ jobs:
    - run: sudo apt-get install xvfb
    - name: Run smoke tests
      run: |
-        xvfb-run --auto-servernum yarn c8 yarn smoke --debug -j=1 --retries=2 --shard=${{ matrix.smoke-test-shard }}/$SHARD_TOTAL
+        xvfb-run --auto-servernum yarn c8 yarn smoke --debug --retries=2 --shard=${{ matrix.smoke-test-shard }}/$SHARD_TOTAL


hmm, does coverage work without -j=1? c8 is supposed to sum over child_processes but smokehouse might be too tricky for it

can we even trust the coverage report these days?

the linked coverage: https://codecov.io/gh/GoogleChrome/lighthouse/compare/f2db965f75a4f9d2a84614a3cd5a1466a41e99bc...ea372c7da03ac84e4b398df80123c962be008d34/changes

is saying that all the FR stuff is unused. but we have non-smoke tests that cover that. so...

connorjclark · 2022-05-05T19:45:03Z

Should we land this before #13968?

I was thinking we should do #13968 first so we can get a clearer before/after of timing differences from changing to BuildJet.

I assume we have all the approvals we need?

Yup!

brendankenny · 2022-05-05T20:35:37Z

I was thinking we should do #13968 first so we can get a clearer before/after of timing differences from changing to BuildJet.

cool, though for a completely fair comparison, some of the current -j=1 should maybe be bumped to -j=2 :)

crohr · 2024-01-23T08:51:54Z

I suppose this is no longer needed now that GitHub doubled the vCPU and RAM count of default runners. If you'd like to experiment with much bigger CPU counts, RunsOn might also be an option.

connorjclark added 7 commits April 28, 2022 13:22

wip

e5776a0

wip

d6fdeaa

wip

4498d43

wip

ff82b1a

wip

7948462

wip

73614de

ok

12b743b

connorjclark requested a review from a team as a code owner April 28, 2022 21:38

connorjclark requested review from brendankenny and removed request for a team April 28, 2022 21:38

devtools-bot assigned brendankenny Apr 28, 2022

devtools-bot added the waiting4reviewer label Apr 28, 2022

dont run serial

cacdf8f

connorjclark commented Apr 28, 2022

View reviewed changes

vercel bot deployed to Preview April 28, 2022 22:03 View deployment

connorjclark commented Apr 28, 2022

View reviewed changes

ok

ea372c7

vercel bot deployed to Preview April 28, 2022 22:22 View deployment

connorjclark mentioned this pull request May 2, 2022

tests(smoke): run bundle smokes in a worker #13947

Merged

brendankenny reviewed May 5, 2022

View reviewed changes

brendankenny mentioned this pull request May 5, 2022

tests(ci): shard all smoke tests #13968

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests(ci): use BuildJet for ubuntu jobs #13943

tests(ci): use BuildJet for ubuntu jobs #13943

connorjclark commented Apr 28, 2022

brendankenny commented Apr 28, 2022

connorjclark Apr 28, 2022

brendankenny May 5, 2022

connorjclark Apr 28, 2022

connorjclark Apr 28, 2022

connorjclark Apr 28, 2022

adamshiervani Apr 28, 2022

brendankenny Apr 28, 2022 •

edited

Loading

connorjclark Apr 28, 2022

adamshiervani Apr 28, 2022

connorjclark Apr 28, 2022

brendankenny Apr 28, 2022

connorjclark Apr 28, 2022

brendankenny Apr 28, 2022

connorjclark Apr 28, 2022

connorjclark Apr 29, 2022 •

edited

Loading

brendankenny Apr 29, 2022

brendankenny commented May 5, 2022

brendankenny May 5, 2022

connorjclark May 5, 2022

connorjclark commented May 5, 2022

brendankenny commented May 5, 2022

crohr commented Jan 23, 2024

tests(ci): use BuildJet for ubuntu jobs #13943

Are you sure you want to change the base?

tests(ci): use BuildJet for ubuntu jobs #13943

Conversation

connorjclark commented Apr 28, 2022

brendankenny commented Apr 28, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brendankenny Apr 28, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

connorjclark Apr 29, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brendankenny commented May 5, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

connorjclark commented May 5, 2022

brendankenny commented May 5, 2022

crohr commented Jan 23, 2024

brendankenny Apr 28, 2022 •

edited

Loading

connorjclark Apr 29, 2022 •

edited

Loading