build: run flaky tests in Travis #27158
Conversation
Skipping flaky tests in CI is an anti-pattern that should be avoided, as we do in our ownCI. Failing flaky tests don’t need to be blockers for a green CI result, but they should be run and reported *somehow*.
nodejs-github-bot
added
the
build
Trott
requested review from
richardlau,
lpinca,
refack and
BridgeAR
and removed request for
richardlau
|
IMO the travis run should be a minimal sanity test that returns results ASAP. For me test coverage is a non-goal. I'd go as far as advocating for skipping more tests, not less. ATM a typical Travis run takes a minimum of 12.5 minutes: |
Interesting. For me, finishing quickly in Travis is a non-goal. (Finishing quickly for local runs, however, is.) Making Travis as useful as possible is a goal, and that means not skipping tests we don't need to skip. |
|
For example this PR, 15 minutes after submission, our Lite-CI finished, and the Travis job hasn't even started |
I just don't see that as a problem, to be honest. If tests aren't working in a Travis environment, they're probably not going to work for some users in some environments. So I want to know. |
|
Practically speaking this PR only affects a small number of tests. e.g. for #27164 our CI ran 2615 tests (https://ci.nodejs.org/job/node-test-commit-linuxone/nodes=rhel72-s390x/12353/) while the Travis run ran 2610 tests (https://travis-ci.com/nodejs/node/jobs/191674829#L1063). Five tests isn't going to make a huge difference to the overall running time. |
Trott
added
the
author ready
|
Landed in 56354d4 |
Skipping flaky tests in CI is an anti-pattern that should be avoided, as we do in our ownCI. Failing flaky tests don’t need to be blockers for a green CI result, but they should be run and reported *somehow*. PR-URL: nodejs#27158 Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Richard Lau <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Yongsheng Zhang <[email protected]> Reviewed-By: Luigi Pinca <[email protected]>
* skip compilation by using cache * split testing into two jobs * DRY with YAML anchors PR-URL: nodejs#27182 Refs: nodejs#27158 Reviewed-By: Richard Lau <[email protected]> Reviewed-By: Rich Trott <[email protected]>
Continuation of #27015
Skipping flaky tests in CI is an anti-pattern that should be avoided,
as we do in our ownCI. Failing flaky tests don’t need to be blockers
for a green CI result, but they should be run and reported somehow.
Checklist
make -j4 test(UNIX), orvcbuild test(Windows) passes