test: add runner watch mode isolation tests #54888

pmarchini · 2024-09-11T14:40:20Z

This PR extends the existing tests for the Node.js test runner's --watch mode by adding support for the --experimental-test-isolation option.

@cjihrig I was unable to find a test that covers this combination.
I also noticed that the should run new tests when a new file is created in the watched directory test behaves differently depending on the isolation modes.
What do you think?

nodejs-github-bot · 2024-09-11T14:40:26Z

Review requested:

@nodejs/test_runner

cjihrig · 2024-09-11T14:53:51Z

What do you think?

Without having investigated the discrepancy, I'd say they should behave the same.

codecov · 2024-09-11T16:41:05Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 88.06%. Comparing base (6db320a) to head (0eab852).
Report is 75 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #54888      +/-   ##
==========================================
- Coverage   88.07%   88.06%   -0.01%     
==========================================
  Files         651      651              
  Lines      183538   183538              
  Branches    35861    35864       +3     
==========================================
- Hits       161652   161635      -17     
+ Misses      15145    15140       -5     
- Partials     6741     6763      +22

see 34 files with indirect coverage changes

cjihrig

Mostly LGTM. Two small comments. Have you been able to identify why the one test fails without isolation?

cjihrig · 2024-09-13T14:01:11Z

test/parallel/test-runner-watch-mode.mjs

+        await testWatch({ fileToUpdate: 'test.js', action: 'rename', isolation });
+      });
+
+      it('should not throw when delete a watched test file', { skip: common.isAIX }, async () => {


We already skip the entire file at the very top when running on AIX.

cjihrig · 2024-09-13T14:02:21Z

test/parallel/test-runner-watch-mode.mjs

+        await testWatch({ fileToUpdate: 'test.js', action: 'delete', isolation });
+      });
+
+      if (isolation !== 'none') {


Instead of an if statement, we could make it a TODO or SKIP test. TODO will run but a failure will not fail the test suite. SKIP won't run at all.

I left the comment because I was planning to fix the issue in this PR, but I haven't had the time in the last two days. I'm going to address your comments, and then I'll investigate the issue ASAP.
(Thanks, as always, for the feedback 😁)

updated! 🚀

pmarchini · 2024-09-14T19:55:46Z

Mostly LGTM. Two small comments. Have you been able to identify why the one test fails without isolation?

Hey @cjihrig, I just had the chance to take a look, and every time that a file changes (rename, delete, update), if we are in isolation none, then restartTestFile restarts all the tests without filtering based on the "added, updated, removed" logic.

I'll try to address this issue in a separate PR if you agree.

cjihrig · 2024-09-14T20:07:15Z

That makes sense. Thanks for digging in. I'd say if you want to try working on it and think you can improve the experience then go for it! If not, it's still good to have these tests 😄

pmarchini · 2024-09-14T20:11:20Z

That makes sense. Thanks for digging in. I'd say if you want to try working on it and think you can improve the experience then go for it! If not, it's still good to have these tests 😄

I'll definitely give it a try 😁

nodejs-github-bot · 2024-09-14T20:12:05Z

CI: https://ci.nodejs.org/job/node-test-pull-request/62436/

nodejs-github-bot · 2024-09-15T15:43:52Z

CI: https://ci.nodejs.org/job/node-test-pull-request/62457/

nodejs-github-bot · 2024-09-17T00:43:41Z

CI: https://ci.nodejs.org/job/node-test-pull-request/62489/ 💛

mcollina

lgtm

nodejs-github-bot · 2024-09-20T13:21:18Z

Commit Queue failed

- Loading data for nodejs/node/pull/54888
✔  Done loading data for nodejs/node/pull/54888
----------------------------------- PR info ------------------------------------
Title      test: add runner watch mode isolation tests (#54888)
Author     Pietro Marchini <[email protected]> (@pmarchini)
Branch     pmarchini:test/test-runner-watch-mode-isolation -> nodejs:main
Labels     test, author ready
Commits    3
 - test: add runner watch mode isolation tests
 - test: remove unused skip
 - test: add todo test
Committers 1
 - Pietro Marchini <[email protected]>
PR-URL: https://github.com/nodejs/node/pull/54888
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
------------------------------ Generated metadata ------------------------------
PR-URL: https://github.com/nodejs/node/pull/54888
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
--------------------------------------------------------------------------------
   ℹ  This PR was created on Wed, 11 Sep 2024 14:40:20 GMT
   ✔  Approvals: 2
   ✔  - Colin Ihrig (@cjihrig): https://github.com/nodejs/node/pull/54888#pullrequestreview-2308136143
   ✔  - Matteo Collina (@mcollina) (TSC): https://github.com/nodejs/node/pull/54888#pullrequestreview-2318229689
   ✔  Last GitHub CI successful
   ℹ  Last Full PR CI on 2024-09-17T00:43:41Z: https://ci.nodejs.org/job/node-test-pull-request/62489/
- Querying data for job/node-test-pull-request/62489/
   ✔  Last Jenkins CI successful
--------------------------------------------------------------------------------
   ✔  No git cherry-pick in progress
   ✔  No git am in progress
   ✔  No git rebase in progress
--------------------------------------------------------------------------------
- Bringing origin/main up to date...
From https://github.com/nodejs/node
 * branch                  main       -> FETCH_HEAD
   99433a2d7a..f79fd03f41  main       -> origin/main
✔  origin/main is now up-to-date
main is out of sync with origin/main. Mismatched commits:
 - e84fdebd7b test_runner: add support for coverage via run()
 - f79fd03f41 test_runner: add support for coverage via run()
--------------------------------------------------------------------------------
HEAD is now at f79fd03f41 test_runner: add support for coverage via run()
   ✔  Reset to origin/main
- Downloading patch for 54888
From https://github.com/nodejs/node
 * branch                  refs/pull/54888/merge -> FETCH_HEAD
✔  Fetched commits as f79fd03f4161..0eab8529cfdc
--------------------------------------------------------------------------------
[main 2096064b13] test: add runner watch mode isolation tests
 Author: Pietro Marchini <[email protected]>
 Date: Sun Sep 15 10:46:24 2024 +0200
 1 file changed, 27 insertions(+), 18 deletions(-)
[main 69a35d3c4d] test: remove unused skip
 Author: Pietro Marchini <[email protected]>
 Date: Sun Sep 15 10:46:36 2024 +0200
 1 file changed, 3 insertions(+), 3 deletions(-)
[main 82759fbd55] test: add todo test
 Author: Pietro Marchini <[email protected]>
 Date: Sun Sep 15 10:46:36 2024 +0200
 1 file changed, 7 insertions(+), 6 deletions(-)
   ✔  Patches applied
There are 3 commits in the PR. Attempting autorebase.
Rebasing (2/6)
Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

test: add runner watch mode isolation tests
PR-URL: #54888

Reviewed-By: Colin Ihrig <[email protected]>

Reviewed-By: Matteo Collina <[email protected]>
[detached HEAD ea69604a6f] test: add runner watch mode isolation tests

Author: Pietro Marchini <[email protected]>

Date: Sun Sep 15 10:46:24 2024 +0200

1 file changed, 27 insertions(+), 18 deletions(-)

Rebasing (3/6)

Rebasing (4/6)
Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

test: remove unused skip
PR-URL: #54888

Reviewed-By: Colin Ihrig <[email protected]>

Reviewed-By: Matteo Collina <[email protected]>
[detached HEAD 5dc478195c] test: remove unused skip

Author: Pietro Marchini <[email protected]>

Date: Sun Sep 15 10:46:36 2024 +0200

1 file changed, 3 insertions(+), 3 deletions(-)

Rebasing (5/6)

Rebasing (6/6)
Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

test: add todo test
PR-URL: #54888

Reviewed-By: Colin Ihrig <[email protected]>

Reviewed-By: Matteo Collina <[email protected]>
[detached HEAD 72ccb24833] test: add todo test

Author: Pietro Marchini <[email protected]>

Date: Sun Sep 15 10:46:36 2024 +0200

1 file changed, 7 insertions(+), 6 deletions(-)
Successfully rebased and updated refs/heads/main.
ℹ  Add commit-queue-squash label to land the PR as one commit, or commit-queue-rebase to land as separate commits.

https://github.com/nodejs/node/actions/runs/10960018630

RedYetiDev · 2024-09-20T13:23:28Z

(Ignore my last [deleted] comment) I've added commit-queue-squash to this PR, it should be good to commit now.

nodejs-github-bot · 2024-09-20T13:41:41Z

Landed in e35299a

PR-URL: #54888 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Matteo Collina <[email protected]>

PR-URL: nodejs#54888 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Matteo Collina <[email protected]>

nodejs-github-bot added needs-ci PRs that need a full CI run. test Issues and PRs related to the tests. labels Sep 11, 2024

cjihrig reviewed Sep 13, 2024

View reviewed changes

cjihrig approved these changes Sep 14, 2024

View reviewed changes

cjihrig added the request-ci Add this label to start a Jenkins CI on a PR. label Sep 14, 2024

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Sep 14, 2024

pmarchini added 3 commits September 15, 2024 10:46

test: add runner watch mode isolation tests

37654fe

test: remove unused skip

dbb8292

test: add todo test

0eab852

pmarchini force-pushed the test/test-runner-watch-mode-isolation branch from 63a09ed to 0eab852 Compare September 15, 2024 08:49

mcollina added the request-ci Add this label to start a Jenkins CI on a PR. label Sep 15, 2024

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Sep 15, 2024

cjihrig approved these changes Sep 17, 2024

View reviewed changes

cjihrig added author ready PRs that have at least one approval, no pending requests for changes, and a CI started. and removed needs-ci PRs that need a full CI run. labels Sep 17, 2024

mcollina approved these changes Sep 20, 2024

View reviewed changes

mcollina added the commit-queue Add this label to land a pull request using GitHub Actions. label Sep 20, 2024

nodejs-github-bot added commit-queue-failed An error occurred while landing this pull request using GitHub Actions. and removed commit-queue Add this label to land a pull request using GitHub Actions. labels Sep 20, 2024

RedYetiDev removed the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Sep 20, 2024

cjihrig added the commit-queue Add this label to land a pull request using GitHub Actions. label Sep 20, 2024

nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Sep 20, 2024

nodejs-github-bot merged commit e35299a into nodejs:main Sep 20, 2024
74 checks passed

targos pushed a commit that referenced this pull request Oct 4, 2024

test: add runner watch mode isolation tests

98ff615

PR-URL: #54888 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Matteo Collina <[email protected]>

aduh95 mentioned this pull request Oct 9, 2024

V22.10.0 proposal #55343

Merged

louwers pushed a commit to louwers/node that referenced this pull request Nov 2, 2024

test: add runner watch mode isolation tests

ce7d45f

PR-URL: nodejs#54888 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Matteo Collina <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: add runner watch mode isolation tests #54888

test: add runner watch mode isolation tests #54888

pmarchini commented Sep 11, 2024

nodejs-github-bot commented Sep 11, 2024

cjihrig commented Sep 11, 2024

codecov bot commented Sep 11, 2024 •

edited

Loading

cjihrig left a comment

cjihrig Sep 13, 2024

pmarchini Sep 14, 2024

cjihrig Sep 13, 2024

pmarchini Sep 13, 2024

pmarchini Sep 14, 2024

pmarchini commented Sep 14, 2024

cjihrig commented Sep 14, 2024

pmarchini commented Sep 14, 2024

nodejs-github-bot commented Sep 14, 2024

nodejs-github-bot commented Sep 15, 2024

nodejs-github-bot commented Sep 17, 2024 •

edited by jakecastelli

Loading

mcollina left a comment

nodejs-github-bot commented Sep 20, 2024

PR-URL: #54888
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>

PR-URL: #54888
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>

PR-URL: #54888
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>

Successfully rebased and updated refs/heads/main.

RedYetiDev commented Sep 20, 2024

nodejs-github-bot commented Sep 20, 2024

test: add runner watch mode isolation tests #54888

test: add runner watch mode isolation tests #54888

Conversation

pmarchini commented Sep 11, 2024

nodejs-github-bot commented Sep 11, 2024

cjihrig commented Sep 11, 2024

codecov bot commented Sep 11, 2024 • edited Loading

Codecov Report

cjihrig left a comment

Choose a reason for hiding this comment

cjihrig Sep 13, 2024

Choose a reason for hiding this comment

pmarchini Sep 14, 2024

Choose a reason for hiding this comment

cjihrig Sep 13, 2024

Choose a reason for hiding this comment

pmarchini Sep 13, 2024

Choose a reason for hiding this comment

pmarchini Sep 14, 2024

Choose a reason for hiding this comment

pmarchini commented Sep 14, 2024

cjihrig commented Sep 14, 2024

pmarchini commented Sep 14, 2024

nodejs-github-bot commented Sep 14, 2024

nodejs-github-bot commented Sep 15, 2024

nodejs-github-bot commented Sep 17, 2024 • edited by jakecastelli Loading

mcollina left a comment

Choose a reason for hiding this comment

nodejs-github-bot commented Sep 20, 2024

PR-URL: #54888 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Matteo Collina <[email protected]>

PR-URL: #54888 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Matteo Collina <[email protected]>

PR-URL: #54888 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Matteo Collina <[email protected]>

Successfully rebased and updated refs/heads/main.

RedYetiDev commented Sep 20, 2024

nodejs-github-bot commented Sep 20, 2024

codecov bot commented Sep 11, 2024 •

edited

Loading

nodejs-github-bot commented Sep 17, 2024 •

edited by jakecastelli

Loading

PR-URL: #54888
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>

PR-URL: #54888
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>

PR-URL: #54888
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>