Runtime errors preventing test from being run don't count test as failed

[bardt]

Given a code like this:

// ../someModule.js does not exist!
import SomeModule from '../someModule';

I expect to get final report as following:

1 test failed, 70 tests passed (71 total in 18 test suites, run time 16.805s)

But instead I get:

70 tests passed (70 total in 18 test suites, run time 16.805s)

Explanation

If import fails (wrong path or something) Jest output this test as Failed followed by stack trace, but doesn't count it this way in final report. So, I get 70 tests passed (70 total in 18 test suites, run time 16.805s) and command line process exits with code 0. This is probably not an issue for local development, because Fail message is pretty obvious, but my CI thinks that everything passed because of code 0.

[cpojer]

Hey! Are you using jest 0.7.0? This was fixed recently.

[bardt]

Yeah, my bad. New version works just fine!

[btipling]

This is still broken for me with 0.7.1 and even when trying latest at 0.7.2. I get exit status of 0 when there are run time errors in my jest tests.

[btipling]

bash-3.2$ npm test

> [email protected] test /Users/bjorn/dev/MW
> BABEL_JEST_STAGE=0 ./node_modules/.bin/jest --captureExceptions

Using Jest CLI v0.7.2
 FAIL  src/__tests__/components/TestPostHandler.js
● Runtime Error
Error: /Users/bjorn/dev/MW/src/__tests__/components/TestPostHandler.js: /Users/bjorn/dev/MW/src/shared/components/PostHandler.js: /Users/bjorn/dev/MW/src/shared/actions/CommentActions.js: Cannot find module 'lodash.custom' from '/Users/bjorn/dev/MW/src/shared/actions/CommentActions.js'
bash-3.2$ echo $?
0

[cpojer]

You are right, I fixed this in the haste2 PR, see #599, and more specifically f518a92

I should probably cherry pick this into 0.8.0. node-haste2 will likely be 0.9.0.

[btipling]

Ok good to know! Thank you, was about to open a new issue.

[cpojer]

Sorry, this entire issue is confusing, there were a bunch of other small counting issues when we changed to test suites vs. tests and I should have been more thorough during code review.

[btipling]

No problem, although turns out that I get an exit status of 0 for any kind of test failure. Looking at the PR and f518a92 it wasn't clear to me if that was also part of that. I purposefully made a test fail and got this:

 FAIL  src/__tests__/components/TestPostHandler.js (7.705s)
● PostHandler › it advances position when swiping forward
  - Expected: 2 toEqual: 200
        at Spec.<anonymous> (src/__tests__/components/TestPostHandler.js:104:49)
1 test failed, 4 tests passed (5 total in 1 test suite, run time 8.148s)
bash-3.2$ echo $?
0

Just an FYI. Thanks.

[cpojer]

There we go: #618

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.