Reporting Async Test Fails

[IPWright83]

Issue description or question

When running an async test, it seems that Wallaby isn't correctly reporting the failure. I don't know if this is a jest/jasmine specific issue or not. Looking at the cases below:

it("Wallaby Reporting Sync Fails", () => {
    expect(true).toBe(false);
});

This correctly reports the error as

Error: expect(received).toBe(expected)

Expected value to be (using ===):
false
Received:
true

If I wrap this up as an Async test however:

it("Wallaby Not Reporting Async Fails", (done) => {
   setTimeout(() => {
      expect(true).toBe(false);
      done();
   }, 100);
});

then I get a much less helpful error, which makes tracking the actual problem down more difficult. I'm currently having to console.log my actual results to diagnose faults.

Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.

Wallaby.js configuration file

module.exports = function (wallaby) {
    process.env.NODE_ENV = 'wallaby';
    process.env.BABEL_DISABLE_CACHE=1;

    return {
        files: [
            { pattern: "./tests/**/*.snap", load: false, instrument: false },
            { pattern: 'node_modules/babel-polyfill/dist/polyfill.js', instrument: false },
            { pattern: 'src/**/*.css', load: false },
            { pattern: 'src/**/*.js', load: false },
            { pattern: 'tests/utils/*.js', load: false }
        ],
        tests: [
            // Exclude specific tests, that won't run in Jest (since they require a browserr object)
            { pattern: './tests/d3/utils/spark-points.test.js', ignore: true },
            { pattern: './tests/d3/utils/tooltips.test.js', ignore: true },
            // Load all other tests
            { pattern: './tests/**/*.test.js' }
        ],
        filesWithNoCoverageCalculated: ['src/**/index.js', 'tests/**/*.js'],
        hints: {
            ignoreCoverage: /ignore coverage/
        },
        env: {
            type: "node",
            runner: "node"
        },
        testFramework: 'jest',
        compilers: {
            '**/*.js': wallaby.compilers.babel({ babel: require('babel-core') })
        },
        setup: (target) => {
           // process.env.BABEL_ENV = 'test';
           if (!global._babelPolyfill) {
               require("./node_modules/babel-polyfill/dist/polyfill.js");
           }
           target.testFramework.configure({
               moduleNameMapper: {
                   '.*\.(less|css)$': require('path').join(target.localProjectDir, '/tests/preProcessors/jest.js')
               },
           });
       },
    };
};

Code editor or IDE name and version

Sublime Text v3

OS name and version

Windows 10 x64

[ArtemGovorov]

What happens if you run the async test via jest CLI directly (without wallaby)?

[ArtemGovorov]

Just tried your test with the plain jest CLI without wallaby,

it("Not Reporting Async Fails", (done) => {
   setTimeout(() => {
      expect(true).toBe(false);
      done();
   }, 100);
});

and I'm getting the same results:

 FAIL  src/__tests__/component.spec.js (5.877s)
  ● Not Reporting Async Fails

    Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
      
      at Timeout.callback [as _onTimeout] (node_modules/jsdom/lib/jsdom/browser/Window.js:480:19)
      at ontimeout (timers.js:380:14)
      at tryOnTimeout (timers.js:244:5)
      at Timer.listOnTimeout (timers.js:214:5)

  ✕ Not Reporting Async Fails (5006ms)

Test Suites: 1 failed, 1 total
Tests:       1 failed, 1 total
Snapshots:   0 total
Time:        6.716s
Ran all test suites.

So it looks like it's just the way jest works.

[IPWright83]

Thanks Artem - Jest is just taking an age to run via the CLI for me, otherwise I'd have posted the result. That's a bit of a pain, guess a bug that needs raising with Jest. Is that something you want me to do?

[ArtemGovorov]

Looks like there's an opened issue for it the Jest repo: jestjs/jest#2059. There seem to be some promise/wrapper based workarounds suggested.

[ArtemGovorov]

Also this one jestjs/jest#2685