Proposal: Cypress should preprocess all of the spec files once

[paulfalgout]

Current behavior:

In v3 for each spec file, the preprocessor is run which has costly overhead.

Desired behavior:

In v2 since all the tests were run at once there was a single larger preprocessor event. For large test suites running the preprocessor once is significantly faster overall, which is much more noticeable in CI environments than it was locally. If possible, I'd love to see the preprocessor run once and the test splitting to happen post processing rather than processing each spec during the run.

In the meantime I've resorted to serializing the tests myself when running on CI via:

describe('Integration Tests', function() {
    const req = require.context('./', true, /^(.*\.(js$))[^.]*$/i);
    req.keys().forEach(function(key) {
        req(key);
    });
});

This certainly reduces some of the features of the v3 dashboard, but reduces our run time by over half. I don't think we could upgrade to v3 without it.

Versions

Cypress v3, Travis-CI, Chrome/Chromium

[paulfalgout]

related: #1879

[paulfalgout]

Ok some logs/evidence:
Travis serialize run completed in 8 min 19 sec with 2 spec files (one of which imports 44 other spec files).
cypress:webpack get occurred 3 times: ci.js - 35s support/index.js - 25s unit.js - 13s
cypress:webpack get overhead ~ 73s
https://gist.github.com/paulfalgout/8693cbb8fab358f3cb95067fb2e95056

Normal v3 run completed in 18 min 28 sec with 45 spec files
cypress:webpack get occurred 46 times: first spec file - 34s, support/index.js - 27ms, 44 additional spec files - 12-13s
cypress:webpack get overhead ~590s

https://gist.github.com/paulfalgout/738a1a84d2eda1e701fcf87fa9f1fee8

However I think maybe the log names were misleading a bit.. looking at the cypress:webpack close it looks like those just reflect the test time and there's no closing anyhow since there's no watching going on.. so the logs are just logging what happened up until the event? So I initially thought the cost was webpack and the complexity of our build.. however I suspect the cypress:webpack get reflects events up until, but not including the preprocessor in which case I'm not sure what's adding the overhead, but hopefully it could be combined but still maintain the test run separation?

[paulfalgout]

Might be what #2184 is doing

[paulfalgout]

related #1915

[topheman]

I ended up doing something pretty similar: topheman/npm-registry-browser@4f7bcbe - though, I statically import the spec files. Your use of require.context is interesting on this use case.

[paulfalgout]

I'm not doing this anymore. Other perf improvements plus various parallel functions makes this far less likely to be a thing I think.