[CLOSED] 24 failures when running all unit tests #6873
Comments
|
Lately, I have been seeing the same thing on Windows. |
|
I get a ton of errors running integration tests all at once on Windows but I thought that was a known issue. |
|
What I generally see on Mac is that if I run "All", Brackets ends up using lots of memory and getting very slow, which I think starts to cause some tests to timeout. This is most likely because we have memory leaks when we open lots of separate windows in the Integration tests. So when I run unit tests, I generally run the Unit, Performance, and Extensions tabs, then quit and restart Brackets and run Integration, and that seems to usually work fine.
In any case, we should spend some time to investigate what Medium pri to |
|
Most of the failures are |
|
What I have noticed in the SpecRunner on Windows with Task Manager or Process Explorer, when I run a suite of tests (e.g. Unit, Integration) if I (1) "click All" versus (2) "clicking on each spec one at a time", the amount of memory used for (1) is roughly double the amount if memory used for (2).
Obviously, it would be nice to be able to click "All" and use half the memory. Also, none of the memory is released when you close the SpecRunner window -- you have to close Brackets. I also notice that we're using Jasmine version 1, so we should probably upgrade to version 2. The APIs have changed (all |
|
|
|
|
|
Issue #4185 may be relevant, however. If Maybe memory is the issue. Another thing to consider, maybe it is the unit tests occurring before these unit tests. If some unit tests aren't cleaning up after themselves, leaving behind functional side effects of some sort, it might affect these unit tests. Smaller subsets of unit test runs seem to avoid this issue. Maybe the problem unit tests aren't in the smaller subsets. |
|
Update on this issue. As of Brackets sprint 40 development build 0.40.0-0 (master 4d9502f14), the number of unit test failures is now up to 35. There are still 14 failures of JavaScript Code Hinting but now there are 21 failures with Url Code Hinting. Note that these two sets of test suites are fired off very close to the end of the run after almost all of the other test suites have completed. Might still be a memory problem. Might be a functional side effect from another test suite that doesn't clean up after itself. |
|
The general feeling on the team is that we don't have enough unit tests and we need to add more, so until we find a major breakthrough in releasing memory after usage, I think this problem is going to continue to grow. The workaround for now is to not run All/All tests, but to run All from each category (Unit, Integration, etc) at a time. But, even running a single category (most notably Integration) can hit the same problem, so I submitted pull request adobe/brackets#7758 to split off the Live Development tests into a new category. Note that this PR also moves some "integration" type tests out of the Unit category to make those tests run faster. |
|
We keep adding unit tests, so it's impossible as far as I know to run All/All unit tests at this point. I created a wiki page with Tips and Tricks for running Brackets unit tests to help people work around this issue. |
|
I'm closing this issue as resolved by documentation per |
Friday Apr 25, 2014 at 00:16 GMT
Originally opened as adobe/brackets#7637
OS: Mac OSX Mavericks
Brackets Version: sprint 39 development build 0.39.0-0 (test 4d968b2c2).
Repro Steps:
Alltab and click onAlltests.Expected Results:
All unit tests should pass.
Observed Results:
There are 24 total failures every time. Ten failures in the URL Code Hinting and fourteen failures in the JavaScript Code Hinting.
Other Notes:
If you run the URL Code Hinting and JavaScript Code Hinting suites individually, they all pass. If you select
Allunit tests from theExtensionstab, which also includes both these test suites, all tests pass.The text was updated successfully, but these errors were encountered: