Error sources occasionally being dropped from karma-mocha-reporter summary

[rsimha]

We on ampproject/amphtml recently started using karma-mocha-reporter as the reporter for our tests on Travis-CI. I'm noticing that some of the time, the failures reported by karma-mocha-reporter are not accompanied by error sources. For example, see https://travis-ci.org/ampproject/amphtml/jobs/256983271

Any idea what's going on here? Is this a bug in how errors are being generated in the tests? Or is the reporter dropping them? Note that the progress indicator being used is super-dots, which does not print errors inline.

FYI, the source code for the failing tests in the example above are here, here, and here.

[4kochi]

Well, this issue is hard to track down. I ran the test locally and there were some errors. And the error sources are all listed. There were cases in the past were the error stack was not correctly reported to the reporter either by karma or the browsers. But was nothing what the reporter could have done.

[rsimha]

@4kochi, thanks. I'll ping this bug if we see this again.

[rsimha]

@4kochi Here's an update.

I encountered several more instances of unexpected error reports from karma-mocha-reporter. In some cases, it reported several alleged failures even though only one test failed. It's possible that this has to do with the fact that we're triggering tests on Travis and running them on Sauce Labs browsers. Either way, I thought you might find this useful to look at.

Here's a bug that describes what we're seeing: ampproject/amphtml#11491 (I had to go use a different reporter for our tests that run on Sauce Labs.)

If you look at the logs in https://travis-ci.org/ampproject/amphtml/jobs/281179276#L694, you'll see that there was only one test failure, and yet, karma-mocha-reporter lists dozens of supposedly failing tests.