🚀 Feature: Add separate category for "skipped" tests besides "pending"

[binarykitchen]

Summary example

  27 passing (1m)
  2 pending
  1 failing

You know, you can skip tests with .skip() and these appear in the summary at the end under the pending category which is IMO not correct. Pending != Skipped. Pending can be interpreted as "not completed yet" whereas "skipped" can mean, code is not complete yet or is breaking for whatever reason.

Any chance skipped tests can be summarised under a new group, i.E.

  27 passing (1m)
  2 skipped
  1 failing

[danielstjules]

The final output could be:

  27 passing (1m)
  1 skipped
  1 pending
  1 failing

Introducing a new state for specs would be a major breaking change, so it'd have to target 3.0.0. I don't think this would be likely to get implemented in the near future. However, I do like the idea of distinguishing between the two. :)

[bananu7]

I'd like to voice that I think it's a very important feature. Could anyone with relevant knowledge elaborate on the "breaking changes" mentioned by @danielstjules? Is this something that could be reasonably contributed by an outsider?

Additionally, I could imagine specifying a reason for the test being skipped to be quite useful as well.

[mekdev]

+1 I think we need a skipped status, currently we have to triage all "pending" tests and see when one is actually pending because of a failed hook or actually skipped.

[ScottFreeCode]

In principle I'm generally for being able to differentiate pending tests from tests that are skipped due to hook failures. (Something breaking in the test itself, on the other hand, should typically be a test failure rather than skipping.) That being said...

• It isn't commonly reported in the JS testing ecosystem in general -- see here for a survey at least as of a couple years ago and the resulting spec to this day does not have a separate "pending" status, only a "skipped" status. (If it matters, Mocha doesn't currently implement this spec to my knowledge, but there was interest in updating it to do so.)

Could anyone with relevant knowledge elaborate on the "breaking changes"...

There are definitely custom (ie third-party) Mocha reporters relying on the current set of statuses. Additionally, there could (although it's both a hairier situation and a less likely one) be users or third-party tools parsing the reporter output that are likewise relying on the current set of statuses.

Additionally, I could imagine specifying a reason for the test being skipped to be quite useful as well.

I don't have the link/number handy, but we do have an issue somewhere requesting this. What I'd like to point out here, however, is that this could potentially be an alternate resolution of the difference between pending and skipped: if xit/it.skip, this.skip, it("<name with no callback>"), describe.skip and being skipped due to hook failure were all given different default "reason" messages for the case where the user doesn't specify the reason, it would become pretty easy to differentiate them (at least looking at the individual tests even if the summary is the same) even with them all being "pending". Thus, if we want to implement skip/pending-reason messages anyway (which I believe we do -- or at any rate, I know I've had real use cases in work projects!), we may not need a new status in addition.

[beck]

Another use-case for skip: Writing tests for polyfill and testing across browsers.

If polyfill.foo is natively supported, then the test for .foo is skipped. If all the tests are skipped, then that browser can be removed from the list of environments where polyfill applies.

Output that says pending is confusing.

[onesandzeroes-maker]

Would anyone know a workaround for this issue?

[JoshuaKGoldberg]

This would be a somewhat significant breaking change. I'm not against it, just per #5027 we're trying not to make any big changes any time soon. Marking as status: blocked on us getting through a major version or two first. Once we on the new maintainer team are more familiar with Mocha, we can revisit this.