feat: support test retries

[kuceb]

• fix Support mocha test retries / retry test failures #1313
• fix Cypress counts overwritten screenshots as unique in run results #8079
• fix ReporterStats show more tests passing than cypress stats #7730 buffer mocha pass event to prevent reporters from showing extra passed tests on hook failures

User facing changelog

• You can now retry tests a specific number of times by specifying retries from your configuration.

Additional details

How has the user experience changed?

warning message when incompatible cypress-plugin-retries is installed:

Interactive test with 2 retries (all failed)

Test during second attempt (1 previous failed):

Test after successful retry (1 previous failed):

Test after expanding previous failed attempt:

Terminal reporter (runMode) output:

Set retries on only some tests/suites:

it('retry only this test', {
    retries: 2
  }, () => {...}
})

describe('retry only tests in this suite', {
    retries: 2
  }, () => {...}

To get the current retry number from inside a running test use cy.state('test').currentRetry()

You can check if test retries feature exists by looking at Cypress.getTestRetries. Here is a trick to skip a suite of tests if it is not

const describeOrSkip = Cypress.getTestRetries ? describe : describe.skip
describeOrSkip('Test', () => {

PR Tasks

• Have tests been added/updated?
• Has the original issue been tagged with a release in ZenHub?
• Have API changes been updated in the type definitions?
• Have new configuration options been added to the cypress.schema.json?
• bug fix 'runnable:after:run:async' / screenshots in hooks after pass
  fix runnable:run:async only fires once for each hook #3744
• add .snapshot() function for agents (spies and stubs) to reduce memory
  fix allow disable of DOM snapshots for stubs and spies #3849
• allow test runner to be ran in an isolated window for ui tests
• isolated test runner specs, snapshot tests for emitted events
• add prevAttempts onto the test runnable instances in driver and server-reporter
• reporter ui changes to display previous test attempts
• allow retries configuration value
• send _currentRetry and _retries runnable props in events (to recreate runnables)
• update snapshot tests for module API and dashboard results
• fix failing specs
• pass along 'retry' event from mocha retry event is only fired in mocha 6+
• fix merge conflicts from develop
• disable retries by default in open mode, support config: enableTestRetriesInOpenMode: false (default)
• rename retries config to numTestRetries
• throw custom error when user sets retry via this.retries or on hook
• screenshot filenames on retry, add tests (address inconsistent screenshot filenames kuceb/cypress-plugin-retries#10)
• wait for Dashboard API to be updated to accept new payload
• upgrade chai devDep in packages/server, fix resulting errors in tests
• fix flaky cli/unit test
• seemingly broken css after merge
• change reties configuration to take one object instead of 2 separate options
• emit a test retry event (for plugin authors, custom reporters)
• wait for PR: isolated runner testing isolated runner changes #6799
• wait for PR: per-test configuration feat: test config overrides #5346
• update default mocha reporter to show retry information
  • while test is running
  • in final report
• log retries as attempt number in server/reporter e.g. (Attempt n of m) - only in spec reporter (default reporter)
• buffer mocha "pass" event to prevent passing w/failure in afterEach hook (open issue since changes output even with retries disabled)
  • maybe: chunk PR to buffer mocha pass event (prevents different stats for mocha stats vs cypress stats)
• update default retries to be {runMode:2, openMode:0} [BREAKING CHANGE]
• re-run before hooks as well as beforeEach, afterEach, and test body
• finalize new breaking schema of postRunResults
  • wait for dashboard to accept new schema
• update runRestults test.error.stack to only have stacktrace lines
• make sure retries are disabled on test/e2e projects and cypress tests for runner/driver

Docs / changes

• fix duplicate screenshot entries when using cy.screenshot in a test retry

• fix cannot access scrollTop of null error in test.tsx breaking e2e/navigation_spec

• Docs PR: TBD

• fix css for attempt item border-left

• hover styles on individual attempt blocks

• setup screenshot diffing for retries ui states

• fix bug marking last test passed instread of pending

• add warning message when project has cypress-plugin-retries installed

  • move docs link below

• add more comments

• make sure subsequent afterEach hooks do not run after failing one

• only run subsequent hooks of the same nesting level on hook failures

• incorrect error thrown with inline retry config

• closes Try creating a CircleCI job that monitors other jobs #8054 (better Percy finalize job)

• write tests for cy.screenshot w/ multiple attempts (think about fixing flake)

TODO:

• demo isolated runner screenshot diffing
• add screenshots for other reporter ui states

non-blocking

• color dots beside test attempt collapsibles

• you can set retries in cypress.json to either number or { openMode: number, runMode: number } to set different values in run mode vs open mode

  • to enable retries only in runMode (CI), you'd use:

{
  "retries": {
     "runMode": 2,
     "openMode": 0
  }
}

Migrating from retries plugin:

• remove plugin-retries devDependencies and plugin-retries code in support files
• remove usage of Cypress.currentTest in favor of test config overrides it('test title', { retries: 2 }, () => {...})
• remove usage of this.retries(n) (not supported)
• Use Cypress.config('retries') instead of Cypress.env('RETRIES') (or see cypress.json instructions above

TR-58

[kuceb]

pls review these error messages

[kuceb]

pls review this error message

[lukeapage]

Will this feature cover this scenario: kuceb/cypress-plugin-retries#10

I can work-around it (I think) by listening to screenshot events, recording the filenames and then if I can detect a retry in the before each (same spec name?) I can delete the files before the test starts.. but I'd like to know if I should implement a work-around or if you plan to ship this working with this initial release, I'll just wait for it.

[kuceb]

@lukeapage yes, thank you for that. I'm adding that task here, the final attempt should produce the same artifacts as a normal passing test.

Edit: after considering we would have to rename files on retry, I think we will simply increment the filename with (attempt 2) and so on for retried attempts

[lukeapage]

That’s a shame. Any reason why? I can grep but the logic is a bit more complicated than ignoring files based on a regex (as we can with failed). I might have a go at deleting the files myself then.

And will the logic work like this..

Testname-1
Testname-2
Testname (failed)
Testname-1 (attempt 2)
Testname-2 (attempt 2)
Testname-3 (attempt 2)

[lukeapage]

And if I do delete the files myself, will you still add on (attempt 2) on the second attempt?

[kuceb]

@lukeapage so in plugins you will be able to listen to after:screenshot and rename any screenshot with (attempt ...) to overwrite the original file

but will add (attempt ...) regardless if a file exists

[ro-savage]

Awesome work on this @bkucera

Disappointed this didn't get into the latest release.

We often get random failures, that a re-try will fix. It would be really nice if we could auto-retry on failures.

[lukeapage]

@ro-savage you can use https://github.com/Bkucera/cypress-plugin-retries

[Cleudi]

Will information about retried tests show up in the statistics? This would be really important in order to not hide unstable tests...

[psurana0329]

Do we have a time line for this to be released???

[brian-mann]

this can already be done by cy.stub(...).log(false)

[brian-mann]

expect(check('foo/index.js', 'foo')).to.be.false

[jennifer-shehane]

I'm sure you already know this, but I was pulling down the repo just to write the docs. With test retries turned off - the assertion failures print twice.

[jennifer-shehane]

I updated the cypress.schema.json with the configuration values, but it looks like you have a line item on here to maybe change the configuration options?

I also could not get the test retries actually working from this branch.

Overall, I don't really see this PR in a state that is ready to document as far as I can tell, but am willing to schedule a meeting to discuss documentation changes as necessary.

[lukeapage]

@bkucera happy to see this land. What about the failed screenshot from previous attempts? Is that skipped when there is another try remaining? or if I don't want that output do I have to delete them after the test has completed, if the test is successful?

[jennifer-shehane]

@lukeapage The full documentation for test retries can be found here: cypress-io/cypress-documentation#2925

All screenshots for every failed attempt are kept (if you have screenshots on failure on) and named accordingly (Attempt 1). If you wanted these deleted, you'd have to manually delete.