Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FTR support mocha dry run #125915

Merged
merged 9 commits into from
Feb 23, 2022
Merged

Conversation

pheyos
Copy link
Member

@pheyos pheyos commented Feb 17, 2022

Summary

This PR adds support for the mocha CLI flag --dry-run in our functional test execution.

Details

  • This required a mocha upgrade - as far as I can see, the breaking changes for 9.0.0 don't really impact us.
    • The dry-run flag has been introduced with mocha 9.0.0, but I took the chance to update to the latest version that I could find "@types/mocha" for, which is 9.1.0.
  • Added the --dry-run flag to our functional_test_runner and functional_tests scripts and made them pass it through to mocha
  • Updated setupMocha to accept reporter and reporterOptions overrides, used this for dry runs to pass in a json reporter with the output file target/functional-tests/dryRunOutput.json

Why do we want this

  • The dry-run flag allows us to quickly collect all defined tests (active and skipped ones) without actually running them.
  • The generated json file allows follow-up processing via script, e.g. for test coverage overview purposes.

Output file

  • The json ouiput file contains some overall stats as well as lists of all tests, passing tests, pending (skipped) tests and failed tests (failed tests list is empty in this case since we don't actually run the tests), e.g.
    image

  • A test entry in these lists gives us details like the title, full title and file name, e.g.
    image

@pheyos pheyos self-assigned this Feb 17, 2022
@pheyos pheyos added release_note:skip Skip the PR/issue when compiling release notes v7.17.1 v8.0.1 v8.1.0 v8.2.0 labels Feb 17, 2022
@pheyos pheyos marked this pull request as ready for review February 17, 2022 11:43
@pheyos pheyos requested a review from a team as a code owner February 17, 2022 11:43
@pheyos pheyos requested a review from spalger February 17, 2022 11:43
@spalger
Copy link
Contributor

spalger commented Feb 17, 2022

Interesting, this seems more comprehensive than what we added with the --test-stats flag, I'll take a look today

@pheyos
Copy link
Member Author

pheyos commented Feb 21, 2022

@elasticmachine merge upstream

@pheyos
Copy link
Member Author

pheyos commented Feb 22, 2022

@elasticmachine merge upstream

@kibanamachine
Copy link
Contributor

merge conflict between base and head

Copy link
Contributor

@spalger spalger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, with one nit

@pheyos pheyos added the auto-backport Deprecated - use backport:version if exact versions are needed label Feb 23, 2022
@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @pheyos

@pheyos pheyos merged commit b7db60a into elastic:main Feb 23, 2022
@pheyos pheyos deleted the ftr_support_mocha_dry_run branch February 23, 2022 10:34
@kibanamachine
Copy link
Contributor

💔 All backports failed

Status Branch Result
8.1 Backport failed because of merge conflicts
8.0 Backport failed because of merge conflicts
7.17 Backport failed because of merge conflicts

How to fix

Re-run the backport manually:

node scripts/backport --pr 125915

Questions ?

Please refer to the Backport tool documentation

pheyos added a commit to pheyos/kibana that referenced this pull request Feb 23, 2022
This PR adds support for the mocha CLI flag `--dry-run` in our functional test execution.

(cherry picked from commit b7db60a)

# Conflicts:
#	yarn.lock
pheyos added a commit to pheyos/kibana that referenced this pull request Feb 23, 2022
This PR adds support for the mocha CLI flag `--dry-run` in our functional test execution.

(cherry picked from commit b7db60a)

# Conflicts:
#	yarn.lock
pheyos added a commit to pheyos/kibana that referenced this pull request Feb 23, 2022
This PR adds support for the mocha CLI flag `--dry-run` in our functional test execution.

(cherry picked from commit b7db60a)

# Conflicts:
#	package.json
#	packages/kbn-test/src/functional_test_runner/cli.ts
#	packages/kbn-test/src/functional_test_runner/functional_test_runner.ts
#	yarn.lock
@pheyos
Copy link
Member Author

pheyos commented Feb 23, 2022

💚 All backports created successfully

Status Branch Result
8.1
8.0
7.17

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

pheyos added a commit that referenced this pull request Feb 23, 2022
This PR adds support for the mocha CLI flag `--dry-run` in our functional test execution.

(cherry picked from commit b7db60a)

# Conflicts:
#	package.json
#	packages/kbn-test/src/functional_test_runner/cli.ts
#	packages/kbn-test/src/functional_test_runner/functional_test_runner.ts
#	yarn.lock

Co-authored-by: Kibana Machine <[email protected]>
pheyos added a commit that referenced this pull request Feb 23, 2022
* FTR support mocha dry run (#125915)

This PR adds support for the mocha CLI flag `--dry-run` in our functional test execution.

(cherry picked from commit b7db60a)

# Conflicts:
#	yarn.lock

* Rebuild @kbn/pm

* Rebuild @kbn/pm part 2

Co-authored-by: Kibana Machine <[email protected]>
pheyos added a commit that referenced this pull request Feb 23, 2022
* FTR support mocha dry run (#125915)

This PR adds support for the mocha CLI flag `--dry-run` in our functional test execution.

(cherry picked from commit b7db60a)

# Conflicts:
#	yarn.lock

* Rebuild @kbn/pm

* Rebuild @kbn/pm part 2

Co-authored-by: Kibana Machine <[email protected]>
lucasfcosta pushed a commit to lucasfcosta/kibana that referenced this pull request Mar 2, 2022
This PR adds support for the mocha CLI flag `--dry-run` in our functional test execution.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-backport Deprecated - use backport:version if exact versions are needed release_note:skip Skip the PR/issue when compiling release notes v7.17.1 v8.0.1 v8.1.0 v8.2.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants