Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement] Add full support for cucumber.js CLI options #2962

Closed
tim-yao opened this issue Dec 1, 2021 · 4 comments · Fixed by #2978
Closed

[Enhancement] Add full support for cucumber.js CLI options #2962

tim-yao opened this issue Dec 1, 2021 · 4 comments · Fixed by #2978
Assignees
Labels
Milestone

Comments

@tim-yao
Copy link
Contributor

tim-yao commented Dec 1, 2021

Is your feature request related to a problem? Please describe.
Nightwatch v2.0.0-beta.1 has included support for using cucumber-js as test runner, and we can run cucumber-js by using nightwatch CLI(like npx nightwatch --env cucumber-js).

However, not all cucumber.js CLI options are supported. See discussion in #2939 (comment). For example, if I want to run cucumber "Dry Run" mode, npx nightwatch --env cucumber-js --dry-run won't work. Also use it in the test runner options like below won't work either.

test_runner: {
    // set cucumber as the runner
    type: 'cucumber',

    // define cucumber specific options
    options: {
      feature_path: 'features/**/*.feature', // `feature_path` is the only one will work in my test.
      retry: 2,
      'dry-run': true,
    }
}

Describe the solution you'd like

If Nightwatch want to support cucumber-js, it should support all cucumber-js options in my opinion. All options should able to be forwarded by nightwatch arguments or added via test runner options.

Not sure if there are conflicts between cucumber-js and nightwatch.js CLI options. If not, we can forward all cucumber-js options in CLI. Otherwise, we may need other way to add namespace I think. For example: --c-dry-run? No idea what is the best practice.

A list of cucumber-js options(✅ means already supported in Nightwatch.js v2)

options forward test runner options
--dry-run
--name
--tags
--require
--format
--format-options
--fail-fast
--retry
--retry-tag-filter
--require-module
--no-strict
--parallel
--config
--profile
--world-parameters

Describe how the solution will work

Add all above support in getCliArgvForCucumber()

Describe alternatives you've considered

Additional context

@beatfactor
Copy link
Member

Ok, we'll look into these. Some of them might not work, for instance I don't know what --dry-run would do or how it would be used.

@tim-yao
Copy link
Contributor Author

tim-yao commented Dec 1, 2021

@gravityvi gravityvi self-assigned this Dec 9, 2021
@beatfactor beatfactor added the v2 label Dec 9, 2021
@beatfactor beatfactor added this to the Nightwatch v2 milestone Dec 9, 2021
@gravityvi
Copy link
Member

Some of them would create conflicts, for example supporting config from cucumber CLI. We might need a better way to support them.

@phil-davis
Copy link

phil-davis commented Jan 14, 2022

Ok, we'll look into these. Some of them might not work, for instance I don't know what --dry-run would do or how it would be used.

An example use case - I have some CI for a project that is in development and there are test scenarios that run and are known to fail (developers are working to make them pass). We process the list of fails from a real test run and compare it to a list of "expected failures". If the failures match the "expected failures" then the PR does not make things any worse! Hopefully more tests start passing, and developers can delete scenarios from the expected-failures list over time.

If someone refactors a scenario that is already failing, and they accidentally make a typo in a test step, then we don't notice that in CI - the scenario is already an "expected failure"!

I would like to be able to do a dry-run in an early CI pipeline as a Gherkin "linter". It should fail if the format of any feature file is not parseable by the Gherkin parser, and/or if any test step is undefined. That will help catch "crud" in feature files early, before actually running the tests against a real system-under-test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants