api: add runCucumber function internally

[davidjgoss]

Description

This adds the async runCucumber function for doing a test run programatically. It's not yet exposed publicly, but is used internally by the CCK tests.

This has involved:

• Pulling most of the guts out of the Cli class, since that is now a consumer of this new function
• Handling the i18n-related commands purely in the cli module
• Various other tweaks and refactors

Motivation & context

First step towards #1711.

Now seems a good time to validate the approach on a few fronts:

• Defaulting of configuration options is generally moving inwards, so it's not tied to the argv parsing and so that API consumers can call with a very small signature if appropriate.
• The shape of the config object passed in to runCucumber. The idea is that the support block could be replaced with an existing support code library on subsequent runs in the same process, which solves a problem for a few framework authors.
• No breaking changes in the existing Cli, Runtime etc - framework authors are using these in the wild and we need to have a stable and compelling alternative before pulling them.

If we're happy and this goes in, the rest can follow quite quickly. Next steps:

• Implement the ability to reuse support code library
• Add a mechanism to load configuration from argv, files etc in a sane way
• Add integration tests specifically for this API
• Expose publicly (probably with an "experimental" qualifier), add documentation

Type of change

• Refactoring/debt (improvement to code design or tooling without changing behaviour)
• New feature (non-breaking change which adds new behaviour)

Checklist:

• I have read the CONTRIBUTING document.
• My code follows the code style of this project.
• I have added tests to cover my changes.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have added an entry to the "Unreleased" section of the CHANGELOG, linking to this pull request.

[aurelien-reeves]

After a first look, it looks really great! Well done!

I'll have a deeper look later

[nicojs]

Great to see progress here! 😍

The shape of the config object passed in to runCucumber. The idea is that the support block could be replaced with an existing support code library on subsequent runs in the same process, which solves a problem for a few framework authors.

Good to hear it is still on the radar. It will dramatically improve mutation testing performance ⏩💨

[mattwynne]

I can only offer nit-picks, this looks great to me!

[aurelien-reeves]

It looks really great!
I agree with Matt's comment.
Beside that, this is an approval 🚀

[davidjgoss]

Thanks for the feedback @aurelien-reeves @nicojs @mattwynne - I'm going to address Matt's very fair nit-pick, beef up the tests a bit and then get this merged and on to the next part.

[nicojs]

Question: does this break the current (undocumented) API?

This is what we're using in our cucumber plugin right now:

const argv = [
      'node',
      'cucumber-js',
      '--retry',
      '0',
      '--parallel',
      '0',
      '--format',
      require.resolve('./stryker-formatter'),
      ...bailArgs,
      ...tagsArgs,
      ...profileArgs,
      ...testFilterArgs,
    ];
    const cli = new Cli({
      argv,
      cwd: process.cwd(),
      stdout: process.stdout,
    });
   await cli.run();

[aurelien-reeves]

Question: does this break the current (undocumented) API?

This is what we're using in our cucumber plugin right now:

const argv = [
      'node',
      'cucumber-js',
      '--retry',
      '0',
      '--parallel',
      '0',
      '--format',
      require.resolve('./stryker-formatter'),
      ...bailArgs,
      ...tagsArgs,
      ...profileArgs,
      ...testFilterArgs,
    ];
    const cli = new Cli({
      argv,
      cwd: process.cwd(),
      stdout: process.stdout,
    });
   await cli.run();

I took a look at the code in the branch. As far as I can tell, your snippet won't be broken. It will use the new API under the hood, but it will work as expected :)

[davidjgoss]

@nicojs @aurelien-reeves yep, very much a conscious choice to avoid any breaking changes to existing Cli and Runtime for the near to mid term at least.

[binarymist]

Question: does this break the current (undocumented) API?

For the record: Yes.

This is what we're using in our code that depends on cucumber right now and have done for the last 3-4 years.

[davidjgoss]

@binarymist I don't think getConfiguration was intended to be public, it perhaps should have been a private method. I think I understand your use case though having looked through the code and this should be catered for via the new API in the next RC.

[binarymist]

this should be catered for via the new API in the next RC

Great!

Problme is... If we didn't use getConfiguration, then we wouldn't be able to show the Build User (think end user) the test plan that they are about to run. Infact when we implemented this, the Cucumber CLI was barely usable. Quite a few of our large architectural decissions were made due to the fact that Cucumber just wouldn't do what we needed it to do easily.