Document configuring code coverage for component tests (webpack, vite) #5519

jhiester · 2021-06-03T14:28:23Z

Description

Not clear on how to do code coverage for component testing.

Why is this needed?

It has been really helpful to me and my organization to be able to use the code coverage reports from e2e/integration/unit tests to guide our testing and development. Having component level coverage tests would help in a similar way.

More Info

Comment: Document configuring code coverage for component tests (webpack, vite) #5519
Official Docs: https://docs.cypress.io/guides/tooling/code-coverage

Acceptance Criteria

Document how to set up Code Coverage for Webpack and Vite with simple example repositories
See this issue for lots of info - ideally we should document doing code coverage w/ Webpack and Vite workflows, and provide 2 example repos.
If we can add something to docs, that might be good - we've got some info on E2E, but not much for CT (although process is mostly the same): https://docs.cypress.io/guides/tooling/code-coverage

yann-combarnous · 2021-06-07T11:28:01Z

Example setup for code coverage with Create-React-App, in cypress/plugins/index.js:

// Component / storybook testing
import injectDevServer from '@cypress/react/plugins/react-scripts';
import findReactScriptsWebpackConfig from '@cypress/react/plugins/react-scripts/findReactScriptsWebpackConfig';
import { startDevServer } from '@cypress/webpack-dev-server';
// Code coverage
import codeCoverageTask from '@cypress/code-coverage/task';

// This function is called when a project is opened or re-opened (e.g. due to
// the project's config changing)

/**
 * @type {Cypress.PluginConfig}
 */
// eslint-disable-next-line no-unused-vars
module.exports = (on, config) => {
  // `on` is used to hook into various events Cypress emits
  // `config` is the resolved Cypress config

  // Tell Cypress to report on code coverage
  codeCoverageTask(on, config);

  // Unit/Component testing web dev server setup for instrumenting code coverage
  // For integration testing, this is done in CRA start command
  if (config.testingType === 'component') {
    injectDevServer(on, config);
    const webpackConfig = findReactScriptsWebpackConfig(config, {
      webpackConfigPath: 'react-scripts/config/webpack.config',
    });
    const rules = webpackConfig.module.rules.find((rule) => !!rule.oneOf).oneOf;
    const babelRule = rules.find((rule) => /babel-loader/.test(rule.loader));
    babelRule.options.plugins.push(require.resolve('babel-plugin-istanbul'));

    on('dev-server:start', (options) => {
      return startDevServer({ options, webpackConfig });
    });
  }
  return config;
};

elevatebart · 2021-06-07T16:05:18Z

Hello @jhiester

I believe there could be two subjects to cover here:

How to get component testing code coverage

If you are using create react app see in this repo https://github.com/elevatebart/cy-ct-cra
If you are using Vue see this example https://github.com/cypress-io/cypress/tree/npm/vue/v2/npm/vue/examples/code-coverage

How to merge it with e2e testing coverage

On that front there is good news:
If you run the e2e job and the component job successively, their test coverage going in the same folder, istanbul is smart and will merge the two reports automagically.

If this does not answer your question, feel free to respond below.
If it does, you can close this issue.

Thank you in advance.

Dorious · 2021-06-14T16:37:15Z

Hey @yann-combarnous, I have similar conf and it doesn't collect coverage for CT, for E2E it does it but after running cypress run-ct it doesn't generate anything.

elevatebart · 2021-06-14T20:57:20Z

Hello @Dorious

What @yann-combarnous is doing above should be extracting the coverage in Component testing.

But this line

injectDevServer(on, config);

Will do behind the scenes the same as these lines.

on('dev-server:start', (options) => {
  return startDevServer({ options, webpackConfig });
});

They will be redundant.

The first one will inject the server without setting up any coverage.
The second one will inject and have coverage.

We probably should check that in the processing and send a warning.

Did you see that?

Dorious · 2021-06-15T06:48:42Z

@elevatebart I don't have injectDevServer because I don't use react-scripts. My cypress plugins/index.ts

/// <reference types="cypress" />

import browserify from '@cypress/browserify-preprocessor';
import task from '@cypress/code-coverage/task';
import { initPlugin } from 'cypress-plugin-snapshots/plugin';
import { startDevServer } from '@cypress/webpack-dev-server';

import webpackConfig from '../../webpack.cypress';

/**
 * @type {Cypress.PluginConfig}
 */
const configurePlugins: Cypress.PluginConfig = (on, config) => {
  task(on, config);
  initPlugin(on, config);

  on(
    'file:preprocessor',
    browserify({
      typescript: require.resolve('typescript'),
    })
  );

  if (config.testingType === 'component') {
    // console.log(webpackConfig.module.rules[4].use[0].options);
    on('dev-server:start', (options) => startDevServer({ options, webpackConfig }));
  }

  return config;
};

export default configurePlugins;

The console log is there to check is istanbul there and it is.

Dorious · 2021-06-15T08:45:10Z

Also I see that the coverage is there each test:

But it doesn't push it into coverage dir. In integration tests (cypress run) it works.

Dorious · 2021-06-15T10:20:23Z

@elevatebart ok I figured out what was wrong in my case, in support/ I have separate component.ts and we need to put import '@cypress/code-coverage' :) now coverage is collected properly. @jhiester maybe that the trick for you too.

elevatebart · 2021-06-15T14:43:29Z

Lovely !!!

Code coverage implementation might be a good "recipe" to add to our docs.

To get coverage, you need:

In plugins/index.js, instrumentation of your code (using babel or another istanbul plugin) in the config you pass to the dev server (that is always the hard part).
In plugins/index.js implement the code coverage task with the proper task.

const installCoverageTask = require("@cypress/code-coverage/task");
module.exports = (on, config) => {
  installCoverageTask(on, config)
}

In support/index.js import the code-coverage implementation to run this task when needed.

import '@cypress/code-coverage/support'

vrknetha · 2021-07-19T12:45:36Z

@elevatebart I have followed all the steps mentioned in the repo. But the instrumentation is not happening at all in my case. we use the same react scripts.

elevatebart · 2021-07-21T20:23:34Z

@vrknetha What is your stack? Webpack? vite? vue? react?

elevatebart · 2021-07-28T21:19:21Z

@vrknetha just re-read your answer...
Sorry about that, you already said you are indeed using react-scripts.

Can you share your setup?

package.json
cypress.json
cypress/plugins & cypress/support

A repository would be even better, but I understand not everyone can take this time.

jovancacvetkovic · 2021-08-03T08:08:51Z

Same thing here, CT code coverage is not working. CRA + Typescript. What finally made it work I added

{
        test: /\.(tsx|ts)?$/,
        include: [SRC_PATH],
        use: [{
          loader: 'istanbul-instrumenter-loader',
          options: { esModules: true }
        }]
      }

just before my typescript loader, in webpack config for

on('dev-server:start', options => startDevServer({
    options,
    webpackConfig
  }));

This is what I have:

const webpackConfig = {
    module: {
      rules: [{
        test: /\.(tsx|ts)?$/,
        include: [SRC_PATH],
        use: [{
          loader: 'istanbul-instrumenter-loader',
          options: { esModules: true }
        }]
      }, {
        test: /\.(tsx|ts)?$/,
        include: [SRC_PATH, CYPRESS_PATH, CYPRESS_NODE_PATH],
        use: [{
          loader: 'awesome-typescript-loader',
          options: {
            configFileName: CONFIG_PATH
          }
        }]
      }]
    },
    resolve: {
      extensions: ['.tsx', '.ts', '.js', '.json']
    }
  };

  on('dev-server:start', options => startDevServer({
    options,
    webpackConfig
  }));

And it works for me like this...
Hope it helps someone.

elevatebart · 2021-08-04T14:35:12Z

@YeYaLab that really helps. You rock.
@cowboy can you check and add this to the examples when you get a chance?

cowboy · 2021-08-04T17:17:42Z

Yeah, adding code coverage is absolutely on my list!

chasinhues · 2021-08-26T00:48:35Z

Thanks @elevatebart - this repo helped me a bunch: https://github.com/elevatebart/cy-ct-cra

I'm not currently using integration testing, and was struggling to find resources that demo'd code coverage for a CRA app using only component testing.

In case anyone runs into the same issue, the saving grace for me was in /cypress/plugins/index.js, importing "@cypress/instrument-cra". All other resources I'd found up until now said to add -r @cypress/instrument-cra to the yarn start script. This looks like it's for folks using integration testing though.

My final /cypress/plugins/index.js:

/// <reference types="cypress" />

require("@cypress/instrument-cra");
const injectCraDevServer = require("@cypress/react/plugins/react-scripts");
const installCoverageTask = require("@cypress/code-coverage/task");

/**
 * @type {Cypress.PluginConfig}
 */
module.exports = (on, config) => {
    // `on` is used to hook into various events Cypress emits
    // `config` is the resolved Cypress config

    installCoverageTask(on, config);

    if (config.testingType === "component") {
        injectCraDevServer(on, config);
    }

    return config;
};

My package.json:

     "devDependencies": {
+        "@cypress/code-coverage": "^3.9.10",
+        "@cypress/instrument-cra": "^1.4.0",
+        "@cypress/react": "^5.9.4",
+        "@cypress/webpack-dev-server": "^1.4.0",
+        "@testing-library/cypress": "^8.0.0",
+        "cypress": "^8.3.0",
+        "html-webpack-plugin": "4",

yann-combarnous · 2021-09-07T18:13:18Z

One thing I have not figured out is that I want to have nyc report on all files, which is why I used in package.json:
"nyc": { "all": true, "report-dir": "coverage", "reporter": [ "text-summary", "json", "json-summary", "lcov" ], "include": [ "src/**/*.{js,jsx,mjs,ts,tsx}" ],

But it seems that for component testing, not all files are included in coverage percentage. Any idea how to have component testing honor the nyc setup for scope @elevatebart ?

charleshimmer · 2021-10-06T20:40:00Z

@elevatebart you mentioned code coverage reports should be merged automagically, but I am not finding this true (or at least not in how I have it setup currently). I am not able to run both the cypress e2e runner AND the cypress component test runner at the same time. I run one after the other and whichever one run's lasts is whose coverage report is left in the /coverage folder. Is there anything one needs to do to merge the reports?

elevatebart · 2021-10-07T14:10:22Z

Hello @charleshimmer,

Great question indeed.
I did not dig into it too much yet, I just went with the istambul-combine recommendations.

jamestalmage/istanbul-combine#2

Let me see if I can find a read documentation/tutorial using istanbul report

elevatebart · 2021-10-07T14:20:01Z

Update: report your coverage into separate json files then use nyc merge or nyc report to combine them.
I though it worked on my repo. I might have miscalculated.

Some help:
https://github.com/istanbuljs/nyc#what-about-nyc-merge

yann-combarnous · 2021-12-03T10:23:07Z

@elevatebart , now trying with Vite+React to get code coverage for component testing, but getting some issues.

Using vite-plugin-istanbul 2.2.2, Vite 2.7.0-beta.10, @vitejs/plugin-react 1.1.0, and Cypress 8.7.0.

vite.config.js:

    plugins: [
      react(),
      istanbul({
        include: ['src/**/*.{js,jsx,mjs,ts,tsx}'],
        exclude: [
          'coverage',
          'node_modules',
          'src/**/*.{types,stories,spec,faker,test}.{js,jsx,mjs,ts,tsx}',
        ],
        extension: ['.js', '.ts', '.tsx'],
        requireEnv: true,
     })

cypress/plugins/index.js

import { startDevServer } from '@cypress/vite-dev-server';
import codeCoverageTask from '@cypress/code-coverage/task';

module.exports = (on, config) => {
  // `on` is used to hook into various events Cypress emits
  // `config` is the resolved Cypress config

  // Tell Cypress to report on code coverage
  codeCoverageTask(on, config);

  if (config.testingType === 'component') {
    on('dev-server:start', (options) => {
      return startDevServer({
        options,
        viteConfig: {
          configFile: path.resolve(__dirname, '..', '..', 'vite.config.js'),
        },
      });
    });
  }

When running tests, tests do not run in UI. Error in Cypress UI is:

The following error originated from your test code, not from Cypress.
  > Unexpected identifier
When Cypress detects uncaught errors originating from your test code it will automatically fail the current test.
Cypress could not associate this error to any specific test.
We dynamically generated a new test to display this failure.

at Object.runScripts (http://localhost:3000/__cypress/runner/cypress_runner.js:194369:63)
at $Cypress.onSpecWindow (http://localhost:3000/__cypress/runner/cypress_runner.js:183210:76)
at <unknown> (http://localhost:3000/__cypress/src/@fs//Users/xyz/Code/cbre-360-portal/node_modules/@cypress/vite-dev-server/client/initCypressTests.js:22:18)

In addition, Vite is watching coverage folder, getting the following line updated for all files there:

11:18:14 [vite] page reload coverage/lcov-report/src/constants/icons.ts.html

When I just changed vite.config.js and disabled Istanbul plugin, component test just run fine.

Any idea on how to get this to work with Vite + React?

cowboy · 2021-12-03T15:00:16Z

In case it helps, we've been working on getting code coverage happening over here. Hopefully we'll get some of these PRs merged in soon!

https://github.com/cypress-io/cypress-component-testing-examples/pulls?q=is%3Apr+%22code+coverage%22

samtsai · 2021-12-03T15:05:00Z

What is your folder structure? We saw the same behavior and it almost always had to do with how we're including/excluding files.

yann-combarnous · 2021-12-06T09:03:34Z

In case it helps, we've been working on getting code coverage happening over here. Hopefully we'll get some of these PRs merged in soon!

https://github.com/cypress-io/cypress-component-testing-examples/pulls?q=is%3Apr+%22code+coverage%22

Very helpful, thank you! Found my issue: one "process.env.NODE_ENV" left in my code. Interestingly, it only crashed with Vite code coverage plugin and Cypress. Vite build and dev server ran fine on their own.

joshwooding · 2021-12-16T13:24:19Z

@yann-combarnous How did you end up solving this? I'm seeing the same error due to process.env.NODE_ENV being replaced in the vite-plugin-istanbul sourcesContent output.

function cov_26q8djqde0() {
    ...
    var coverageData = {
        inputSourceMap: {
            version: 3,
            sourcesContent: ["export const reproduction = (item: any) => {\n  if ("development" !== \"production\") {\n    console.log(\"TEST\");\n  }\n\n  return \"\";\n};\n"],
        },
    };
   
}

I also see iFaxity/vite-plugin-istanbul#8 has been raised.

yann-combarnous · 2021-12-16T13:43:52Z

@yann-combarnous How did you end up solving this? I'm seeing the same error due to process.env.NODE_ENV being replaced in the vite-plugin-istanbul sourcesContent output.
function cov_26q8djqde0() {
    ...
    var coverageData = {
        inputSourceMap: {
            version: 3,
            sourcesContent: ["export const reproduction = (item: any) => {\n  if ("development" !== \"production\") {\n    console.log(\"TEST\");\n  }\n\n  return \"\";\n};\n"],
        },
    };
   
}
I also see iFaxity/vite-plugin-istanbul#8 has been raised.

Replace process.env.NODE_ENV in your code by import.meta.env.MODE. See https://vitejs.dev/guide/env-and-mode.html#env-variables .

joshwooding · 2021-12-16T13:59:24Z

Thanks for the quick reply. Sadly I don't think that will work for me since we're building with more than vite.

IlCallo · 2022-03-29T14:44:07Z

If anyone is interested, we just published oob code coverage support for Cypress + TS + Vite + Quasar, check it out: https://github.com/quasarframework/quasar-testing/releases/tag/e2e-cypress-v4.1.0

This commit can be used to replicate the setup on other frameworks/tools too, as I annotate many details in the readme

joshwooding · 2022-05-05T23:05:00Z

@yann-combarnous How did you end up solving this? I'm seeing the same error due to process.env.NODE_ENV being replaced in the vite-plugin-istanbul sourcesContent output.
function cov_26q8djqde0() {
    ...
    var coverageData = {
        inputSourceMap: {
            version: 3,
            sourcesContent: ["export const reproduction = (item: any) => {\n  if ("development" !== \"production\") {\n    console.log(\"TEST\");\n  }\n\n  return \"\";\n};\n"],
        },
    };
   
}
I also see iFaxity/vite-plugin-istanbul#8 has been raised.

Just to provide an update. I finally got around to investigating this and raised a fix that was released in [email protected]

Ancient-Dragon · 2022-08-26T18:18:48Z

Has anyone got a working example of vue 3 + vite working with coverage?

I have installed the vite plugin with the following config options:

istanbul({
        include: 'src/*',
        exclude: ['node_modules'],
        extension: ['.js', '.ts', '.vue'],
        /**
         * This allows us to omit the INSTRUMENT_BUILD env variable when running the production build via
         * npm run build.
         * More details below.
         */
        requireEnv: false,
        /**
         * If forceBuildInstrument is set to true, this will add coverage instrumentation to the
         * built dist files and allow the reporter to collect coverage from the (built files).
         * However, when forceBuildInstrument is set to true, it will not collect coverage from
         * running against the dev server: e.g. npm run dev.
         *
         * To allow collecting coverage from running cypress against the dev server as well as the
         * preview server (built files), we use an env variable, INSTRUMENT_BUILD, to set
         * forceBuildInstrument to true when running against the preview server via the
         * instrument-build npm script.
         *
         * When you run `npm run build`, the INSTRUMENT_BUILD env variable is omitted from the npm
         * script which will result in forceBuildInstrument being set to false, ensuring your
         * dist/built files for production do not include coverage instrumentation code.
         */
        forceBuildInstrument: Boolean(process.env.INSTRUMENT_BUILD),
      })

Cypress config:

  component: {
    devServer: {
      framework: 'vue',
      bundler: 'vite',
    },
    screenshotOnRunFailure: false,
    video: false,
  },

And my package.json commands looks like:

    "instrument-build": "cross-env INSTRUMENT_BUILD=true vite build",
    "test:component": "npm run instrument-build && cypress run --component --reporter spec",

I'm using cypress 10.4.0 & cypress coverage 3.10.0. Any help would be greatly appreciated

lmiller1990 · 2023-02-22T02:36:41Z

I think we need a definitive guide on coverage. Some resources (other than the ones here):

CRA + webpack: https://muratkerem.gitbook.io/cctdd/ch30-appendix/combined-code-coverage
Vite (some info floating around here) instrument react source code with cypres iFaxity/vite-plugin-istanbul#22

lmiller1990 · 2023-02-22T06:31:44Z

Hey team! Please add your planning poker estimate with Zenhub @astone123 @marktnoonan @mike-plummer @warrensplayer @ZachJW34

muratkeremozcan · 2023-02-22T13:10:44Z

In case it helps, we've been working on getting code coverage happening over here. Hopefully we'll get some of these PRs merged in soon!

https://github.com/cypress-io/cypress-component-testing-examples/pulls?q=is%3Apr+%22code+coverage%22

Those are very useful, because we need framework x bundler many recipes.

lmiller1990 · 2023-02-23T02:22:32Z

Yep, we are picking this up in our next block of work, Feb 28->Ma 14, the goal is just to document common combinations and how it works in general, so people with less common combos can configure it, too.

muratkeremozcan · 2023-03-06T16:33:31Z

Yep, we are picking this up in our next block of work, Feb 28->Ma 14, the goal is just to document common combinations and how it works in general, so people with less common combos can configure it, too.

I updated the CCTDD book with the instructions for the Vite version of combined code coverage :
https://app.gitbook.com/s/jK1ARkRsP5OQ6ygMk6el/ch30-appendix/combined-code-coverage#addendum-code-coverage-with-vite-instead-of-webpack .

Here's the app & working combined code cov https://github.com/muratkeremozcan/tour-of-heroes-react-vite-cypress-ts.

lmiller1990 · 2023-03-08T23:17:02Z

Hey team! Please add your planning poker estimate with Zenhub @astone123 @marktnoonan @mike-plummer @warrensplayer @jordanpowell88

lmiller1990 · 2023-03-12T22:36:11Z

Relevent cypress-io/cypress-component-testing-apps#16

elevatebart added the type: question label Jun 7, 2021

JessicaSachs removed the type: question label Aug 18, 2021

cypress-bot bot added the stage: backlog label Apr 29, 2022

lmiller1990 added the CT label Aug 15, 2022

luxaritas mentioned this issue Sep 8, 2022

Test coverage for Cypress component tests eternagame/workspace-helpers#176

Open

lmiller1990 mentioned this issue Feb 24, 2023

CT Devservers should support instrumenting code coverage as a baked in option cypress-io/cypress#21945

Closed

lmiller1990 changed the title ~~Code coverage for component tests~~ Document configuring code coverage for component tests (webpack, vite) Mar 8, 2023

lmiller1990 mentioned this issue Mar 12, 2023

Code Coverage for Cypress Component Tests cypress-io/cypress-component-testing-apps#16

Open

jennifer-shehane transferred this issue from cypress-io/cypress Oct 4, 2023

jennifer-shehane removed the stage: backlog label Apr 26, 2024

Document configuring code coverage for component tests (webpack, vite) #5519

Document configuring code coverage for component tests (webpack, vite) #5519

Comments

jhiester commented Jun 3, 2021 • edited by lmiller1990 Loading

Description

Why is this needed?

More Info

Acceptance Criteria

yann-combarnous commented Jun 7, 2021

elevatebart commented Jun 7, 2021

How to get component testing code coverage

How to merge it with e2e testing coverage

Dorious commented Jun 14, 2021

elevatebart commented Jun 14, 2021

Dorious commented Jun 15, 2021

Dorious commented Jun 15, 2021

Dorious commented Jun 15, 2021 • edited Loading

elevatebart commented Jun 15, 2021 • edited Loading

vrknetha commented Jul 19, 2021

elevatebart commented Jul 21, 2021

elevatebart commented Jul 28, 2021

jovancacvetkovic commented Aug 3, 2021 • edited Loading

elevatebart commented Aug 4, 2021 • edited Loading

cowboy commented Aug 4, 2021

chasinhues commented Aug 26, 2021

yann-combarnous commented Sep 7, 2021 • edited Loading

charleshimmer commented Oct 6, 2021 • edited Loading

elevatebart commented Oct 7, 2021

elevatebart commented Oct 7, 2021

yann-combarnous commented Dec 3, 2021 • edited Loading

cowboy commented Dec 3, 2021

samtsai commented Dec 3, 2021

yann-combarnous commented Dec 6, 2021

joshwooding commented Dec 16, 2021 • edited Loading

yann-combarnous commented Dec 16, 2021

joshwooding commented Dec 16, 2021 • edited Loading

IlCallo commented Mar 29, 2022

joshwooding commented May 5, 2022

Ancient-Dragon commented Aug 26, 2022

lmiller1990 commented Feb 22, 2023

lmiller1990 commented Feb 22, 2023

muratkeremozcan commented Feb 22, 2023 • edited Loading

lmiller1990 commented Feb 23, 2023

muratkeremozcan commented Mar 6, 2023

lmiller1990 commented Mar 8, 2023

lmiller1990 commented Mar 12, 2023

jhiester commented Jun 3, 2021 •

edited by lmiller1990

Loading

Dorious commented Jun 15, 2021 •

edited

Loading

elevatebart commented Jun 15, 2021 •

edited

Loading

jovancacvetkovic commented Aug 3, 2021 •

edited

Loading

elevatebart commented Aug 4, 2021 •

edited

Loading

yann-combarnous commented Sep 7, 2021 •

edited

Loading

charleshimmer commented Oct 6, 2021 •

edited

Loading

yann-combarnous commented Dec 3, 2021 •

edited

Loading

joshwooding commented Dec 16, 2021 •

edited

Loading

joshwooding commented Dec 16, 2021 •

edited

Loading

muratkeremozcan commented Feb 22, 2023 •

edited

Loading