Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: [M3-7667] - (Proof of Concept) Add tagging capability for Cypress tests #10475

Merged
merged 10 commits into from
Jun 12, 2024

Conversation

jdamore-linode
Copy link
Contributor

@jdamore-linode jdamore-linode commented May 15, 2024

Description 📝

Proof of concept to give us the capability to apply arbitrary tags to our Cypress tests and run Cypress against subsets of tags.

Motivation

The main goal of this effort is to give us more flexibility in how/when we run our tests. M3-7667 and M3-8072 explore some of the challenges we're facing as our test suite grows and as more teams rely on Cloud and its tests for their own purposes.

Usage

Tagging Tests

When developing tests, we can tag them using cy.tag or cy.addTag. Tags can be applied to individual tests, or to multiple tests using beforeEach hooks. cy.tag will replace any tags that have already been set, while cy.addTag will add tags in addition to any that may have been set via hooks.

A quick example of what this looks like:

describe('a group of tests', () => {
  beforeEach(() => {
    cy.tag('feat:linodes', 'method:e2e');
  });
  
  it('tests a thing', () => {
    // This test will have 'feat:linodes' and 'method:e2e' tags.
  });

  it('tests a thing with mocks', () => {
    cy.tag('feat:linodes', 'method:mock');
    // This test will have 'feat:linodes' and 'method:mock' tags.
  });

  it('tests a last thing', () => {
    cy.addTag('feat:placementGroups');
    // This test will have 'feat:linodes', 'feat:placementGroups', and 'method:e2e' tags.
  });
});

Running Tagged Tests

By default, running yarn cy:run will run all tests regardless of how they're tagged. To specify which tests should run using tags, users can use the CY_TEST_TAGS environment variable:

echo "Run all the tests with 'feat:linodes' tag"
CY_TEST_TAGS='feat:linodes' yarn cy:run

Tags can be composed and negated:

echo "Run all the tests that have either 'feat:linodes' or 'feat:placementGroups' tags"
CY_TEST_TAGS='feat:linodes,feat:placementGroups' yarn cy:run

echo "Run all the tests that have 'feat:linodes' and 'feat:placementGroups' tags"
CY_TEST_TAGS='feat:linodes feat:placementGroups' yarn cy:run

echo "Run all the tests that do not have the 'feat:linodes' tag"
CY_TEST_TAGS='-feat:linodes' yarn cy:run

echo "Run all the tests that have either 'feat:linodes' or 'feat:placementGroups' tags, but not the 'method:e2e' tag"
CY_TEST_TAGS='feat:linodes,feat:placementGroups -method:e2e' yarn cy:run

Implementation

The tag utils work by modifying each test's Mocha context when Cypress executes the test, which happens before the test runs. Because cy.tag and cy.addTag are not part of Cypress's command queue, they can be called at any point in a test and will still apply.

Why Roll Our Own?

I opted to implement our own tagging mechanism rather than rely on an existing plugin or package. There are two packages that are relevant, and inspiration was drawn from each, but neither is perfectly suitable for our needs:

  • cypress-tags - This works by adding a TypeScript preprocessing step that searches for patterns in the test code itself to construct the tag graph, then strips out the tagging code and passes it to Cypress for execution. I didn't like this approach because a lot of time is already required to preprocess our specs via Vite and I wanted to avoid increasing our preprocessing time even further, and because of concerns that this method could have compatibility issues with our existing TypeScript/Vite preprocessing config and setup.
  • mocha-tags - This package isn't very well supported (last updated 7 years ago, no GitHub repo) and works by mutating the global it(...), describe(...), etc. functions exposed by Mocha. I did take a lot of inspiration from its tag querying/composing mechanism, however.

Downsides, Caveats, etc.

Performance

Because skipped tests are identified at runtime, Cypress must still preprocess all spec files included in its run, even if all the tests within the spec ultimately get skipped. Consequently, running the test suite with all tests skipped still takes approximately 10 minutes (when running locally on my machine) to account for the preprocessing.

Changes to cy.defer()

This implementation required changes to cy.defer() so that it consumes a function that returns a Promise rather than consuming a Promise directly. This is necessary because tests that get skipped still get executed by Cypress (without being run), so Promises that are initialized outside of Cypress's command queue get rejected when the test is skipped, resulting in errors and exceptions in certain cases (e.g. Axios requests getting aborted).

Reliance on cy.state()

This implementation requires use of the cy.state() function in order to access the tests' Mocha contexts, which is undocumented and could be changed. We're already relying on cy.state() (albeit to a much lesser extent) because it exposes the only method available to skip a Cypress test at runtime. Even if this implementation were to change, we can adapt as long as Cypress continues to expose the Mocha context.

Changes 🔄

List any change relevant to the reviewer.

  • Adds tag and addTag utils, as well as global aliases cy.tag and cy.addTag
  • Adds CY_TEST_TAGS environment variable to allow users to specify which tests to run using tags
  • Defines a test tag type with a couple tags, but this is mostly for demonstration purposes (we should probably put more thought into what types of tags we're actually going to want)
  • For testing/demonstration purposes only, applies some tags to the tests in create-linode.spec.ts

Preview 📷

Running Linode create tests that are tagged with the e2e tag:
Screenshot 2024-05-15 at 5 11 54 PM

Tag info that gets logged when Cypress starts, intended to help troubleshoot tag queries:
Screenshot 2024-05-15 at 5 13 55 PM

Error message when the user specifies an invalid tag query:
Screenshot 2024-05-15 at 5 15 00 PM

How to test 🧪

  • Confirm that running yarn cy:run runs all of the tests (you can stop after a few tests have run, no need to wait for the entire suite -- CI will help us double check this, too)
  • Confirm that running CY_TEST_TAGS='method:e2e' yarn cy:run -s "cypress/e2e/core/linodes/create-linode.spec.ts" runs only the tests in create-linode.spec.ts which are tagged with method:e2e
  • Feel free to experiment with more advanced tag queries and other combinations of spec files

As an Author I have considered 🤔

Check all that apply

  • 👀 Doing a self review
  • ❔ Our contribution guidelines
  • 🤏 Splitting feature into small PRs
  • ➕ Adding a changeset
  • 🧪 Providing/Improving test coverage
  • 🔐 Removing all sensitive information from the code and PR description
  • 🚩 Using a feature flag to protect the release
  • 👣 Providing comprehensive reproduction steps
  • 📑 Providing or updating our documentation
  • 🕛 Scheduling a pair reviewing session
  • 📱 Providing mobile support
  • ♿ Providing accessibility support

@jdamore-linode jdamore-linode self-assigned this May 15, 2024
@jdamore-linode jdamore-linode requested review from a team as code owners May 15, 2024 21:24
@jdamore-linode jdamore-linode requested review from cliu-akamai, cpathipa and AzureLatte and removed request for a team May 15, 2024 21:24
Copy link

github-actions bot commented May 15, 2024

Coverage Report:
Base Coverage: 82.76%
Current Coverage: 82.76%

Copy link
Contributor

@AzureLatte AzureLatte left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yarn cy:run works, Also test for create-linode.spec.ts passed
✔ All specs passed!

Copy link
Contributor

@jaalah-akamai jaalah-akamai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this looks good to me 👍

I believe I read that cy.state is deprecated and will be removed in future releases (albeit it seems to have stuck around for several major releases). https://docs.cypress.io/guides/references/changelog#10-5-0

cy.state('subject') is deprecated and reading from it will log a warning to the console. Prefer cy.currentSubject() instead. Addresses [#23092](https://github.com/cypress-io/cypress/issues/23092).

Unless I'm misunderstanding something?

@jaalah-akamai jaalah-akamai added the Add'tl Approval Needed Waiting on another approval! label May 30, 2024
@jdamore-linode
Copy link
Contributor Author

Thanks @jaalah-akamai

I believe I read that cy.state is deprecated and will be removed in future releases (albeit it seems to have stuck around for several major releases). https://docs.cypress.io/guides/references/changelog#10-5-0

Not quite -- cy.state('subject') specifically was deprecated and replaced with a dedicated command, but cy.state() has not been deprecated. It is undocumented, but I haven't seen any indication that it's planned to be removed (it's used a lot by Cypress itself internally)

@bnussman-akamai bnussman-akamai added Approved Multiple approvals and ready to merge! and removed Add'tl Approval Needed Waiting on another approval! labels Jun 5, 2024
@jdamore-linode
Copy link
Contributor Author

Merging despite test failure because the failing test is already present in develop, not introduced by this PR

@jdamore-linode jdamore-linode merged commit 230b623 into linode:develop Jun 12, 2024
17 of 18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Approved Multiple approvals and ready to merge! Proof of Concept
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants