Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Epic]: Refactor e2e tests to increase coverage and reduce complexity #2283

Open
18 tasks
mpoke opened this issue Sep 18, 2024 · 0 comments
Open
18 tasks

[Epic]: Refactor e2e tests to increase coverage and reduce complexity #2283

mpoke opened this issue Sep 18, 2024 · 0 comments
Assignees
Labels
admin: epic An EPIC -- meta issue used to track a body of work admin: key-result A key result (in the context of OKRs)

Comments

@mpoke
Copy link
Contributor

mpoke commented Sep 18, 2024

Problem

Currently, the e2e tests consists of the following steps (see tests/e2e/README.md for more details):

  • start a docker container
  • run a defined sequence of actions with expected states
  • after each action, query the system state and compare against the expected state

Also, there are a set of predefined test cases (more than 20) that can be executed directly. Run go run ./tests/e2e/... --help for details.

Although at the high level this e2e framework is good, there are several issues that need to be addressed:

  • Some of the tests are non-deterministic as they depend on timing assumptions. As a result, tests are sometimes failing and need to be rerun.
  • Most of the existing test cases have unnecessary actions resulting in tests with a high number of steps (e.g., the happy-path has 39 steps).
    • This makes the testing time long. As a result, the non-deterministic aspect has an even higher impact, e.g., if step 38 in the happy-path test case is randomly failing, the entire test case needs to be retried.
    • This also makes the tests complex and hard to modify and debug, e.g., there is no reason to start consumer chains or create IBC channels to test the consumer lifecycle on the provider.
  • Many of the test cases are not part of the CI or the nightly tests, which means that are rarely run.

Closing criteria

The tasks below are completed.

Task list

Must have

  1. scope: testing
    stana-miric

Questions

Nice to have

  1. good first issue scope: testing
  2. scope: testing

Others (to triage)

  1. scope: testing
    bermuell
  2. scope: testing type: bug
    stana-miric
  3. scope: testing type: tech-debt
    MSalopek
  4. scope: testing type: tech-debt
  5. 6 of 9
    admin: epic scope: testing type: tech-debt
  6. scope: testing
    stana-miric
@mpoke mpoke added admin: epic An EPIC -- meta issue used to track a body of work admin: key-result A key result (in the context of OKRs) labels Sep 18, 2024
@mpoke mpoke self-assigned this Sep 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
admin: epic An EPIC -- meta issue used to track a body of work admin: key-result A key result (in the context of OKRs)
Projects
Status: 🏗 F3: InProgress
Development

No branches or pull requests

1 participant