End-to-end test coverage

[kschiffer]

Summary

This umbrella issue keeps track of our end-to-end test coverage.

• Applications
  • CRUD Add application e2e tests #3650
  • Checking overview (information displayed as expected) Add overview e2e tests #3663
  • Payload formatters (set all types, remove) Add application payload formatters e2e tests #3841
  • Integrations
    • MQTT (Check credentials)
    • Webhooks (CRUD, with and without webhook template)
    • Pub/sub (CRUD) Add application pub sub tests #3720
  • Collaborators (CRUD) Refactor application collaborators  #3443, Add collaborators e2e tests #3646
  • API Keys (CRUD) Refactor application-api-key components #3452
• End devices
  • CRUD
    • Create
    • Read
    • Update
    • Delete
  • Import
  • Checking overview (information displayed as expected) Add overview e2e tests #3663
  • Scheduling downlinks Update end device messaging downlink functionality #3883
  • Location (setting and removing) @asmulko on this atm
  • Payload formatters (set types, remove, check precedence over app level PFs) Add application payload formatters e2e tests #3841
  • Claim authentication code (setting and removing)
  • General Settings (test various sub-forms, removing gateway)
• Gateways
  • CRUD Add gateway tests #3672
  • Checking overview (information displayed as expected) Add overview e2e tests #3663
  • Location (setting and removing) Add gateway location tests #3681
    • Confirm updating via status message
  • Collaborators (CRUD) Refactor application collaborators  #3443, Add collaborators e2e tests #3646
  • API Keys (CRUD) Refactor application-api-key components #3452
• Organizations
  • CRUD console: Add organization tests #3670
  • Checking overview (information displayed as expected) Add overview e2e tests #3663
  • Collaborators (CRUD) Refactor application collaborators  #3443, Add collaborators e2e tests #3646
  • API Keys (CRUD) Refactor application-api-key components #3452
• Authentication and authorization
  • Login
  • Logout (Console and OAuth app)
  • User registration
  • Change password (when authenticated)
  • Change password (using temporary password)
  • Authorizing an OAuth client
  • Verifying contact info
  • Retrieving authorization code (/oauth/code)
• Miscellaneous
  • Feature toggles
    • Check various rights setups and confirm:
      • Correct visibility of items
      • Correct write restriction of items
  • Data / Event streams (all entities)
    • we need some work to enabling simulating events from within the cypress test Add frontend end-to-end tests that use traffic simulation #6694
    • Confirm correct rendering of various event types
    • Test pausing and clearing
  • Error pages
  • User management (CRUD)
  • Search functionality in tables
  • Some smoke testing of responsive styles
• Add functional tests to storybook Add functional tests to storybook #3478
• Add page title assertions to e2e tests Add page title assertions to e2e tests  #3479
• Speed up e2e execution in CI Speed up e2e execution in CI #3480
• Add clipboard support for e2e tests Add clipboard support for e2e tests #3481
• Add direct database access for end-to-end tests Add direct database access for end-to-end tests #3605
• Save screenshots as artifacts for failed e2e tests Save screenshots as artifacts for failed e2e tests #3523
• Packet broker
• MAC Settings

Also note the complementary umbrella issue for enterprise features:
https://github.com/TheThingsIndustries/lorawan-stack/issues/2575

Everyone please keep this up to date and add test-cases that you think are important, or split up cases into more detailed sub-cases.

For everyone that wants to dive into writing end-to-end tests, please see the guide: https://github.com/TheThingsNetwork/lorawan-stack/blob/v3.10/DEVELOPMENT.md#end-to-end-tests

[kschiffer]

I'll add the rest of the dev team as assignees once #3611 is done.

[asmulko]

@kschiffer I assume that Application / Linking e2e tests can be removed from the list, because it will be removed in v3.11, see #3614 and we can put Payload formatters on hold as well

[kschiffer]

Yes indeed. Linking can be removed, though Payload formatters will stay untouched as far as I understand.

[johanstokking]

What is the status here? What can we improve on the short term still?

[kschiffer]

@asmulko made a lot of progress with this recently and I think the coverage is pretty good already. The most important thing short-term is traffic simulation, which will enable us putting a whole range of critical end-to-end tests in place. This is also tracked in #3611. Then we need to start looking into performance optimizations #3480.

[johanstokking]

OK. Should we track end-to-end testing in this issue or in #3611, or is there a reason to keep both?

[kschiffer]

I'd prefer keeping and maintaining this overview here as well. We can update this issue with a lot more test cases and onboard more of the team to write end-to-end tests after #3611 is done.

[kschiffer]

Little reminder for me, @bafonins, and @ryaplots to keep the list above updated and to periodically check back whether some of those can be picked up by us.

[kschiffer]

I think we an close this now. The rest of the missing coverage is either tracked by individual issues or not planned currently.