🌱 Tests cleanup

[Adirio]

Description

The following list defines all the checks and tests that we currently have, with some consideration on prerequisites and under which conditions are being executed (go version, devide OS/arch, and kubernetes version).

Lint

• Description: verifies the code style.
• Prerequisites: none.
• Make: make lint to report the errors or make lint-fix to solve them.
• CI: yes (reporting, GH Actions).
• Parameters: single execution.

Unit tests

• Description: unit tests defined in *_test.go files run by go test ... that test the different parts of the library isolated.
• Prerequisites: lint. *1
• Make: make test-unit.
• CI: yes (GH Actions & Prow). *2
• Parameters: multiple executions based on device OS/arch as right now. *3

Coverage *4

• Description: obtains coverage info of unit tests and upload it to coveralls.
• Prerequisites: unit tests.
• Make: make test-coverage obtains the coverage info.
• CI: yes (GH Actions).
• Parameters: single execution.

Integration tests *5

• Description: runs some basic kubebuilder commands to test the CLI.
• Prerequisites: unit tests.
• File: test/integration.sh.
• Make: test-integration.
• CI: yes (Prow).
• Parameters: single execution.

Testdata check

• Description: verifies that the code under the testdata directory is up to date.
• Prerequisites: lint. *6
• File: test/testdata/check.sh.
• Make: make check-testdata.
• CI: yes (GH Actions).
• Parameters: single execution. *7

Testdata tests

• Description: executes the unit tests of the scaffolded projects in the testdata directory.
• Prerequisites: integration tests. *8
• File: test/testdata/test.sh.
• Make: make test-testdata.
• CI: yes (Prow).
• Parameters: single execution. *7

E2E tests

• Description: end-to-end tests.
• Prerequisites: none. *9
• File: test/e2e/local.sh or test/e2e/ci.sh.
• Make: make test-e2e-local or make test-e2e-ci.
• CI: yes (Prow).
• Parameters: multiple executions for several kubernetes versions.

Notes:

1. Linting is not really a prerequisite for unit tests in the most strict sense, but as it is pretty fast, I don't think that it matters that much.
2. We should remove the overlap between GH Actions and Prow.
3. May be interesting to also test for multiple go versions if we do support more than one major go version.
4. We may be able to obtain the coverage info directly from one of the unit tests instead of reruning the tests again for coverage but this may be a bit tricky as we only need to upload one coverage report but unit tests are being executed for different OS/arch.
5. Integration tests are pretty old and have very few cases tested. Testdata tests below are also integrations tests that have way more cases tested and the generated code is actually checked. We should probably remove the old integration tests in favor of the testdata tests.
6. In order to check the testdata folder we need to regenerate it, which requires a running kubebuilder CLI, so testdata check should have as prerequisite the unit tests, not the lint.
7. It may be interesting to run testdata check and tests for several go versions and device OS/arch.
8. We should set check testdata as the prerequisite for test testdata, which would require to move this test to GH Actions.
9. I think that we should set as prerequisites all the previous tests (including unit tests, and testdata tests) which would require to move this tests to GH Actions.

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[Adirio]

Yep, but if we add config/v3 we may need to modify the generate testdata file so I will rebase after that.

[Adirio]

Rebased. There is a single really small conflict with config/v3, so the order in which they are merged doesn't really matter.

[camilamacedo86]

It shows fine for me

/lgtm

Great job. 🥇