Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update KEP-0004 to describe testing procedures, types, and policies. #342

Merged
merged 6 commits into from
Jun 20, 2019
Merged

Update KEP-0004 to describe testing procedures, types, and policies. #342

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
0ac3c4e
Update KEP-0004 to describe testing procedures, types, and policies.
jbarrick-mesosphere Jun 13, 2019
9e86c56
Add section on testing in Kubernetes clusters.
jbarrick-mesosphere Jun 13, 2019
accdc03
Remove editor line breaks.
jbarrick-mesosphere Jun 13, 2019
bcfafbc
Address review comments.
jbarrick-mesosphere Jun 13, 2019
fe8de10
Switch integration tests to only reference the test harness and not G…
jbarrick-mesosphere Jun 13, 2019
0a88ce7
Update goals and graduation criteria
jbarrick-mesosphere Jun 14, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
120 changes: 77 additions & 43 deletions keps/0004-add-testing-infrastructure.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,41 +19,46 @@ superseded-by:

## Table of Contents

- [Table of Contents](#table-of-contents)
- [Summary](#summary)
- [Motivation](#motivation)
- [Goals](#goals)
- [Non-Goals](#non-goals)
- [Proposal](#proposal)
- [User Stories](#user-stories)
- [Story 1](#story-1)
- [Story 2](#story-2)
- [Implementation Details/Notes/Constraints [optional]](#implementation-detailsnotesconstraints-optional)
- [Risks and Mitigations](#risks-and-mitigations)
- [Graduation Criteria](#graduation-criteria)
- [Implementation History](#implementation-history)
- [Infrastructure Needed [optional]](#infrastructure-needed-optional)

[tools for generating]: https://github.com/ekalinin/github-markdown-toc
* [Summary](#summary)
* [Motivation](#motivation)
* [Goals](#goals)
* [Non-Goals](#non-goals)
* [Proposal](#proposal)
* [Tests](#tests)
* [Unit tests](#unit-tests)
* [Integration tests](#integration-tests)
* [Framework tests](#framework-tests)
* [Kubernetes clusters](keps/0004-add-testing-infrastructure.md#kubernetes-clusters)
* [CICD](#cicd)
* [Pull Requests](#pull-requests)
* [Master Branch](#master-branch)
* [Tests](#tests-1)
* [Pushes](#pushes)
* [Schedule](#schedule)
* [Base Image Change](#base-image-change)
* [Tags/Release](#tagsrelease)
* [User Stories](#user-stories)
* [Story 1](#story-1)
* [Story 2](#story-2)
* [Risks and Mitigations](#risks-and-mitigations)
* [Graduation Criteria](#graduation-criteria)
* [Implementation History](#implementation-history)
* [Infrastructure Needed [optional]](#infrastructure-needed-optional)
jbarrick-mesosphere marked this conversation as resolved.
Show resolved Hide resolved

## Summary

As the complexity and scope of KUDO grows, it becomes impossible to manually validate
exisiting frameworks and capabilities still function as expected. As part of a robust
CICD pipeline, a testing suite needs to be developed that can validate each commit, pull
request, or even deployment of KUDO on a cluster.
As the complexity and scope of KUDO grows, it becomes impossible to manually validate exisiting frameworks and capabilities still function as expected. As part of a robust CICD pipeline, a testing suite needs to be developed that can validate each commit, pull request, or even deployment of KUDO on a cluster.

These testing packages should also assure conformance for the `kudobuilder/frameworks` repository
to particular versions of KUDO.
These testing packages should also assure conformance for the `kudobuilder/frameworks` repository to particular versions of KUDO.

[KEP-0008](https://github.com/kudobuilder/kudo/blob/master/keps/0008-framework-testing.md) outlines the design of a testing harness for validating KUDO and frameworks. This document outlines testing procedures and policies - e.g., what, where, when and how we will test.

## Motivation

### Goals

- Ensure validation of API objects is functioning correctly
- Ensure controllers execute known process flows correctly
- Validate Framework and FrameworkVersions in kudobuilder/frameworks adhere to the API spec defined by Kudo. Provide common
misconfigurations and validate that the testing framework notifies the users/developer of failure
- Reduce review time for code changes by not requiring reviewers to validate functionality of test cases
- Reduce developer time for code changes by providing tools to validate functionality
- Provide developers clear tooling for addition additional tests to infrastructure to validate bug fixes and new features
Expand All @@ -68,36 +73,72 @@ This is a two phase proposal.

### Tests

Add testing packages to:
KUDO will be tested by several different test suites that validate individual components, KUDO, and frameworks.

#### Unit tests

Unit tests test a single component and do not require an Internet connection or the Kubernetes API. These should be fast, run on every pull request, and test that individual KUDO components behave correctly. Unit tests are written as standard Go tests.

All packages should incorporate unit tests that validate their behavior. Some examples of tests that should be included are:
jbarrick-mesosphere marked this conversation as resolved.
Show resolved Hide resolved

- pkg/apis - Conformance of objects, serialization, validation
- pkg/controller - Controllers implement the correct process flows for changes in Instance objects, and creation of PlanExecutions
- pkg/util - Util functions behave as expected

Integration tests
Go coverage reports will be used to ensure that changes do not reduce test coverage.

#### Integration tests

Integration tests test the KUDO controller in its entirety. These tests can be run against either a real Kubernetes cluster or a local control plane. Integration tests will be manually run after review, but prior to merging using the [Circle CI manual approval feature](https://circleci.com/docs/2.0/triggers/#manual-approval). Integration tests can be written using the test harness designed in KEP-0008.

The integration tests will consist of a set of representative Frameworks and FrameworkVersions that utilize real world KUDO features. Integration tests should be added for new KUDO capabilities at the discretion of the capability author or reviewers.

Integration tests will be hidden behind a Go build tag and will only run when the `integration` tag is specified.

#### Framework tests
jbarrick-mesosphere marked this conversation as resolved.
Show resolved Hide resolved

Framework tests test that a framework works correctly. These require a full Kubernetes cluster to run and will be run in CI for the Frameworks repository using the latest released version of KUDO. Instead of running every test on every pull request, we will only run the tests that test the framework changed in any given pull request. Framework tests will also be run against master and release builds of KUDO to verify that KUDO changes do not break frameworks. Frameworks are tested using the KUDO test harness from KEP-0008.

- Identify and build a common set of Frameworks and FrameworkVersions that can demonstrate/validate that functionality of the controller
exists in real world situations.
Framework tests live in the `kudobuilder/frameworks` repository, with the file structure defined in KEP-0010.

### Kubernetes clusters

It is important that tests are run against many different configurations of Kubernetes to ensure that KUDO and frameworks are compatible with common Kubernetes configurations and distributions.

Framework tests will be run against several different Kubernetes clusters:

- A local cluster using [kind](https://github.com/kubernetes-sigs/kind) or [k3s](https://github.com/rancher/k3s).
- A [GKE cluster](https://cloud.google.com/kubernetes-engine/).
- An [EKS cluster](https://aws.amazon.com/eks/).
- An [AKS cluster](https://docs.microsoft.com/en-us/azure/aks/).

These clusters can be started either as a part of CI jobs or maintained long term to be used across many CI jobs.

### CICD

Use [CircleCI](https://circleci.com/docs/) and the [GitHub Plugin](https://github.com/marketplace/circleci/plan/MDIyOk1hcmtldHBsYWNlTGlzdGluZ1BsYW45MA==#pricing-and-setup).
For OpenSource projects we will recieve 1,000 monthly build minutes. With the test suite below, that should suffice as a baseline.

For OpenSource projects we will receive 1,000 monthly build minutes. With the test suite below, that should suffice as a baseline.

#### Pull Requests

All Pull Requests into master need to have the following checks pass. These should be ordered in fastest to slowest to reduce the time spent when/if failures occur
All Pull Requests into master need to have the following checks pass. These should be ordered in fastest to slowest to reduce the time spent when/if failures occur.

0. Check author has signed CLA
1. `go fmt` does not change anything
1. `go lint` and `go vet` both pass
1. `make check-formatting passes.
1. All unit tests pass (with `-race` flag)
1. Dockerfile builds (this requires all dependencies in the vendor folder)
1. All integration tests pass.

Perform the same set of tests after merging master into the branch.

#### Master Branch

##### Tests

The master branch will run all the same tests that pull requests do as well as the complete set of framework tests.

##### Pushes

Once merged into master a build process occurs to deploy the latest image at `kudobuilder/controller:latest`
Expand All @@ -112,28 +153,20 @@ Since we don't have any tests that validate the image works (no integration test

#### Tags/Release

- All unit, integration, and framework tests are run.
- Build process occurs to deploy the image at `kudobuilder/controller:latest`.
- Create the YAML to deploy Kudo, and package up to store in GitHub Release

### User Stories

#### Story 1

As a developer, I want to ensure my changes don't break existing functionality, even if I don't understand all the capabilities
of Kudo.
As a developer, I want to ensure my changes don't break existing functionality, even if I don't understand all the capabilities of Kudo.

#### Story 2

As a repository owner, I don't want to have to validate the execution of common plans/functionality as part of the review process.

### Implementation Details/Notes/Constraints [optional]

- Are new tests required for all (code) PRs? If fixing a bug, it's re-assuring if you can provide a test that demonstrates it not working,
but the level of effort is significantly increased to push code. This might be counter productive in trying to encourage
developers from contributing.
- Being able to run the "Test Frameworks" on a running cluster might provide value (e.g. sonobouy), but requires a maturation
of that capability and might not be worth the effort

### Risks and Mitigations

- Complexity of a test and CICD system
Expand All @@ -145,8 +178,9 @@ How will we know that this has succeeded?

- When repository owners can feel confident that code changes are not breaking functionality.
- Tests pass for the API objects
- Leverage testing scaffolding provided by (and subsequently removed by us) Kubebuilder for
controller logic.
- Integration tests are run on PRs to KUDO.
- Framework tests are run on KUDO releases.
- Framework tests are run on PRs to Frameworks.

## Implementation History

Expand Down