Skip to content

Commit

Permalink
docs: improve and make clear info about setup envtest and its binaries
Browse files Browse the repository at this point in the history
  • Loading branch information
@camilamacedo86
camilamacedo86 committed Aug 17, 2024
1 parent 7b21ee0 commit 2e4b024
Show file tree
Hide file tree
Showing 3 changed files with 53 additions and 39 deletions.
5 changes: 3 additions & 2 deletions RELEASE.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,8 @@ This action will caall the job [./build/.goreleaser.yml](./build/.goreleaser.yml

### (Deprecated) - To build the Kubebuilder-tools: (Artifacts required to use ENV TEST)

> We are working on to move all out from GCP Kubebuilder project. As it fits as part of Controller-Runtime domain of responsability it is under a ongoing work to change the build of those binaries to controller-tools and how it is implemented in controller-runtime. For further information see the PR: https://github.com/kubernetes-sigs/controller-runtime/pull/2811
> We no longer build the artifacts and the promotion of those is deprecated. For more info
see: https://github.com/kubernetes-sigs/kubebuilder/discussions/4082

Kubebuilder projects requires artifacts which are used to do test with ENV TEST (when we call `make test` target)
These artifacts can be checked in the service page: https://storage.googleapis.com/kubebuilder-tools
Expand All @@ -86,7 +87,7 @@ For further information see the [README](https://github.com/kubernetes-sigs/kube

### (Deprecated) - To build the `kube-rbac-proxy` images:

> We no longer build and promote those images. For more info
> We no longer build the images and the promotion of those images is deprecated. For more info
see: https://github.com/kubernetes-sigs/kubebuilder/discussions/3907

These images are built from the project [brancz/kube-rbac-proxy](https://github.com/brancz/kube-rbac-proxy).
Expand Down
61 changes: 37 additions & 24 deletions docs/book/src/reference/artifacts.md
View file
Original file line number Diff line number Diff line change
@@ -1,44 +1,57 @@
# Artifacts

# Artifacts

<aside class="note warning">
<h1>IMPORTANT: Kubebuilder no longer produces artifacts</h1>
<h1>IMPORTANT: Action Required: Ensure that you no longer use https://storage.googleapis.com/kubebuilder-tools </h1>

**Artifacts provided under [https://storage.googleapis.com/kubebuilder-tools](https://storage.googleapis.com/kubebuilder-tools) are deprecated and Kubebuilder maintainers are no longer able to support, build, or ensure the promotion of these artifacts.**

Kubebuilder has been building those artifacts binaries to allow users
to use the [ENV TEST][env-test-doc] functionality provided by [controller-runtime][controller-runtime]
for several years. However, Google Cloud Platform has [deprecated the Container Registry](https://cloud.google.com/artifact-registry/docs/transition/transition-from-gcr),
which has been used to build and promote these binaries tarballs.
You will find the [ENVTEST][env-test-doc] binaries available in the new location from k8s release `1.28`, see: [https://github.com/kubernetes-sigs/controller-tools/blob/main/envtest-releases.yaml](https://github.com/kubernetes-sigs/controller-tools/blob/main/envtest-releases.yaml).
Also, binaries to test your controllers after k8s `1.29.3` will no longer be found in the old location.

Additionally, ongoing changes and the phase-out of the previous GCP infrastructure mean
that **Kubebuilder maintainers are no longer able to build or ensure the promotion of these binaries.**
**New binaries are only promoted in the new location**.

Therefore, since those have been building to allow the controller-runtime
[ENV TEST][env-test-doc] library to work, it has been started to be built by [controller-runtime][controller-runtime] itself
under the [controller-gen releases page][controller-gen]. From [controller-runtime][controller-runtime]
release `v0.19.0` the binaries will begin to be pulled out from this page instead.
For more information, see the PR that introduces this change [here](https://github.com/kubernetes-sigs/controller-runtime/pull/2811).
**You should ensure that your projects are using the new location.**
Please ensure you use `setup-envtest` from the controller-runtime `release v0.19.0` to be able to download those.
**This update is fully transparent for Kubebuilder users.**

The artefacts for [ENVTEST][env-test-doc] k8s `1.31` are exclusively available at: [Controller Tools Releases][controller-gen].

You can refer to the Makefile of the Kubebuilder scaffold and observe that the envtest setup is consistently aligned across all controller-runtime releases. Starting from `release-0.19`, it is configured to automatically download the artefact from the correct location, **ensuring that kubebuilder users are not impacted.**

```shell
ENVTEST_K8S_VERSION = 1.31.0
ENVTEST_VERSION ?= release-0.19
...
.PHONY: envtest
envtest: $(ENVTEST) ## Download setup-envtest locally if necessary.
$(ENVTEST): $(LOCALBIN)
$(call go-install-tool,$(ENVTEST),sigs.k8s.io/controller-runtime/tools/setup-envtest,$(ENVTEST_VERSION))
```
</aside>

To test your controllers, you will need to use the tarballs containing the required binaries:

Kubebuilder publishes test binaries and container images in addition
to the main binary releases.
```shell
./bin/k8s/
└── 1.25.0-darwin-amd64
├── etcd
├── kube-apiserver
└── kubectl
```

## **(Deprecated)** - Test Binaries (Used by ENV TEST)
These tarballs are released by [controller-tools](https://github.com/kubernetes-sigs/controller-tools),
and you can find the list of available versions at: [envtest-releases.yaml](https://github.com/kubernetes-sigs/controller-tools/blob/main/envtest-releases.yaml).

You can find test binary tarballs for all Kubernetes versions and host platforms at `https://go.kubebuilder.io/test-tools`.
You can find a test binary tarball for a particular Kubernetes version and host platform at `https://go.kubebuilder.io/test-tools/${version}/${os}/${arch}`.
When you run `make envtest` or `make test`, the necessary tarballs are downloaded and properly
configured for your project.

<aside class="note">
<h1>Setup ENV TEST tool</h1>
To know more about the tooling used to configure ENVTEST which is used in the setup-envtest target in the Makefile
of the projects build with Kubebuilder see the [README][readme]
of its tooling.
To learn more about the tooling used to configure ENVTEST, which is utilized in the `setup-envtest`
target in the Makefile of projects built with Kubebuilder, see the [README](https://github.com/kubernetes-sigs/controller-runtime/blob/main/tools/setup-envtest/README.md)
of its tooling. Additionally, you can find more information by reviewing the Kubebuilder [ENVTEST][env-test-doc] documentation.
</aside>


[env-test-doc]: ./envtest.md
[controller-runtime]: https://github.com/kubernetes-sigs/controller-runtime
[controller-gen]: https://github.com/kubernetes-sigs/controller-tools/releases
[readme]: https://github.com/kubernetes-sigs/controller-runtime/blob/main/tools/setup-envtest/README.md
26 changes: 13 additions & 13 deletions docs/book/src/reference/envtest.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,18 @@ Kubernetes API server, without kubelet, controller-manager or other components.
Installing the binaries is as a simple as running `make envtest`. `envtest` will download the Kubernetes API server binaries to the `bin/` folder in your project
by default. `make test` is the one-stop shop for downloading the binaries, setting up the test environment, and running the tests.

The make targets require `bash` to run.

You can refer to the Makefile of the Kubebuilder scaffold and observe that the envtest setup is consistently aligned across all controller-runtime releases. Starting from `release-0.19`, it is configured to automatically download the artefact from the correct location, **ensuring that kubebuilder users are not impacted.**

```shell
ENVTEST_K8S_VERSION = 1.31.0
ENVTEST_VERSION ?= release-0.19
...
.PHONY: envtest
envtest: $(ENVTEST) ## Download setup-envtest locally if necessary.
$(ENVTEST): $(LOCALBIN)
$(call go-install-tool,$(ENVTEST),sigs.k8s.io/controller-runtime/tools/setup-envtest,$(ENVTEST_VERSION))
```

## Installation in Air Gapped/disconnected environments
If you would like to download the tarball containing the binaries, to use in a disconnected environment you can use
Expand All @@ -25,7 +36,7 @@ make envtest
Installing the binaries using `setup-envtest` stores the binary in OS specific locations, you can read more about them
[here](https://github.com/kubernetes-sigs/controller-runtime/tree/master/tools/setup-envtest#where-does-it-put-all-those-binaries)
```sh
./bin/setup-envtest use 1.21.2
./bin/setup-envtest use 1.31.0
```

### Update the test make target
Expand All @@ -42,15 +53,6 @@ NOTE: The `ENVTEST_K8S_VERSION` needs to match the `setup-envtest` you downloade
no such version (1.24.5) exists on disk for this architecture (darwin/amd64) -- try running `list -i` to see what's on disk
```
## Kubernetes 1.20 and 1.21 binary issues
There have been many reports of the `kube-apiserver` or `etcd` binary [hanging during cleanup][cr-1571]
or misbehaving in other ways. We recommend using the 1.19.2 tools version to circumvent such issues,
which do not seem to arise in 1.22+. This is likely NOT the cause of a `fork/exec: permission denied`
or `fork/exec: not found` error, which is caused by improper tools installation.
[cr-1571]:https://github.com/kubernetes-sigs/controller-runtime/issues/1571
## Writing tests
Using `envtest` in integration tests follows the general flow of:
Expand Down Expand Up @@ -97,8 +99,6 @@ Ie,
├── etcd
├── kube-apiserver
└── kubectl
1 directory, 3 files
```
You can use environment variables and/or flags to specify the `kubectl`,`api-server` and `etcd` setup within your integration tests.
Expand Down

0 comments on commit 2e4b024

Please sign in to comment.