Auto-scale vtgate with HPA

[siadat]

Problem

We'd like to be able to auto-scale number of vtgate replicas using HorizontalPodAutoscaler.

Solution

In order to make a CRD scalable by HPA, it should define the Scale subresource.

We also make sure if a vtgate is being autoscaled, the replicas value is not copied from VitessCluster. Otherwise, autoscaled values will be overwritten.

Context

https://vitess.slack.com/archives/CNE9WP677/p1725015568336609

[frouioui]

Thanks for the contribution 🚀 Would it be possible to add an end-to-end test for this? We have examples of other features being covered by E2E tests under ./test/endtoend/.

[frouioui]

nit: should be 2024

Suggested change

	Copyright 2019 PlanetScale Inc.
	Copyright 2024 PlanetScale Inc.

[frouioui]

The ./test/endtoend/operator/operator-latest.yaml file we use for our tests must be updated with the newly compiled CRDs.

To do this, you can use the following command: kustomize build ./deploy > build/_output/operator.yaml. The output found in ./build/_output/operator.yamlcan be copied over the test file.

The changes made to the Deployment named vitess-operator will have to be reverted before pushing, as the new file you'll copy will override some defaults we use for our tests.

Finally, the changes made to ./test/endtoend/operator/operator-latest.yaml will have to be copied to the Vitess repository in a separate PR, into the file ./examples/operator/operator.yaml.

[frouioui]

nit picking here too, the comment should begin with HpaSpec, it throws a warning in the IDE otherwise

[frouioui]

The HorizontalPodAutoscaler resource is not something that the operator or even the ReconcileVitessCell object know about. It leads to the error I attached below and that can be found in vitess-operator's logs.

 pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v2.HorizontalPodAutoscaler: failed to list *v2.HorizontalPodAutoscaler: horizontalpodautoscalers.autoscaling is forbidden: User "system:serviceaccount:default:vitess-operator" cannot list resource "horizontalpodautoscalers" in API group "autoscaling" in the namespace "default"

We must give the ReconcileVitessCell object the proper information to start watching this resource: this can be done by adding *v2.HorizontalPodAutoscaler to the watchResources slice in the vitesscell pkg.

Moreover, the K8S roles needs to be updated to give the proper permission for K8S to communicate with the API, this can be done by adding the following snippet to the ./deploy/role.yaml file.

- apiGroups:
  - autoscaling
  resources:
  - horizontalpodautoscalers
  verbs:
  - '*'

[frouioui]

Without this change, creating a cluster does not fail as the vitess-operator can still run. But no HPA is created:

$ kubectl get hpa
No resources found in default namespace.

With this change however, it looks good, vitess-operator is healthy and we can see the HPA:

$ kubectl get hpa
NAME                            REFERENCE                                  TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
example-zone1-vtgate-bc6cde92   VitessCell/example-zone1-vtgate-bc6cde92   <unknown>/10%   1         10        0          33s

Note that the target is unknown because my metrics-server was disabled at the time.