external metrics error

[dudicoco]

Hi,

We've upgraded from v0.0.5 to v0.1.9 and changed the hpa spec to the following:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: my-hpa
  annotations:
    metric-config.external.http-requests-total.prometheus/interval: 30s
    metric-config.external.http-requests-total.prometheus/query: |
      http_requests_total{release="my-deploy"}
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-deploy
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: External
    external:
      metric:
        name: http-requests-total
        selector:
          matchLabels:
            type: prometheus
      target:
        type: AverageValue
        averageValue: 1

We are seeing the following in the logs:

time="2021-01-21T15:23:21Z" level=info msg="Event(v1.ObjectReference{Kind:\"HorizontalPodAutoscaler\", Namespace:\"mynamespace\", Name:\"my-deploy\", UID:\"1046055a-9df7-4286-bc27-f3b74713a5db\", APIVersion:\"autoscaling/v2beta2\", ResourceVersion:\"124405892\", FieldPath:\"\"}): type: 'Warning' reason: 'CreateNewMetricsCollector' Failed to create new metrics collector: no plugin found for {External {http-requests-total &LabelSelector{MatchLabels:map[string]string{type: prometheus,},MatchExpressions:[]LabelSelectorRequirement{},}}}"

When using the latest image tag the error is gone.

In addition, it seems that kube-metrics-adapter is still able to collect metrics from hpas in API version autoscaling/v2beta1 , although this API is not supported in the new version.

[mikkeloscar]

@dudicoco unfortunately the docs are not aligned with the latest release (they are newer). Will try to cut a new release soonish.

In addition, it seems that kube-metrics-adapter is still able to collect metrics from hpas in API version autoscaling/v2beta1 , although this API is not supported in the new version.

True, we don't block this, but some feature won't be available if you define your HPA in autoscaling/v2beta1 format.

[dudicoco]

@mikkeloscar thanks!

Can you please elaborate on which features won't be available in autoscaling/v2beta1?
We would like to plan the best course of action for upgrading to the latest kube-metrics-adapter + autoscaling/v2beta2, ideally it would be great if we could first upgrade kube-metrics-adapter without it breaking our current hpas, and later on upgrade the hpas to autoscaling/v2beta2.

[szuecs]

@mikkeloscar I added a PR to always release merge to master as stackset-controller

[mikkeloscar]

@dudicoco I don't think I can list them all, but my proposal for you would be to look at the docs for the release v0.1.9: https://github.com/zalando-incubator/kube-metrics-adapter/tree/v0.1.9#kube-metrics-adapter

These are still in autoscaling/v2beta2 format, but basically anything you can't express in autoscaling/v2beta1 would not work. I believe in this version you can express everything, the format is just a bit different.
You can also check this commit to see how we changed from autoscaling/v2beta1 to autoscaling/v2beta1. May help understand the migration: 9d78fff#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5

[dudicoco]

@mikkeloscar thank you.

It seems that k8s somehow makes both api versions available within the cluster, and automatically converts to the correct format.

If you run kubectl -n my-namyspace get hpa.v2beta1.autoscaling and kubectl -n my-namespace get hpa.v2beta2.autoscaling you get the same list of hpas in both versions, this is without me deploying any hpa resources in API v2beta2.

Comparing the different versions, each has the correct syntax for that API version:

$ kubectl -n my-namespace get hpa.v2beta1.autoscaling my-hpa -o yaml

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  annotations:
    metric-config.external.prometheus-query.prometheus/requests_per_seconds: |
      sum(rate(http_requests_total{app="my-app"}[2m]))
  name: my-hpa
  namespace: my-namespace
spec:
  maxReplicas: 120
  metrics:
  - resource:
      name: memory
      targetAverageUtilization: 80
    type: Resource
  - external:
      metricName: prometheus-query
      metricSelector:
        matchLabels:
          query-name: requests_per_seconds
      targetAverageValue: "13"
    type: External
  - resource:
      name: cpu
      targetAverageUtilization: 80
    type: Resource
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app

$ kubectl -n my-namespace get hpa.v2beta2.autoscaling my-hpa -o yaml

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  annotations:
    metric-config.external.prometheus-query.prometheus/requests_per_seconds: |
      sum(rate(http_requests_total{app="my-app"}[2m]))
  name: my-hpa
  namespace: my-namespace
spec:
  maxReplicas: 120
  metrics:
  - resource:
      name: memory
      target:
        averageUtilization: 80
        type: Utilization
    type: Resource
  - external:
      metric:
        name: prometheus-query
        selector:
          matchLabels:
            query-name: requests_per_seconds
      target:
        averageValue: "13"
        type: AverageValue
    type: External
  - resource:
      name: cpu
      target:
        averageUtilization: 80
        type: Utilization
    type: Resource
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app

It seems like this makes the two versions somewhat backwards compatible.

[mikkeloscar]

@dudicoco indeed. That is what I meant. The only problem would be if there is a feature used from autoscaling/v2beta2 which you could not express in autoscaling/v2beta1, but I don't think that is the case with the mentioned version.