Add latest k8s versions in testing

[ChrsMark]

What does this PR do?

This PR adds more k8s versions for testing.

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2021-07-07T14:10:41.056+0000

• Duration: 137 min 22 sec

• Commit: 5547284

Test stats 🧪

Test	Results
Failed	0
Passed	48617
Skipped	5266
Total	53883

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	48617
Skipped	5266
Total	53883

[ChrsMark]

It seems that default ports for proxy and scheduler have been changed in latest versions. We need to tune them accordingly for our test.

[ChrsMark]

I wonder why the config for proxy metricset targets 10252 port at

beats/metricbeat/module/kubernetes/test/integration.go

Line 74 in 64f6e80

"hosts": []string{"localhost:10252"},

, while the port should be 10249 like at

beats/metricbeat/module/kubernetes/_meta/config.reference.yml

Line 76 in 2a0e099

hosts: ["localhost:10249"]

.

Also we need to find a way to set the proper ports for newest versions of k8s for scheduler and controllermanger since pure http is not support anymore and we should target secure ports now (see https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/).

@jsoriano do you have any ideas on how we could check for what version we are running and set the config accordingly at

beats/metricbeat/module/kubernetes/test/integration.go

Line 79 in 64f6e80

func GetSchedulerConfig(t *testing.T, metricSetName string) map[string]interface{} {

?

[jsoriano]

@jsoriano do you have any ideas on how we could check for what version we are running and set the config accordingly?

More than checking for the version, we might explicitly provide it via environment variables or something, the job invoking the test knows the version, so it could set an environment variable, and then in the methods to get the config we can chose what config to use depending on the version.

Other option is to configure the services to listen always in the same ports, it is possible to customize the control plane with kubeadm, so invoking kind with something like this config might do the trick for the scheduler (not tested):

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: ClusterConfiguration
    scheduler:
      extraArgs:
        secure-port: "0"
        port: "10251"

Or we can try to enable always the new port, I guess that by setting the secure-port flag. But maybe this doesn't work if older versions don't have this flag.

(Take into account that it is not only a port change, but also a change from HTTP to HTTPS.)

[jsoriano]

It'd be nice to support 1.21 too 🙂

[ChrsMark]

kind version required an update to work with 1.21

[jsoriano]

Maybe we can start dropping some of the old versions from the matrix? I would leave the older one we support, but we can consider removing some of the others.

[ChrsMark]

I removed 1.15 and 1.16

[jsoriano]

@jsoriano do you have any ideas on how we could check for what version we are running and set the config accordingly?

I have been trying some things and we cannot use only the old port because the scheduler pod has a healthcheck for the new one.

Good thing about this is that, if needed, we can use the token in /var/run/secrets/kubernetes.io/serviceaccount/token to query the API and get the port from the health check.

This config works to start with both ports, also in the oldest version:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: ClusterConfiguration
    scheduler:
      extraArgs:
        bind-address: "0.0.0.0"
        port: "10251"
        secure-port: "10259"

[ChrsMark]

👍🏼 @jsoriano thanks! In integrations we access the endpoint using the token too -> https://github.com/elastic/integrations/blob/master/packages/kubernetes/data_stream/scheduler/_dev/test/system/test-default-config.yml#L7.

However there we test only with latest version. I think the patch for kind will do the trick!

[MichaelKatsoulis]

👍

[jsoriano]

LGTM if it looks good to jenkins 🙂

[ChrsMark]

For reference: Versions of k8s should be specified as per https://github.com/kubernetes-sigs/kind/releases/tag/v0.11.1