Add Cluster filter on Kubernetes Overview ECS dashboard

[akshay-saraswat]

What does this PR do?

Added a Cluster name selector on the [Metricbeat Kubernetes] Overview ECS dashboard. This selector will be used as a filter for the dashboard. Also, added a blur for this selector on the Kubernetes module documentation page to explain how this selector works and where it sources the data from. Added a screenshot of the [Metricbeat Kubernetes] Overview ECS dashboard on the Kubernetes module documentation page because we need a visual to explain the Cluster filter.

Why is it important?

We have added a new Cluster name field in metricbeat. this field will not be useful unless users could discover it in some form
in UI.

Checklist

• I have made corresponding changes to the documentation
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Screenshots

[cla-checker-service]

💚 CLA has been signed

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2021-07-16T19:43:02.835+0000

• Duration: 75 min 4 sec

• Commit: a1540a2

Test stats 🧪

Test	Results
Failed	0
Passed	8935
Skipped	2466
Total	11401

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	8935
Skipped	2466
Total	11401

[ChrsMark]

Overall looks good to me, thanks for working on this @akshay-saraswat but I think the Dashboard screenshot needs to be replaced with a more representative one displaying nodes, deployments, etc as well as the usage of the cluster filter.

[ChrsMark]

lgtm!

Feel free to merge it once CI is green (I think you need to sign the CLA).

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[jsoriano]

Dashboard added here has some kind of issue, test_dashboards fails with:

E         Exiting: Failed to import dashboard: Failed to load directory /go/src/github.com/elastic/beats/metricbeat/build/system-tests/run/test_base.Test.test_dashboards/kibana/7/dashboard:
E           error loading /go/src/github.com/elastic/beats/metricbeat/build/system-tests/run/test_base.Test.test_dashboards/kibana/7/dashboard/Metricbeat-kubernetes-overview.json: returned 500 to import file: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."}
E         
E       assert 1 == 0

test_dashboards can be executed with PYTEST_ADDOPTS="-k test_dashboards" mage pythonIntegTest from the metricbeat directory.

It may be because it uses version 7.14.0 that hasn't been released yet, but not sure.

Why wasn't this detected by CI? These were the logs there:

[2021-07-16T20:05:02.815Z] >> python test: Integration-kubernetes Testing
[2021-07-16T20:05:02.815Z] Cache entry deserialization failed, entry ignored
[2021-07-16T20:05:20.912Z] >> python test: Integration-kubernetes Testing - No tests found.

It seems that selective testing was triggered as only one module was modified, and no test was executed in this case. test_dashboards should have been run to detect this issue.

@ChrsMark @akshay-saraswat could you please review this dashboard?

cc @sayden @kvch

[ChrsMark]

@jsoriano I think that Dashboard is ok. The backports to 7.x and 7.14 should work with 7.14 compatible Dasboards. I think that we miss the update of the testing stack, sth like #24957.

[jsoriano]

@jsoriano I think that Dashboard is ok. The backports to 7.x and 7.14 should work with 7.14 compatible Dasboards. I think that we miss the update of the testing stack, sth like #24957.

But test_dashboards is broken in the meantime in master and 7.x, see #26959 for example.

[ChrsMark]

I think that master, 7.x and 7.14 should be compatible with 7.14 dashboards and we can fix it with a change like #24957.

The weird thing is that this PR was green at https://beats-ci.elastic.co/blue/organizations/jenkins/Beats%2Fbeats/detail/PR-26919/5/pipeline, 🤔 .

In any case we need to update

beats/metricbeat/docker-compose.yml

Line 40 in 7f41ce0

image: docker.elastic.co/integrations-ci/beats-kibana:${KIBANA_VERSION:-7.12.0}-1

, no? I can prepare a PR for this if we agree.

[jsoriano]

The weird thing is that this PR was green at https://beats-ci.elastic.co/blue/organizations/jenkins/Beats%2Fbeats/detail/PR-26919/5/pipeline, thinking .

Yes, this is what I mention in #26919 (comment), it seems it was not detected in CI because only the python tests for kubernetes were executed, what doesn't include test_dashboards.

Why wasn't this detected by CI? These were the logs there:

[2021-07-16T20:05:02.815Z] >> python test: Integration-kubernetes Testing
[2021-07-16T20:05:02.815Z] Cache entry deserialization failed, entry ignored
[2021-07-16T20:05:20.912Z] >> python test: Integration-kubernetes Testing - No tests found.

I am looking now at how we can make dashboards tests to be executed for modules too.

In any case we need to update

beats/metricbeat/docker-compose.yml

Line 40 in 7f41ce0

image: docker.elastic.co/integrations-ci/beats-kibana:${KIBANA_VERSION:-7.12.0}-1

We cannot do this change till 7.14 is released. I think we need to revert the change on this dashboard, or make it work with the current tested versions. If not we are going to have broken builds till the release of 7.14.