Add kubernetes composable inputs provider #21480

exekias · 2020-10-02T14:45:44Z

What does this PR do?

Add kubernetes dynamic provider for Pods, this will allow to launch input configurations for discovered Pods in a Kubernetes cluster. Example config:

providers.kubernetes:
  kube_config: /Users/exekias/.kube/config
  node: minikube

inputs:
  - type: logfile
    streams:
      - paths: /var/log/containers/*${kubernetes.container.id}.log

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~[ ] I have made corresponding changes to the documentation~~
~~[ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Closes #19271

elasticmachine · 2020-10-02T15:16:08Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Pull request #21480 updated]
Start Time: 2020-10-06T14:25:07.403+0000
Duration: 37 min 22 sec

Test stats 🧪

Test	Results
Failed	0
Passed	1386
Skipped	4
Total	1390

ChrsMark · 2020-10-05T12:37:09Z

x-pack/elastic-agent/pkg/composable/providers/kubernetes/kubernetes.go

+
+	p.logger.Debugf("Initializing a new Kubernetes watcher using node: %v", p.config.Node)
+
+	watcher, err := kubernetes.NewWatcher(client, &kubernetes.Pod{}, kubernetes.WatchOptions{


Do we have in mind how/if we will handle different resources other than Pods? Like Services and Nodes?

Not yet, I'm only adding Pods in this PR as a first step. I will open an issue to discuss more resources

WIP

elasticmachine · 2020-10-05T21:02:21Z

Pinging @elastic/integrations-platforms (Team:Platforms)

blakerouse · 2020-10-05T21:17:40Z

x-pack/elastic-agent/pkg/composable/providers/kubernetes/kubernetes.go

+	client, err := kubernetes.GetKubernetesClient(p.config.KubeConfig)
+	if err != nil {
+		// info only; return nil (do nothing)
+		p.logger.Infof("Kubernetes provider skipped, unable to connect: %s", err)


Being that this will occur on most hosts not in Kubernetes, should this be Debugf?

good point, I added a message for the cases where this succeeds

blakerouse · 2020-10-05T21:26:18Z

x-pack/elastic-agent/pkg/composable/providers/kubernetes/kubernetes.go

+	composable.Providers.AddDynamicProvider("kubernetes", DynamicProviderBuilder)
+}
+
+type kubernertesContainerData struct {


This struct seems to be unused.

blakerouse · 2020-10-05T21:28:52Z

x-pack/elastic-agent/pkg/composable/providers/kubernetes/kubernetes.go

+	// Emit all containers in the pod
+	p.emitContainers(pod, pod.Spec.Containers, pod.Status.ContainerStatuses)
+
+	// TODO deal with init containers stopping after initialization


Whats the TODO mean here? Does this mean that AddOrUpdate will be called for InitContainers but the Remove will never be called? That seems like it will leak then in the provider, if that is correct.

No, right now Remove will be called when the Pod stops, so inputs won't leak.

The thing is that these containers may stop before the Pod itself, we should be able to detect this and stop them earlier.

elasticmachine · 2020-10-05T21:28:59Z

Pinging @elastic/ingest-management (Team:Ingest Management)

jsoriano

This is looking good, I like how providers can be implemented in the agent.

Is there a place to add the documentation?

x-pack/elastic-agent/pkg/composable/providers/kubernetes/kubernetes.go

blakerouse

Looks good. Seems like a good start.

blakerouse · 2020-10-06T13:09:33Z

x-pack/elastic-agent/pkg/composable/providers/kubernetes/kubernetes.go

+	// Emit all containers in the pod
+	p.emitContainers(pod, pod.Spec.Containers, pod.Status.ContainerStatuses)
+
+	// TODO deal with init containers stopping after initialization


ph · 2020-10-06T13:52:45Z

@exekias looks good what are the remaining steps/next steps for this?

exekias · 2020-10-06T14:25:16Z

Thanks for the reviews! @ph my plan was to get this in for 7.10 and open some follow up issues to discuss next steps. Does that sound good to you?

ph · 2020-10-06T14:27:23Z

I am ok to merge it as is and do followup after.

* Add kubernetes composable inputs provider (cherry picked from commit d6ee968)

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 2, 2020

exekias requested a review from blakerouse October 2, 2020 14:45

exekias added the Team:Platforms Label for the Integrations - Platforms team label Oct 2, 2020

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 2, 2020

exekias requested review from jsoriano and ChrsMark October 5, 2020 09:59

ChrsMark reviewed Oct 5, 2020

View reviewed changes

Add kubernetes composable inputs provider

9875eb9

WIP

exekias force-pushed the kubernetes-provider branch from e608e99 to 9875eb9 Compare October 5, 2020 15:09

Cleanup

bd3d009

exekias force-pushed the kubernetes-provider branch from 0f75d9f to bd3d009 Compare October 5, 2020 20:37

Align namespace field to beats

7922533

exekias added the enhancement label Oct 5, 2020

exekias marked this pull request as ready for review October 5, 2020 21:02

blakerouse reviewed Oct 5, 2020

View reviewed changes

botelastic bot added the Team:Ingest Management label Oct 5, 2020

exekias mentioned this pull request Oct 6, 2020

Refactor autodiscover to make it reusable in the Agent #19271

Closed

exekias added 2 commits October 6, 2020 10:11

Cleanup and better logging

f04a21f

Add changelog

a3e1be2

jsoriano reviewed Oct 6, 2020

View reviewed changes

x-pack/elastic-agent/pkg/composable/providers/kubernetes/kubernetes.go Show resolved Hide resolved

blakerouse approved these changes Oct 6, 2020

View reviewed changes

Merge branch 'master' into kubernetes-provider

d32e03f

exekias merged commit d6ee968 into elastic:master Oct 6, 2020

exekias mentioned this pull request Oct 6, 2020

Cherry-pick #21480 to 7.x: Add kubernetes composable inputs provider #21586

Merged

4 tasks

exekias added the v7.10.0 label Oct 6, 2020

exekias pushed a commit to exekias/beats that referenced this pull request Oct 6, 2020

Add kubernetes composable inputs provider (elastic#21480)

1ec0809

* Add kubernetes composable inputs provider (cherry picked from commit d6ee968)

exekias pushed a commit that referenced this pull request Oct 6, 2020

Add kubernetes composable inputs provider (#21480) (#21586)

5feee4a

* Add kubernetes composable inputs provider (cherry picked from commit d6ee968)

ph mentioned this pull request Oct 15, 2020

[Elastic Agent] Add documentation for the k8s provider #21848

Closed

ChrsMark mentioned this pull request Jan 26, 2021

[Elastic Agent] [Meta] Gathering logs and metrics from Kubernetes without using fleet generated agent config and without central agent management via fleet #23613

Closed

19 tasks

This was referenced Feb 19, 2021

[Agent] Add agent standalone manifests for system module & Pod's log collection #23938

Merged

Add agent_on_k8s docs and k8s provider elastic/observability-docs#401

Merged

This was referenced Mar 11, 2021

Add complete k8s metadata through composable provider #24495

Closed

Add more resources in kubernetes composable provider #24496

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add kubernetes composable inputs provider #21480

Add kubernetes composable inputs provider #21480

exekias commented Oct 2, 2020 •

edited by jsoriano

Loading

elasticmachine commented Oct 2, 2020 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

ChrsMark Oct 5, 2020

exekias Oct 5, 2020

elasticmachine commented Oct 5, 2020

blakerouse Oct 5, 2020

exekias Oct 6, 2020

blakerouse Oct 5, 2020

blakerouse Oct 5, 2020

exekias Oct 6, 2020

blakerouse Oct 6, 2020

elasticmachine commented Oct 5, 2020

jsoriano left a comment

blakerouse left a comment

blakerouse Oct 6, 2020

ph commented Oct 6, 2020

exekias commented Oct 6, 2020

ph commented Oct 6, 2020


		p.logger.Debugf("Initializing a new Kubernetes watcher using node: %v", p.config.Node)

		watcher, err := kubernetes.NewWatcher(client, &kubernetes.Pod{}, kubernetes.WatchOptions{

Add kubernetes composable inputs provider #21480

Add kubernetes composable inputs provider #21480

Conversation

exekias commented Oct 2, 2020 • edited by jsoriano Loading

What does this PR do?

Checklist

Related issues

elasticmachine commented Oct 2, 2020 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Oct 5, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Oct 5, 2020

jsoriano left a comment

Choose a reason for hiding this comment

blakerouse left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ph commented Oct 6, 2020

exekias commented Oct 6, 2020

ph commented Oct 6, 2020

exekias commented Oct 2, 2020 •

edited by jsoriano

Loading

elasticmachine commented Oct 2, 2020 •

edited by jenkins-beats-ci bot

Loading