Refactor autodiscover to make it reusable in the Agent

[exekias]

Describe the enhancement:

We want to support autodiscover features in the Agent (#19225). For this, it would be nice if we could reuse the existing providers and make Agent handle the runner start/stop logic.

Current status

Right now, autodiscover providers send start/stop events through a bus. The autodiscover manager listens into this bus and manages input/module runners according to these events.

Providers are coupled to config generation (through templates or hints), although all this code is contained in libraries. It is used like this:

beats/libbeat/autodiscover/providers/docker/docker.go

Lines 307 to 315 in 93a4591

	// Try to match a config
	if config := d.templates.GetConfig(event); config != nil {
	event["config"] = config
	} else {
	// If no template matches, try builders:
	if config := d.builders.GetConfig(d.generateHints(event)); config != nil {
	event["config"] = config
	}
	}

I don't see any reason why cannot be done by autodiscover manager (and not the providers) ^

Proposal

To make providers reusable, we need to decouple them from the config generation / runner handling, a few things we would need to do:

• Refactor autodiscover to move the template logic away from providers, into the autodiscover manager.
• Re-define the schema for autodiscover events. For instance, we should include hints into them (instead of the generated config). It should be serializable (cannot carry things like a keystore or meta pointer)

Some caveats:

• We are currently using dynamic fields (a pointer we can update in memory) to update metadata without restarting runners, we may need to rethink this?
• Kubernetes autodiscover supports keystore (again, at the provider level), should this be moved into the autodiscover manager? We can leave it out of Agent for now.

Agent will implement a new template logic that can be hooked into provider events (see the parent issue for more details).

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[exekias]

@jsoriano @ChrsMark would love your thoughts on this one

[ChrsMark]

• Kubernetes autodiscover supports keystore (again, at the provider level), should this be moved into the autodiscover manager? We can leave it out of Agent for now.

Note that there are 2 Keystore types, file based one (or local) and k8s' Secret based one.
The local one is available in Autodiscover Manager, the other one is coupled with the Kubernetes Provider and it's not available in Autodiscover Manager. Speaking of this I wonder what would be the relationship of Manager-Provider with the new approach.

[jsoriano]

The plan sounds good to me, any common logic that we can move out from the providers sounds great.

[exekias]

The local one is available in Autodiscover Manager, the other one is coupled with the Kubernetes Provider and it's not available in Autodiscover Manager. Speaking of this I wonder what would be the relationship of Manager-Provider with the new approach.

It would be through the autodiscover events only. The providers will send start/stop events, but won't really know about the templates or what inputs get started.

I understand static secrets (the ones in templates) must be handled by the manager, but we will need to figure out how we make k8s secrets work when coming from hints.

[exekias]

I have implemented a Pod discovery provider here #21480, we are finally not reusing the whole autodiscover providers for this, but implementing things based on the common watcher code.

I expect more conversations to happen around the missing pieces, will open issues for them:

• Discovery of other resources (Node, service)
• Kubernetes secrets
• Hints based autodiscover
• Hopefully improve the logic for delayed stop of Filebeat inputs