Allow AutoDiscover To Work With External Configuration In Kubernetes Environments

[MorrieAtElastic]

Describe the enhancement:
Allow users to specify autodiscover parameters in directories referenced for external configuration.

Describe a specific use case for the enhancement or feature:
This enhancement request arose from a Support Ticket where a customer attempted to do exactly that: include his autodiscovery parameters in a set of files added to the directory specified by his external configuration parameter (~modules.d in his case). We eventually determined that specification of autodiscovery information within the context of external discovery is not possible. According to the internal discussion with developers:

"
Under metricbeat.config.modules tier only module configuration can be attached so as the final result to be sth like:

metricbeat.config.modules:
  - module: istio
    metricsets: ["galley"]
    period: 10s
    hosts: ["localhost:15014"]

What they are trying to do will result into something like:

metricbeat.config.modules:
  metricbeat.autodiscover:
  ....

which is not a valid configuration.
"

So the purpose of this enhancement is allow the user's construct to be valid. The user who opened the ticket offers the following comment:

"
We have a list of tenants that we scrape within k8s and there are at least 30 tenants. The reason why we initially moved to loading of external configs was for easier management of each tenant across multiple environments (directory paths).By using this directory format, we could load a specific set of tenant configs based on environments dynamically. This is the sole reason why we attempted the external configs route.

Our only alternative is to use the metricbeat.yml file which means we require a long config file that will be about 2000 lines of code, making it harder for users to maintain in the long run via PRs.
"

For all of the reasons which the user specified, allowing autodiscovery to be defined within the contest of external configuration would allow for complex autodiscovery parameter sets to be stored outside of the primary yml file; it would allow for much easier maintenance of individual sets of autodiscovery parameters and would provide a much more elegant way to configure autodiscover in complex systems.

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[jsoriano]

Hey @MorrieAtElastic,

Thanks for opening this enhancement request.

I am worried that this feature could increase the complexity of autodiscover configuration and cause some confusion. We already have two ways of configuring autodiscover (templates and hints), that can be combined, and this already causes some confusion. Adding a third way of configuring autodiscover can be even more confusing.

Also, something like this may be covered by Agent. In Agent, autodiscover configurations are defined as any other input (with the called dynamic inputs, see #19225). So any method of composing configuration in Agent would also benefit autodiscover.

We have a list of tenants that we scrape within k8s and there are at least 30 tenants. The reason why we initially moved to loading of external configs was for easier management of each tenant across multiple environments (directory paths).By using this directory format, we could load a specific set of tenant configs based on environments dynamically. This is the sole reason why we attempted the external configs route.

For this exact use case, beats support hints-based autodiscover, that allows to define monitoring configurations in annotations in the monitored Kubernetes resources, so each tenant can define their own configuration. You can read more about this in the following links:

• https://www.elastic.co/blog/docker-and-kubernetes-hints-based-autodiscover-with-beats (can be a bit outdated, but it explains well the idea)
• https://www.elastic.co/guide/en/beats/metricbeat/7.9/configuration-autodiscover-hints.html
• https://www.elastic.co/guide/en/beats/filebeat/7.9/configuration-autodiscover-hints.html

I think that having more general composable configuration files could be good, not only for autodiscover, but it has many implications, and as mentioned in #17842 (comment), we are not planning to add this in Beats.

// cc: @exekias

[exekias]

I agree here, can we zoom in a little bit into why they need ~2000 lines to configure autodiscover for they needs? It would be interesting to see how that config looks

[MorrieAtElastic]

In answer to @exekias' question: as stated previously, the customer has 30+ tenants for whom they have to generate autodiscovery parameters. I did not count, but the customer claims that this totals out at 2000 lines.

I will pass your responses back to the customer.

[MorrieAtElastic]

@exekias , @jsoriano

After I discussed your response with the customer and after I closed this enhancement, customer came back with the following comments:

"
Please pass on my thanks to the developers who have provided alternatives for our use case. However, I see that hints based autodiscover requires specific k8s pod annotations which means tenants would have to perform some config on their side.

Since we are providing a service to our own customers (tenants), our ethos is to enable seamless onboarding for all of our customers onto our self-hosted ECE. This means no configuration on their side since that would have to be embedded into their own sprints etc, which can effect their roadmap/deadlines

In addition to the above, the business mentioned that we were not able to use daemonsets. Hence why we were only left with the MB deployment option without the use of hints based autodiscover.
"

Any thoughts on these comments? btw: customer did make the point early in the ticket that he was not in a position to user daemonsets. Is the customer's use case a common use case?

[jsoriano]

Is the customer's use case a common use case?

It is somehow common, this is why Beats support the scope: cluster option.

What looks to me perhaps not so frequent is how they are managing the configurations for each tenant. Beats support two main use cases depending on who manages the monitoring configuration:

• If it is managed by the Kubernetes cluster administrator, then they can place the configuration in the main configuration file.
• If it is managed by teams deploying apps in the Kubernetes cluster, then they can place the configuration in hints, in the annotations of the resources they manage.

Here it seems that they have a mix. They have multiple tenants with their own configurations, but they still they want to centrally manage the configuration.

It occurs to me that one approach to solve this could be to generate the configuration with some external automation tool before starting Metricbeat. They could have the main configuration file, and a set of files with the configurations for each tenant, and in some automated step the configs get combined into a single configuration file, that is the one used by Metricbeat at the end.
The automation step could be part of their configuration management tooling, or could be a custom script in an init container of the Metricbeat pod.
Could something like this work for them?

[jsoriano]

@MorrieAtElastic I am going to close this by now. I think the motivations are valid, and it could make sense to have this feature, but I am afraid that we are not going to implement it. Summarizing the main reasons from my previous comments:

• Implementing this could increase the complexity of a feature that has already many options.
• In previous similar discussions about having more options for composable configuration files we decided not to do something like this for specific features.
• For this specific use case, I think that it should be possible to workaround the issue with external automation or configuration management practices.
• Longer term, something like this may be covered by Agent, where autodiscover templates are configured as normal inputs (dynamic inputs).