[discuss] Add support elastic agent policies and packages

[marc-gr]

For some time we have been investigating what could be the best approach to have E2E benchmark capabilities for the elastic-agent integrations. Some PoC work has been done to both extend current tooling to be able to work with elastic-agent and to figure out the best way to also extend elastic-package to be able to run these benchmarks.

A lot of the work required to happen in elastic-package (or in any new tool built for that purpose) would overlap with a lot of rally's functionality (mostly on reporting and being able to compare benchmark runs), and would most likely be lacking some of rally's other capabilities that could come as handy (for example setting up tracks with queries used in the dashboards built in the integrations).

For this could be a good idea to explore extending rally in a way that can deal with elastic-agent policies and work with corpora in a way that is useful for integrations.

This changes do not intend to make rally handle the required infrastructure to host elastic-agent but to allow policies to be part of a track definition and benefit from its reporting and querying capabilities along with its already existing support in all the performance related ecosystem.

An example of a possible track definition for an elastic-agent setup could be:

{
  "policies": [
    {
      "id": "an-id",
      "name": "A new policy",
      "description": "A description",
      "namespace": "default",
      "monitoring_enabled": ["logs", "metrics"],
    }
  ],
  "corpora": [
    {
      "name": "panw-logs",
      "documents": [
        {
          "base-url": "http://benchmarks.elasticsearch.org.s3.amazonaws.com/corpora/logs",
          "source-file": "panw-181998.json.bz2",
          "document-count": 2708746
        }
      ]
    }
  ],
  "operations": [
    {
      "policy-id": "an-id",
      "name": "install-panw",
      "operation-type": "install-package",
      "package": "panw",
      "version": "3.5.0",
      "input": "logfile",
      "vars": {},
      "data_stream": {
        "name": "panos",
        "vars": {
          "paths": ["/var/logs/panw.log"],
        }
      }
    },
    {
      "name": "log-write",
      "operation-type": "log-write",
      "corpora": "panw-logs",
      "rate": 10,
      "hosts": ["agent-host-1", "agent-host-2"],
      "target-file": "/var/logs/panw.log"
    }
  ]
}

This is adding some new elements:

• policy: defines a set of policies that can be used
• a new operation install-package that will configure the specific package
• a new operation log-write that will write the logs from corpora into a file at the specified hosts

This is just a first draft. Some things to be taken into account:

• The expectation is that agent hosts are created outside of the rally workflow
• A number of new operations to write logs in different ways (eg to a UDP or TCP socket, etc) to support the different inputs available to integrations will be required, or maybe this can be part of an external tool that rally can use.

With this additions to rally we would be able to:

• Extend elastic-package to launch tracks if needed
• Allow for better sizing capabilities for elastic-agent users.

[pquentin]

Hello, and thanks for considering Rally for your Elastic Agent benchmarking needs! I'll have to discuss more with the team, but it does look like at first sight that with our help you could add something like this to Rally without disrupting existing Elasticsearch users and reusing some of the Rally infrastructure.

Sorry for the naive questions, but I'm not familiar with Elastic Agent and Fleet at all.

• What are the prerequisites before running an Elastic Agent track like this? Is a target Elasticsearch cluster enough?
• Can you please explain how Rally would in practice run the above track? I guess elastic-package will install the package but I don't know how the log-write operation will work nor what it actually does. And I don't know how to define policies
• What metrics do you care about? I know there's the ingest pipelines telemetry device at least.
• Is there any relationship between this and the elastic-package benchmark command?

[marc-gr]

• What are the prerequisites before running an Elastic Agent track like this? Is a target Elasticsearch cluster enough?

An elasticsearch setup with a fleet server and a kibana instance. Also having the required elastic agent hosts installed to use if needed to send the corpora data instead to do it directly to ES.

• Can you please explain how Rally would in practice run the above track? I guess elastic-package will install the package but I don't know how the log-write operation will work nor what it actually does. And I don't know how to define policies

Both policy installation and package installation are done through the kibana fleet APIs

The operations to send the data are the ones that I am more unsure about, since integrations can use a number of different inputs (https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html), so maybe this complexity could be taken out to an external tool that rally can reference as a single operation.

• What metrics do you care about? I know there's the ingest pipelines telemetry device at least.

node stats, ingest pipelines and host metrics from the agent hosts (which I assume would use a new telemetry device from what you mention), also query performance of the dashboards, but I am pretty sure this is already in place

[marc-gr]

Is there any relationship between this and the elastic-package benchmark command?

that command support only synthetic benchmarks for pipelines, would be useful to extend it to be able to generate or use tracks like the ones described, for example, if this ends up being a thing

[aspacca]

hello, chiming in

that command support only synthetic benchmarks for pipelines, would be useful to extend it to be able to generate or use tracks like the ones described, for example, if this ends up being a thing

we are working on a elastic-package command (elastic/elastic-package#1110) in order to generate rally tracks, we'd need the same support for packages installation through fleet API discussed here