Restructure e2e suites layout

[mdelapenya]

The current state of the Fleet and Beats projects is not represented with the current suite layout. We have fleet, helm and k8s-autodiscover test suites, which can be triggered in isolation, even filtering by cucumber tags. But after having the elastic-agent extracted to a separate repo, then the execution of the suites should be changed in consequence, as it's no longer maintainable without effort: what changes trigger what suite.

This is an initial (and rudimentary) proposal:

1. the elastic-agent repo should trigger the fleet test suite, plus k8s-autodiscover suite for the elastic-agent

   • already covered in Include k8s-autodiscover for elastic-agent test suite in fleet nightly build #2140
   • there are scenarios that download binaries for metricbeat/filebeat (running the agent on top of them). We should verify if we need to run the e2e tests after changes on those beat dependencies. I suggest opening a discussion as it's not that trivial without rebuilding all projects each time.

2. the beats repo should trigger the helm charts test suite, plus k8s-autodiscover suite for not the elastic-agent. I wonder if there should be a case where a change on beats' k8s-autodiscover code should trigger all scenarios for k8s-autodiscover on the e2e. But again, because the elastic-agent project live in another repository, it's not that easy to correlate a commit on beats repo (where the core of the k8s-autodiscover feature lives) with a commit on the elastic-agent repo.

   • we could improve the way to filter the CI execution, which currently support selecting a comma-separated list of suites, adding the ability to include/exclude tags using Cucumber boolean operators for tags.

@jlind23 @ruflin @ph @cmacknz @adam-stokes @elastic/observablt-robots, thoughts?

[mdelapenya]

@v1v we would need to also think about current Beats strategy to trigger there e2e, as it always triggers all suites at the main pipeline, and a subset of them (probably outdated) at the packaging pipeline

[jlind23]

@mdelapenya I am quite sure that a change in a single beats can then break the Elastic Agent e2e tests which means that those tests suites should all be triggered.

[mdelapenya]

@mdelapenya I am quite sure that a change in a single beats can then break the Elastic Agent e2e tests which means that those tests suites should all be triggered.

Got it, thanks for the clarification.

Now that the elastic-agent is going to live in a separate repository, we have this question: what versions should be used in the e2e? Let me use examples:

commit: repo@sha

When a change in filebeat is committed, identified by `beats@0123456abcdef`
Then the e2e must be executed for filebeat `beats@0123456abcdef` and elastic-agent `elastic-agent@??`

When a change in elastic-agent is committed, identified by `elastic-agent@0123456abcdef`
Then the e2e must be executed for elastic-agent `elastic-agent@0123456abcdef` and beats `beats@??`

An initial (and simple) approach would be using the version of the stack:

• When beats is modified in main, use the elastic-agent version defined in the main branch for the e2e, which is incrementally modified by the automated bump.
• When beats is modified in 8.1, use the elastic-agent version defined in the 8.1 branch for the e2e, which is incrementally modified by the automated bump.

and for the agent, the same:

• When the elastic-agent is modified in main, use the beats version defined in the main branch for the e2e, which is incrementally modified by the automated bump.
• When the elastic-agent is modified in 8.1, use the beats version defined in the 8.1 branch for the e2e, which is incrementally modified by the automated bump.

[jlind23]

Yes you are right. Always use the last commit available on the matching branch seems to be the best option.

[ruflin]

+1 that for now we should test latest commit available with matching branch. The main downside is that builds are not fully reproducible. I expect a future where inputs are more decoupled and the testing is more similar done to the way we do it with Elasticsearch where we have a daily PR that updates the dependencies.

[mdelapenya]

Ok, with the current implementation provided by #2172, the variables GITHUB_CHECK_SHA1 and GITHUB_CHECK_REPO control what to be tested:

GITHUB_CHECK_SHA1	GITHUB_CHECK_REPO	Binaries for elastic-agent	Binaries for Beats	Use cases
not present (default empty)	not present (default elastic-agent)	Generated for unified release, alongside with stack	Generated for unified release, alongside with stack	Nightly builds Development of the e2e test project
SHA commit from elastic-agent repo	not present (default elastic-agent)	Generated from Fleet CI, using GCP bucket	Generated for unified release, alongside with stack	PRs on elastic-agent repo Merge commits on elastic-agent repo
SHA commit from elastic-agent repo	elastic-agent	Generated from Fleet CI, using GCP bucket	Generated for unified release, alongside with stack	PRs on elastic-agent repo Merge commits on elastic-agent repo
SHA commit from beats repo	beats	Generated for unified release, alongside with stack	Generated from Beats CI, using GCP bucket	PRs on beats repo Merge commits on beats repo

What does "Generated for unified release, alongside with stack" mean?

The project contains a .stack-version file that is automatically bumped with today's hashed snapshot, and this happen per maintenance branch. Because this automation happens in a PR, we know whether the given hashed snapshot is OK or not in terms of passing the e2e. If the tests pass, the bump PR is automatically merged.

With that in mind, a PR on the elastic-agent in the 7.17 branch will use 7.17-LAST_SUCCESSFUL_HASH-SNAPSHOT for the stack, and for the Beats used in scenarios (i.e. installing the agent on top of already running filebeat/metricbeat, or k8s-autodiscover for filebeat). For the scenarios where an elastic-agent must be installed, the binaries for that PR will be used.

What happens with old PRs on beats repo generating elastic-agent artifacts?

This situation happens for PRs created before the repo split, or with old PRs, generating elastic-agent artifacts. We would need to know what to do with them, as it would mean creating corner cases in the automation so that it's possible to consume the artifacts. We intentionally did not cover them to provide support for the new repo for the elastic-agent.

Have more questions?

Please let us know :)