Selective testing in beats and test with latest version of the branch.

[ph]

Beats is a mono repo of many projects and has a lot of tests to run. When a PR is created, our CI will run the tests for all the beats even if no code was changed in that project. This lead to longer test time and bigger CI queues, it not uncommon to have to wait more than one or two hours when we are close to the feature freeze.

I would like to implement selective testings, what I mean by that is:

1. A change of code in Filebeat should only run the test in Filebeat.
2. A change in Auditbeat should only run the tests in Auditbeat
3. A change in Metricbeat should trigger a tests in Metricbeat and Auditbeat.
4. A change in libbeat should run the tests in all the beats.
5. A new vendor should trigger the tests in dependant beats.
6. A change in a module in Filebeat should only trigger the test_modules for that module.
7. A change in a module in Metricbeat should only run the test for that module.
8. When the docs are changed only run the doc tests.

To make the above work we probably need to either use a tool to get the dependency graph and generate what need to be executed or we could probably have a crude implementation purely based on path or maybe something that use both.

The other thing I would like to have is instead of only testing the PR code in the branch is to merge the PR of with the latest version of the target branch and then run the tests. When the PR tests are green and the PR is ready to merge it doesn't mean the branch will be green after the merge.

The above is based on discussion that @urso and myself had.

[ph]

We probably want to investigate baur, this is tool to build and tests monorepo project. It requires to keep states in a postgresql databased. I would prefer to have something stateless and easy to deploy and run locally.

[ruflin]

@ph The apm-server team is currently migrating to pipelines and I think we should too: elastic/apm-server#1466 It will remove lots of the dependency from JJB files and I think it will also allow things similar to what you suggest above. @kuisathaverat can perhaps comment on this?

[ph]

@ruflin I think both are complementarity.

[kuisathaverat]

Indeed they are, as better granularity, you have to run tests isolated, it makes easier to run then in parallel or filter which test you want to run based on files, path, labels, comments, whatever you want to check before to trigger a test, e.g for the APM UI we have a very similar issue, everytime we push something to kibana we have to wait for all the kibana-ci test to know if everything is OK, it is fine to do that, but we need also for us a quick set of checks/tests that should be run on the CI and are faster than run the whole CI tests, so we will run on our apm-ci a job that will identify our PRs based on the changes made on files and trigger a build that makes our tests.

[ph]

@kuisathaverat thanks for the details I will take a closer look.

[jsoriano]

+1 I also wanted to try something like this to avoid running all metricbeat modules even if nothing has changed for them.

The other thing I would like to have is instead of only testing the PR code in the branch is to merge the PR of with the latest version of the target branch and then run the tests. When the PR tests are green and the PR is ready to merge it doesn't mean the branch will be green after the merge.

Big +1 for this too, could you create another issue for that? I think they are two separated topics. For this we'd need an integration pipeline that tests and merges PRs one to one.

[ph]

@jsoriano Created another issue in #9524

[kuisathaverat]

The other thing I would like to have is instead of only testing the PR code in the branch is to merge the PR of with the latest version of the target branch and then run the tests. When the PR tests are green and the PR is ready to merge it doesn't mean the branch will be green after the merge.

Multibranch pipeline project make it, you can choose to build the PR merged with the master (default), only the PR release, or both

This is how to looks like when you choose both

[ruflin]

I started #9543 so we can start to play around with Pipelines for these things in PR's as soon as we have the setup for pipelines in Jenkins.

[kuisathaverat]

This is already implemented in the pipeline, we will continue making incremental improvements.