Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] Implement queue-based approach to running kibana test suites #44934

Closed
brianseeders opened this issue Sep 5, 2019 · 1 comment
Closed
Assignees
Labels
Feature:CI Continuous integration impact:low Addressing this issue will have a low level of impact on the quality/strength of our product. loe:small Small Level of Effort Team:Operations Team label for Operations Team

Comments

@brianseeders
Copy link
Contributor

Re-visit this work after #44925 completes.

My initial testing for this saw full pipeline runtimes as low as 41 minutes, and this approach included creating a tarball of the entire post-build workspace and transferring it to other machines via GCS. With our latest pipeline work, we may actually be able to do this effectively without transferring workspaces, which could bring our pipeline time below 40 minutes.

General Idea (repeated for OSS and X-Pack):

  1. Create a list of all test suites (e.g. individual files with tests in them), along with the config file needed to run each suite
  2. Have multiple, parallel workers keep pulling from the queue of suites until all of the tests are complete
  3. Re-run any suite with failed tests, to combat flaky tests

Considerations

  • Invocations of FTR have 5-10 seconds of overhead, because of on-the-fly Babel transpiling, and this quickly adds up with lots of invocations for small test suites
  • Starting up Elasticsearch and Kibana are time consuming, so workers should not switch between suites requiring different test configs too much. Ideally, they also wouldn't spend 1 minute spinning up ES+Kibana to run 1 test suite for 5 seconds.
  • GitHub checks broken down by ciGroup wouldn't make sense anymore, as ciGroups will be gone. There are too many test suites (100s) to show them individually. What should be shown instead?
  • If we're still primarily using the Jenkins UI for status/etc, these changes will make the problems worse
  • Even some individual test suites take a long time to run (functional.apps.maps.es_geo_grid_source) and should be further broken up or otherwise prioritized
@brianseeders brianseeders added Team:Operations Team label for Operations Team Feature:CI Continuous integration labels Sep 5, 2019
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-operations

@tylersmalley tylersmalley added 1 and removed 1 labels Oct 11, 2021
@exalate-issue-sync exalate-issue-sync bot added impact:low Addressing this issue will have a low level of impact on the quality/strength of our product. loe:small Small Level of Effort labels Feb 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:CI Continuous integration impact:low Addressing this issue will have a low level of impact on the quality/strength of our product. loe:small Small Level of Effort Team:Operations Team label for Operations Team
Projects
None yet
Development

No branches or pull requests

3 participants