[Automation] Improve testing experience with FTR

[dmlemeshko]

Summary

The purpose is to collect the pain points we face today in the test automation process and FTR as a solution specifically, discuss possible solutions and work on it.

Areas to improve/ issues

• Hard to use Page Objects files
  Some POs are way too large (vizualize_page.js, dashboard_page.js) and are not in TS yet, that slows down the process of adding/updating tests and keeping track of available functions.

  • proposal: split functions between several POs and services (visualize, visualize_editor, etc.) and convert current ones into TS.

• Current E2E tests are slow and repetitive in preconditions
  Most of the tests have similar preconditions that we do via UI (e.g. date picker setup). It does not only take time but also may introduce additional flakiness though it is not the main functionality to be tested in particular tests.

  • proposal: launch Kibana with custom state and avoid doing UI pre-conditions

• Improve FTR config type safety Improve FTR config type safety #69393

• Provide API to inspect FTR state Provide API to inspect FTR state #98311

[dmlemeshko]

@elastic/kibana-qa @spalger

[mshustov]

Other pain points:

• hard to understand the final config after all merges. Is it possible to add a flag to show the final config result for a file?
• no docs. The readme file is outdated.
• dynamic nature of getService(…) makes impossible to track deps. TS solves the problem partially.
• Security/Spaces models are not centralized. Should we document them and make reusable?

[mshustov]

Is it possible to use another assertion library in FTR? jest, for example. I understand it can be incompatible with a reporter or a test runner. Although it's the de-facto standard for our unit and integration tests.

[watson]

I'm cleaning up stale Developer Experience issues and came across this one. It's been a while since this issue was raised, and a lot have happened with FTR in the meantime. Is this still a problem, or can we close it?