[TSVB] Visualize runtime fields

[stratoula]

Describe the feature:
With the current implementation of TSVB, index patterns are not supported. This creates some limitations, for example, if we want to support runtime fields and save them to index patterns and not visualizations.

TSVB uses the low-level Search API and

• if the user doesn't specify an index pattern, loads the default
• user can specify multiple index patterns or all (*)
• there is a logic to handle rollup indexes
• has its own API to retrieve the fields for the index patterns that are requested from the users /api/metrics/fields
• the input that the user adds the index pattern(s) is a free text (in that way user can use wildcard or receive data from es indexes)

if the low-level Search API can support runtime fields this means that we should do some changes on the current implementation:

• Change the logic that we load the index patterns and fields
• Changes regarding the way that the runtime fields are going to be saved
• How are we going to handle the free type text?
• Can we be backward compatible?

if the runtime fields will be supported on SearchSource then this means that:

• The backend part of TSVB should be rewritten
• It will require many additions to the SearchSource API too.
• How are we going to handle the free type text?
• Can we be backward compatible?

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[stratoula]

@alexwizp I would love your thoughts on that.

[stratoula]

After syncing with @mattkime and @stacey-gammon it seems that:
There are two ways to support runtime fields:

• As Runtime Fields will be supported in the low-level search-api we could change the mechanism to also retrieve these fields. This means that we continue not supporting Index Patterns on TSVB and TSVB will have a different approach from all the other apps which may introduce inconsistencies.

Problems with this approach:
- Inconsistencies
- Can the TSVB runtime field be used in KQL? (filtering)

• Support index patterns on TSVB. This makes us consistent with the other apps but requires to rewrite the backend of TSVB to retrieve the fields from the index pattern.

Problems with this approach:
- Needs to change the way TSVB retrieve the fields
- Can we be backwards compatible? Right now, the user in TSVB can give index patterns that they don't exist anymore, or add wildcards or ES indices. Can we have this functionality if we switch to index patterns?

cc @mattkime

[stratoula]

We are going to proceed with enhancing the current implementation of TSVB. This means that we are not going to fully support index patterns right now, but we are going to enhance the low-level API to also retrieve the runtime fields and their definitions. TSVB already fetches the index patterns from the indexPatternString for:

• retrieving the default index pattern
• communicate its index patterns on a dashboard (in order filters to work correctly)

so it partially supports index patterns. Moreover, Timelion already does something similar for the scripted fields.

What needs to be done is to change the fields API to also retrieve the index pattern (from the string), recognize if it is a runtime field and get its definitions. In case of a runtime field, the definitions should be populated on the queries.

TSVB won't provide the edit runtime functionality. It will only visualize runtime fields. The main reason for that is that the fields are populated on a combobox so it is not user friendly to have an edit button there.

One important note: The API to retrieve the definitions for a runtime field is available on OSS so this won't cause any problems.

With that being said, I will remove the discuss label and slightly change the title.

[alexwizp]

From my perspective, before we start implementing support of Runtime Fields we should refactor TSVB to fully use Kibana Index Patterns instead of ES indexes. I offer the following UI changes:

1. EuiFieldText should be changed to EuiComboBox (single mode view, create custom options should be disabled)
   

2. Create index pattern button should be added upper to EuiComboBox.
   

3. In case the user does not find the required index in the list, he should click on the Create index pattern. This action open new Flyout component where new Index Pattern can be created.
   

Some code can be probably reused from Stack Management -> Index Patterns -> Create Index Pattern.

Open question:

1. How to correctly migrate existing TSVB visualizations. I see two options:

   • creating all needed indexes through adding a new migration script. But our existing migration framework doesn't allow to create a new saved objects!
   • creating needed index on opening TSVB visualization with showing an info message to user.

2. Permissions: Create index pattern` button should be available only for users with the corresponding permissions for creating index patterns

3. To simplify this approach instead of showing Flyout component we can redirect users to Stack Management -> Index Patterns -> Create Index Pattern.

^^ @timroes @stratoula

[stratoula]

We decided to enable a dual mode on TSVB. This means that it will support both index patterns and strings. Runtime fields will be available on the indexpattern mode. More information here

[timroes]

Blocked on #91367