[New Platform] 8.0 Refactor kibana.yml

[rudolf]

2019-11-19 Updated based on comments

Core's config service uses "paths" to split up kibana.yml into sub-config's with their own schema. We want sub-configs to be grouped by domain, but some of the existing config prefix paths don't correspond to any domains in Core.

For instance (not a complete list):

1. There's no "migrations" domain, these config options belongs to the Saved Objects domain:
   • migrations.scrollDuration - The scroll value used to read batches of documents from the source index. Defaults to 15m.
   • migrations.batchSize - The number of documents to read / transform / write at a time during index migrations
   • migrations.pollInterval - How often, in milliseconds, secondary Kibana instances will poll to see if the primary Kibana instance has finished migrating the
2. The "kibana" domain should be reserved for global configuration options:
   • kibana.enabled remove
   • kibana.defaultAppId deprecated
   • kibana.disableWelcomeScreen moved to home plugin
   • kibana.autocompleteTerminateAfter move to data plugin
   • kibana.autocompleteTimeout move to data plugin
   • path.data move to be under "kibana"
3. some properties should be moved from their legacy namespace:
   • server.uuid should move outside of the server namespace

This refactoring would be a breaking change so it will have to be introduced in 8.0.0. We should audit all of our configuration options and coordinate this with all teams to ensure that we're satisfied with the end result for all of them.

[elasticmachine]

Pinging @elastic/kibana-platform

[legrego]

kibana.index move into saved objects domain as saved_objects.index

This one might get tricky, as there are other plugins which use this field to derive their own data which isn't strictly related to saved objects.

For example, the Security plugin uses the kibana.index value to represent the "tenant" when registering application privileges with elasticsearch. The application name that we register is kibana-${kibana.index}. This allows for multiple kibana installations to use the same ES cluster, while maintaining their own index and privilege/permission sets.

I think the alerting team is also planning on using this value to derive the name of one of their indices, in order to support multi-tenant setups, and perhaps other features.

I'm not sure what domain this would fall under...core? But core.index doesn't make a whole lot of sense for the typical Kibana administrator, who doesn't know/care about how the internals of Kibana are structured.

[joshdover]

I'm not sure what domain this would fall under...core? But core.index doesn't make a whole lot of sense for the typical Kibana administrator, who doesn't know/care about how the internals of Kibana are structured.

You could definitely argue that kibana is actually a good namespace for "global config" values. If that's what we want to go with, there's other namespaces that maybe should be moved underneath this one or combined with this one. For instance path.data

[rudolf]

As part of this refactor it might be worth switching from camel case to snake case convention for consistency across the stack #7444 (thanks @azasypkin)

[lukasolson]

Throwing this out there: elasticsearch.shardTimeout should probably be removed from the Elasticsearch core service because it's not even used by that service. It should be renamed to something like kibana.searchTimeout and exposed in the same manner as other kibana.* settings.

[lukasolson]

kibana.autocompleteTerminateAfter move?
kibana.autocompleteTimeout move?

These should probably be moved into the data plugin/domain.

[joshdover]

For clarity: we are not planning on working on this until after the Platform migration work has been completed (at the earliest). In the meantime, the workaround outlined in #46240 will be the near-term solution.

@lukasolson We'll make sure to include the config keys you listed above and it should be very easy to add additional options in the workaround mechanism in the future if we find more.

[pgayvallet]

There's no "migrations" domain, these config options belongs to the Saved Objects domain:

• migrations.scrollDuration - The scroll value used to read batches of documents from the source index. Defaults to 15m.
• migrations.batchSize - The number of documents to read / transform / write at a time during index migrations
• migrations.pollInterval - How often, in milliseconds, secondary Kibana instances will poll to see if the primary Kibana instance has finished migrating the

I also see a migrations.skip property in src/core/server/saved_objects/saved_objects_config.ts

Should the migrations namespace be renamed to saved_objects ? Or maybe moved under saved_objects?

migrations.scrollDuration -> saved_objects.migrations.scroll_duration ? (or camel case, not sure if camel to snake is in the scope of the issue)

kibana.enabled remove

Seems unused. should probably remove yes

kibana.defaultAppId move?

This seems used by multiple plugins. So I guess this should stay there?

kibana.disableWelcomeScreen move?

Should move to home plugin I think. home.disable_welcome_screen ?

kibana.autocompleteTerminateAfter move to data plugin
kibana.autocompleteTimeout move to data plugin

These are used in src/legacy/core_plugins/input_control_vis/public/control/list_control_factory.js and src/legacy/core_plugins/kibana/server/routes/api/suggestions/register_value_suggestions.js

should is really moves to data plugin?

path.data move to be under "kibana"

kibana.data_path then ? kibana.data_folder ?

some properties should be moved from their legacy namespace:

• server.uuid should move outside of the server namespace

Unclear where this should move to, as it's quite an isolated property.

• Could move to instance.uuid but that force to creates a namespace for a single property.
• Could move to core.uuid (do we have other properties going under core / do we want a 'core' namespace?)
• Could move to kibana.instance_uuid ?
• Should we keep it there for now?

[jbudz]

Some of these are stack globals, so that should be taken into consideration too. path.data

For settings that mean the same thing I'd push for consistency.

[mshustov]

Throwing this out there: elasticsearch.shardTimeout should probably be removed from the Elasticsearch core service because it's not even used by that service. It should be renamed to something like kibana.searchTimeout and exposed in the same manner as other kibana.* settings.

@lukasolson shouldn't it be a part of data plugin as well?

[lukasolson]

@restrry Yeah, it would make more sense there.

[flash1293]

@pgayvallet I started a separate issue for Kibana app related configs: #54497

@nickofthyme Is working on some of those in #55937

kibana.defaultAppId move?

This seems used by multiple plugins. So I guess this should stay there?

defaultAppId is becoming obsolete once we migrated all of the apps currently bundled in the "kibana" app (Home, Discover, Visualize, Dashboard, Management, Dev Tools) - See also #54088 IMHO we should keep those with a deprecation warning till then.

kibana.disableWelcomeScreen move?

Should move to home plugin I think. home.disable_welcome_screen ?

👍

kibana.autocompleteTerminateAfter move to data plugin
kibana.autocompleteTimeout move to data plugin

These are used in src/legacy/core_plugins/input_control_vis/public/control/list_control_factory.js ?> and src/legacy/core_plugins/kibana/server/routes/api/suggestions/register_value_suggestions.js

should is really moves to data plugin?

I think it makes sense. Mid-term we can probably refactor input control vis to leave it up to the data fetching layer to set terminate after and timeout. Short-term we could expose it in the contract of the data plugin so other plugins can also make use of it but it's clear the data plugin owns it.

[flash1293]

@lukeelmers Is the app arch team content with moving kibana.autocompleteTerminateAfter and kibana.autocompleteTimeout into the data plugin? If yes, is there already an issue for it? I had to implement a workaround in #63443 for this and would like to reference properly.

[lukeelmers]

@flash1293 Thanks for the ping; yes moving those settings to the data plugin makes the most sense to me.

@rudolf Is there a plan yet for when the other items might be moved, so that we can coordinate? I've created #66085 to track this on the app arch side, but I'm thinking it might make sense to plan these for the same release, since it is a breaking change.

[rudolf]

@lukeelmers We'll have to do this in 8.0 (I'll update the issue to make this more clear) and we will coordinate this with the wider Kibana team to ensure that we include all teams' changes.

[joshdover]

We'll have to do this in 8.0 (I'll update the issue to make this more clear) and we will coordinate this with the wider Kibana team to ensure that we include all teams' changes.

To be clear, we can start migrating this now and use the config deprecations feature to map the old name to the new name. We just can't remove that deprecation translation until 8.0. Ideally, we do these changes earlier than later so sysadmins have more heads up. I don't think it's critical that these deprecations are done at the same time.

[rudolf]

We should move csp to server.csp

kibana/src/core/server/csp/config.ts

Line 28 in 614bde9

// TODO: Move this to server.csp using config deprecations

[pgayvallet]

Ideally, we do these changes earlier than later so sysadmins have more heads up

Should this be moved to an earlier release then? It's currently on our 8.0 - Tentative board.

[joshdover]

Ideally, we do these changes earlier than later so sysadmins have more heads up

Should this be moved to an earlier release then? It's currently on our 8.0 - Tentative board.

8.0 and 7.last are released at the same time, so they're sorta the same roadmap. We need to make sure the deprecation is in the last minor release though. I will move.

[pgayvallet]

We need to make sure the deprecation is in the last minor release though. I will move.

I was more saying that, as we already know that these are going to be moved/deprecated, maybe we should add the deprecations sooner 😄 ?

I mean, as a user, having the info that these properties are going to be deprecated is better sooner than later?

[lukeelmers]

As doing a Kibana-wide renaming of config keys is nowhere on our near-term horizon, we'll go ahead and close this issue for now. We can reopen later if it becomes necessary.

[TinaHeiligers]

Throwing this out there: elasticsearch.shardTimeout should probably be removed from the Elasticsearch core service because it's not even used by that service. It should be renamed to something like kibana.searchTimeout and exposed in the same manner as other kibana.* settings.

@lukasolson shouldn't it be a part of data plugin as well?

@lukasolson The data plugin still uses elasticsearch.shardTimeout. are there still plans to move the config to the data plugin? If so, we'll need to start warning folks about the change since it's too late now to work that into #116296.

[lukasolson]

Thanks for resurfacing this @TinaHeiligers... Yes, ideally this setting would be in the data plugin rather than elasticsearch, but I'm not really sure what our options are at this point. @lukeelmers Is it possible for us to rename this at this point, or what would you suggest?

[lukeelmers]

@lukeelmers Is it possible for us to rename this at this point, or what would you suggest?

We'd need to follow the new deprecation guidelines before we could remove the old config, meaning the setting would probably linger for a couple years. However, I believe we could still move it to the data plugin starting in 8.1 (too late for 8.0), and just register a deprecation rename for it in the meantime so that users aren't affected.

@kobelb Do we need to discuss deprecations before they are added in the first place, or just need sign-off on the actual breaking changes before they happen?

[kobelb]

@kobelb Do we need to discuss deprecations before they are added in the first place, or just need sign-off on the actual breaking changes before they happen?

Deprecations don't need approval. We just need the approval to make the breaking change itself.

[lukasolson]

Created a new issue for tracking this last change here: #118250