Introduce system index APIs for Kibana

[jaymode]

This commit introduces a module for Kibana that exposes REST APIs that
will be used by Kibana for access to its system indices. These APIs are wrapped
versions of the existing REST endpoints. A new setting is also introduced since
the Kibana system indices' names are allowed to be changed by a user in case
multiple instances of Kibana use the same instance of Elasticsearch.

Additionally, the ThreadContext has been extended to indicate that the use of
system indices may be allowed in a request. This will be built upon in the future
for the protection of system indices.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Core)

[jaymode]

@tylersmalley I was looking at elastic/kibana#49764 and I wanted to make sure I understood the request there for the paths where you have _kibana/{kibana.name}/ and kibana.name would be understood by elasticsearch based on settings.

Right now in this plugin, the setting I have would look like this in the elasticsearch.yml file:

kibana.system_indices: [ ".kibana", ".kibana_task_manager", ".reporting" ]

I think the Kibana issue is actually looking for something more like:

kibana:
  instances:
    my_kibana:
       kibana_index: ".my-kibana"
       task_manager_index: ".my-tm"
       reporting_index: ".my-reports"

Is this correct?

[tylersmalley]

The latter is correct. Ideally, I would like to define a list of names which we could derive each index off of, but that won't work today as users can configure them individually.

In that example, would the endpoint be _kibana/my_kibana/*?

It would probably also make sense to include a default so Kibana instance configuration is only necessary if they intend on having multiple instances.

[jaymode]

Yes that would be the endpoint path. For the default, I think we will still want an instance name so maybe just ‘kibana’ for that?

[tylersmalley]

Agreed. Though, could we use default for the name of the default instance? I feel this should be clear as to what the instance is and avoid _kibana/kibana/*

[rudolf]

What is the risk for not whitelisting kibana instances and indices? From a user's perspective it's error prone to have to adjust both the kibana and elasticsearch config files and keep them in sync. The status quo is that Kibana creates any instances (it's currently just an index prefix) and indices it wants to.

[jaymode]

Every system index plugin needs to declare the system indices it is used for as a way to identify indices that are system indices. If you are ok with just creating all of your indices under a single prefix, then I think that would be fine.

[tylersmalley]

A question was also brought up as to if we would consider this a breaking change if it was introduced before 8.0. Currently, to migrate to the system indices the user would be required to update ES and Kibana configuration.

The prefixing is interesting idea worth exploring, but would require coming up with a plan to migrate to these index names, correct?

[jaymode]

Good point on it being breaking. One thing we can do is allow the APIs under /_kibana to be unrestricted for the indices that could be used in 7.x.

[jaymode]

For the time being, I have gone ahead and removed the limiting of indices accessible by the APIs until we figure out what we should do here regarding these indices and their names.

[gwbrown]

Left a few comments but overall looks really good.

[gwbrown]

We might need wildcard support eventually, although I don't think we do for Kibana (@tylersmalley do you know?). Date math support is more questionable. It might be worth trying to detect it and throwing an exception, similar to what we do here.

[gwbrown]

LGTM, nice work on this.