Composable Templates

[martijnvg]

The meta issue will track the development of the second version of index templates.

Tasks:

• Introduce component templates in cluster state (@dakrone) Add ComponentTemplate to MetaData #53290
• CRUD APIs for component templates. (@dakrone) Add REST API for ComponentTemplate CRUD #53558
  • Add validation for component templates "correctness" (@probakowski) Add validation for component templates #54023
  • Add HLRC support for component template APIs (@probakowski) HLRC support for Component Templates APIs #54635
  • Add HLRC documentation for component template APIs (@andreidan) HLRC: document the index template v2 and component template APIs #56136
• Introduce v2 index template format (@dakrone) Add IndexTemplateV2 to MetaData #53753
• CRUD APIs for v2 index templates (@dakrone) Add REST APIs for IndexTemplateV2Metadata CRUD #54039
  • Add HLRC support for v2 index templates (@probakowski) HLRC support for Index Templates V2 #54838
  • Add HLRC documentation for v2 index templates API (@andreidan) HLRC: document the index template v2 and component template APIs #56136
• Add simulate API for simulating template composition (@andreidan) Add simulate template composition API _index_template/_simulate_index/{name}  #55686
  • Add HLRC support for simulate API (@andreidan) Add HLRC support for simulate index template api #55936
  • Add documentation for simulate API (@andreidan) HLRC: document the index template v2 and component template APIs #56136
• Change GET API for component and index templates to only support a single name or name wildcard (@dakrone) Only allow retrieving a single index or component template #54694
• Enforce v1 & v2 warnings and errors when templates overlap (@dakrone) Add warnings/errors when V2 templates would match same indices as V1 #54367
  • Prevent putting v2 index template (with error) when it would overlap with existing v2 template of the same priority (@probakowski) Prevent putting V2 index template when overlapping with existing template #54933
• Enforce no _doc (or other) document types are specified in v2 component templates or v2 index templates (will be handled by Use V2 index templates during index creation #54669)
• Actually use v2 index templates for new index creation (@dakrone) Use V2 index templates during index creation #54669
• Add ?prefer_v2_templates flag for all requests that can create an index (@dakrone) Add prefer_v2_templates flag and index setting #55411
  • Default prefer_v2_templates to true for 8.0+ (@dakrone) Change prefer_v2_templates parameter to default to true #55489
  • Add support for prefer_v2_templates to the Reindex API (@probakowski) Add prefer_v2_templates parameter to Reindex #56253
• Merge mappings in a manner that allows only a single field configuration (@dakrone) Merge V2 index/component template mappings in specific manner #55607
  • Overwrite mappings for fields using dots in field names (@dakrone) Handle merging dotted object names when merging V2 template mappings #55982
  • Potentially disallow duplicate dynamic templates in mappings (@andreidan) ITV2: disallow duplicate dynamic templates #56291
• Remove 7.x error for template CRUD (@dakrone) Downgrade template update error to a warning for v1 templates #55611
• Emit deprecation warning if multiple v1 templates match with a new index (@probakowski) Emit deprecation warning if multiple v1 templates match with a new index #55558
• Add support for V2 templates to /_cat/templates (@dakrone) Add support for V2 index templates to /_cat/templates #55829
• Documentation (@dakrone) Initial documentation for index templates V2 #55755
• Ensure security compatibility (may not be necessary, just need to double check)
• Mark APIs as experimental for 7.8 (@dakrone) Mark ITv2 APIs as experimental #55874
• Remove feature flag hiding before release (@dakrone) Remove Index Templates V2 feature flag #56123
• Fix issue with null != 0 and negative priority values (@dakrone) Validate non-negative priorities for V2 index templates #56139
• Re-use MetadataIndexTemplateService#validate for V2 index templates, and add * validation (@dakrone) Validate V2 templates more strictly #56170

---

Tasks that may be done after initial release:

• Mark APIs as stable for 7.9
• Switch internal use of templates to use v2 templates (like ILM/SLM history indices, ML, etc)
• Switch all internal tests to use v2 templates instead of v1 templates
• Enforce only delete or update operations for v1 templates are allowed on 8.0+ (no creation of new v1 templates)
• Deprecate the entire v1 template API in 8.0+
• Add a method to migrate a v1 template into a v2 template? (maybe)
• Add simulate API for a template ie. /_index_template/{template-name}/_simulate, without any body. It will return the final resolved version of the template (merging the component templates)

Background

Index templates as it stands today has a number of shortcomings mainly around index template inheritance:

• At index creation time, multiple index templates can be applied. If no template order is specified or templates have the same order then how templates are applied to create the final mappings and index setting is undefined. This can result in settings/mappings being applied in an unexpected manner.
• With template inheritance, multiple index templates can merged together. The end result of that merge can result in invalid settings and mappings.

Version 2 changes

The biggest change is that reuse of template configuration via template inheritance that exists today is going to get replaced by template composition. A new index will no longer be able to match with multiple index templates. If a new index matches with multiple index templates then the index template with the highest priority is selected and used. If multiple index templates match that have the same priority then that results into an error. Either at alias template creation time or when alias templates are evaluated as part of index creation.

New indices may require the same index template configuration be applied to them, but their names may be so different that a single index template's index pattern can't capture them and because with index templates v2 only a single template can match with a new index. To solve this problem, we'll abstract the index template config into reusable ‘component templates', which users can refer to when composing index templates. A component template only contains settings, mappings and aliases. Index templates can be composed of one or more component templates. The order in which the names of component templates are specified in an index template determines the order in which the component templates are applied. The setting, mapping and aliases specified inside an index template are applied last.

Component templates never match at index creation time directly and are only included via a matching index template. Component templates can’t be composed of other component templates.

Index and component templates can only contain settings and mappings that are valid on their own. The settings and mappings of an individual index template should result in a successful index creation.

Component template merging

The order in which the names of component templates are specified in an index template determines the order in which the component templates are applied. The setting, mapping and aliases specified inside an index template are applied last.

Settings and mappings are essentially nested key value pairs. If a component template is applied and a key already exists then it will replace the key and value even if that contains other key value pairs.

For example consider the following snippets:

{
  "my_field": {
      "type": "integer", 
      "ignore_malformed": true
  }
}
{
  "my_field": {
     "type": "keyword", 
     "ignore_above": 1024
  }
}

With the merging of index templates that exist today, the following result would be created:

{
  "my_field": {
      "ignore_malformed": true,
      "type": "keyword", 
      "ignore_above": 1024
  }
}

In order to avoid this, at the time a template with snippet B is applied then that would completely overwrite the ‘my_field’ key in order to avoid creating an illegal mapping. With this logic this would be the result:

{
  "my_field": {
     "type": "keyword", 
     "ignore_above": 1024
  }
}

Merging will happen on a field granular level, and the V2 templates will ensure no doc types are specified.

Also dynamic templates should be merged in the same manner.

Backwards compatibility

We should strive towards ensuring that existing index templates should be able to work like index templates v2. If index templates v1 are not relying on template inheritance then no action should be required.

With index templates v2 configuration reuse is achieved by selective composability whereas with index templates v1 this is achieved via template inheritance. Template inheritance should be allowed throughout the 7.x version timeline, so that users can upgrade their index templates to use component template. In this period of time we should emit deprecation warnings if a new index matches with more than one index template in the v1 structure. After the deprecation period, if multiple v1 templates match we should fail with an error at index creation time. Users will then have to upgrade to use component templates.

For interop with both v1 and v2 templates the following will be used:

If * in v1, allow but warn to put v2
Runtime resolution will favor v2
If PUT v1 (not ‘*’) and overlaps with v2, then error (v2 will take precedence)
If PUT v1 (with ‘*’) and overlaps with v2, then warn (v2 may take precedence)
If PUT v2 with ‘*’ allow the overlap, warn about taking precedence
If PUT v2 that matches a V1 template, warn that v2 will take precedence
In 8.0 - only allow update/delete of v1, not create (provide an escape hatch setting/property)
In 7.x, deprecate the entire /_template endpoint

In 9.0 the deprecated /_template endpoint will be removed.

The structure and field names have changed slightly with index templates v1. The current structure and field names will be supported throughout the 7.x version timeline. After which we can automatically rewrite the older format into the new format upon upgrading to version 8. (we may be able to provide a conversion API, but we cannot automatically upgrade them since v1 templates rely on inheritance)

Config and APIs

Component templates are stored in the cluster state as separate resources. Index templates remain to be stored in the same place inside the cluster state.

Index Templates v2 Config

Config:

{
    "index_patterns": [...],
    "priority": 0-9223372036854775808,
    "template": {
       "settings": {...},
       "mappings": {...},
       "aliases": {...}
    },
    "composed_of": [...],
    "version": 0-9223372036854775808,
    "_meta": { ... }
}

Explanation of the configuration fields:

• index_patterns (required), a string array. Each string contains a wildcard pattern.
• template (optional), a json object at least settings or mappings or aliases.
  • settings (optional), a json object representing index settings.
  • mappings (optional), a json object representing mappings.
  • aliases (optional), a json object representing aliases.
• priority (optional), a numeric value indicating precedence over other index templates. When indexing into a non existing index/alias, the name may match with patterns of multiple index templates, the alias template with the highest order value will be selected. Other index templates will be ignored. Defaults to 0.
• composed_of (optional), a string array with the names of components templates that will be applied on a new index together with the current template. The order in the string array determines the order in which the component templates are applied. If composed_of is specified then the index template is applied last. This is to ensure that the settings/mappings for the index template are never overwritten by component templates it references to.
• version (optional), a numeric value. External systems can use this version number to simplify template management. The version parameter is completely optional and not automatically generated by Elasticsearch. Defaults to null.
• _meta (optional), a json object with meta information about a template, not used by Elasticsearch, but for third party systems and users.
  Index Template APIs
  Index v2 templates will remain to use the existing index template APIs. Additionally there will be an api to simulate what index template would match with a given index name. Also the final index settings, mappings and aliases would be returned.

Simulate index template api:

GET /_index_template/[index_name]/_simulate

We should also support sending an index template (v2) body with the simulate API, so a UI can simulate adding an index template and seeing what it would look like.

Component Template Config

Config:

{
    "template": {
       "settings": {...},
       "mappings": {...},
       "aliases": {...}
    },
    "version": 0-9223372036854775808,
    "_meta": { ... }
}

Explanation of the configuration fields:

• template (required), a json object at least settings or mappings or aliases.
  • settings (optional), a json object representing index settings.
  • mappings (optional), a json object representing mappings.
  • aliases (optional), a json object representing aliases.
• version (optional), a numeric value. External systems can use this version number to simplify template management. The version parameter is completely optional and not automatically generated by Elasticsearch. Defaults to null.
• _meta (optional), a json object with meta information about a template, not used by Elasticsearch, but for third party systems and users.

Component Template APIs

Put component template api:

PUT /_component_template/[name]
{
   ...
}

Get component template api:

GET /_component_template/[name]

Delete component template api:

DELETE /_component_template/[name]

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Indices APIs)