Skip to content

Commit

Permalink
Add template simulation API for simulating template composition (#56842)
Browse files Browse the repository at this point in the history
This adds an API for simulating template composition with or without an index template.

It looks like:

```
POST /_index_template/_simulate/my-template
```

To simulate a template named `my-template` that already exists, or, to simulate a template that does
not already exist:

```
POST /_index_template/_simulate
{
  "index_patterns": ["my-index"]
  "composed_of": ["ct1", "ct2"],
}
```

This is related to #55686, which adds an API to simulate composition based on an index name (hence
the `_simulate_index` vs `_simulate`).

This commit also adds reference documentation for both simulation APIs.

Relates to #53101
Resolves #56390
Resolves #56255
  • Loading branch information
dakrone authored May 18, 2020
1 parent 0b557e4 commit d3ccada
Show file tree
Hide file tree
Showing 12 changed files with 849 additions and 45 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -803,7 +803,8 @@ public void testApiNamingConventions() throws Exception {
"scripts_painless_execute",
"indices.create_data_stream",
"indices.get_data_stream",
"indices.delete_data_stream"
"indices.delete_data_stream",
"indices.simulate_template"
};
//These API are not required for high-level client feature completeness
String[] notRequiredApi = new String[] {
Expand Down
131 changes: 130 additions & 1 deletion docs/reference/indices/index-templates.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ PUT _index_template/template_1
[source,console]
--------------------------------------------------
DELETE _index_template/template_*
DELETE _index_template/*
DELETE _component_template/*
--------------------------------------------------
// TEARDOWN
Expand Down Expand Up @@ -291,6 +291,135 @@ PUT /_index_template/template_1
In this case, an index matching `t*` will have three primary shards. If the order of composed
templates were reversed, the index would have two primary shards.


[[simulating-templates]]
===== Simulating template composition

Since templates can be composed not only of multiple component templates, but also the index
template itself, there are two simulation APIs to determine what the resulting index settings will
be.

To simulate the settings that would be applied to a matching index name:

[source,console]
--------------------------------------------------
POST /_index_template/_simulate_index/myindex
--------------------------------------------------

To simulate the settings that would be applied from a particular template:

[source,console]
--------------------------------------------------
POST /_index_template/_simulate/template_1
POST /_index_template/_simulate
{
"index_patterns": ["foo"],
"template": {
"settings": {
"number_of_replicas": 0
}
}
}
--------------------------------------------------


Here's an example demonstrating simulating both an index name and template name:

[source,console]
--------------------------------------------------
PUT /_component_template/ct1 <1>
{
"template": {
"settings": {
"index.number_of_shards": 2
}
}
}
PUT /_component_template/ct2 <2>
{
"template": {
"settings": {
"index.number_of_replicas": 0
},
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
}
}
}
}
}
PUT /_index_template/final-template <3>
{
"index_patterns": ["logdata-*"],
"composed_of": ["ct1", "ct2"],
"priority": 5
}
POST /_index_template/_simulate_index/logdata-2019-02-01 <4>
POST /_index_template/_simulate/final-template <5>
POST /_index_template/_simulate <6>
{
"index_patterns": ["logdata-*"],
"composed_of": ["ct2"],
"priority": 10,
"template": {
"settings": {
"index.number_of_replicas": 1
}
}
}
--------------------------------------------------
<1> Creating a component template (ct1) setting the number of shards to two
<2> Creating another component template (ct2) setting the number of replicas to zero with mappings
<3> Creating an index template called "final" template using ct1 and ct2
<4> Simulate the settings that would be applied for a new index "logdata-2019-02-01"
<5> Simulate the settings composed using the "final-template" index template
<6> Simulate the settings composed using a custom specified template

The output of the simulate API from the last example call looks like:

[source,console-result]
---------------------------------------------------------
{
"template" : {
"settings" : {
"index" : {
"number_of_replicas" : "1" <1>
}
},
"mappings" : {
"properties" : {
"@timestamp" : { <2>
"type" : "date"
}
}
},
"aliases" : { }
},
"overlapping" : [ <3>
{
"name" : "final-template",
"index_patterns" : [
"logdata-*"
]
}
]
}
---------------------------------------------------------
<1> The number of replicas from the simulated template body
<2> The `@timestamp` field inherited from the "ct2" component template
<3> Any overlapping templates that would have matched, but have lower priority

When simulating a template and specifying a template in the body of the request, the simulated
template is not added to the existing templates, it is only used for the simulation.

===== Index template with index aliases

You can include <<indices-aliases,index aliases>> in an index template.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
{
"indices.simulate_template":{
"documentation":{
"url":"https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-templates.html",
"description": "Simulate resolving the given template name or body"
},
"stability":"stable",
"url":{
"paths":[
{
"path":"/_index_template/_simulate",
"methods":[
"POST"
]
},
{
"path":"/_index_template/_simulate/{name}",
"methods":[
"POST"
],
"parts":{
"name":{
"type":"string",
"description":"The name of the index template"
}
}
}
]
},
"params":{
"create":{
"type":"boolean",
"description":"Whether the index template we optionally defined in the body should only be dry-run added if new or can also replace an existing one",
"default":false
},
"cause":{
"type":"string",
"description":"User defined reason for dry-run creating the new template for simulation purposes",
"default":false
},
"master_timeout":{
"type":"time",
"description":"Specify timeout for connection to master"
}
},
"body":{
"description":"New index template definition to be simulated, if no index template name is specified",
"required":false
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@

- match: {template.settings.index.number_of_shards: "1"}
- match: {template.settings.index.number_of_replicas: "0"}
- match: {template.mappings._doc.properties.field.type: "keyword"}
- match: {template.mappings.properties.field.type: "keyword"}
- match: {overlapping: []}

---
Expand Down Expand Up @@ -77,7 +77,7 @@

- match: {template.settings.index.blocks.write: "true"}
- match: {template.settings.index.number_of_replicas: "2"}
- match: {template.mappings._doc.properties.ct_field.type: "keyword"}
- match: {template.mappings.properties.ct_field.type: "keyword"}
- match: {overlapping.0.name: existing_test}
- match: {overlapping.0.index_patterns: ["te*"]}
- length: {template.aliases: 1}
Expand Down Expand Up @@ -170,7 +170,7 @@

- match: {template.settings.index.number_of_shards: "1"}
- match: {template.settings.index.number_of_replicas: "0"}
- match: {template.mappings._doc.properties.field.type: "keyword"}
- match: {template.mappings.properties.field.type: "keyword"}
- match: {overlapping.0.name: v1_template}
- match: {overlapping.0.index_patterns: ["t*", "t1*"]}
- match: {overlapping.1.name: v2_template}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
---
"Simulate template without a template in the body":
- skip:
version: " - 7.99.99"
reason: "not yet backported"
features: ["default_shards"]

- do:
indices.put_index_template:
name: my-template
body:
index_patterns: other
template:
settings:
number_of_shards: 1
number_of_replicas: 0
mappings:
properties:
field:
type: keyword

- do:
indices.simulate_template:
name: my-template

- match: {template.settings.index.number_of_shards: "1"}
- match: {template.settings.index.number_of_replicas: "0"}
- match: {template.mappings.properties.field.type: "keyword"}
- match: {overlapping: []}

---
"Simulate index template specifying a new template":
- skip:
version: " - 7.99.99"
reason: "not yet backported"
features: ["default_shards"]

- do:
indices.put_index_template:
name: existing_test
body:
index_patterns: te*
priority: 10
template:
settings:
number_of_shards: 1
number_of_replicas: 0
mappings:
properties:
field:
type: keyword

- do:
cluster.put_component_template:
name: ct
body:
template:
settings:
index.number_of_replicas: 2
mappings:
properties:
ct_field:
type: keyword

- do:
indices.simulate_template:
body:
index_patterns: te*
priority: 15
template:
settings:
index.blocks.write: true
aliases:
test_alias: {}
composed_of: ["ct"]

- match: {template.settings.index.blocks.write: "true"}
- match: {template.settings.index.number_of_replicas: "2"}
- match: {template.mappings.properties.ct_field.type: "keyword"}
- match: {overlapping.0.name: existing_test}
- match: {overlapping.0.index_patterns: ["te*"]}
- length: {template.aliases: 1}
- is_true: template.aliases.test_alias

---
"Simulate template matches overlapping V1 and V2 templates":
- skip:
version: " - 7.99.99"
reason: "not yet backported"
features: ["allowed_warnings", "default_shards"]

- do:
indices.put_template:
name: v1_template
body:
index_patterns: [t*, t1*]
settings:
number_of_shards: 5

- do:
allowed_warnings:
- "index template [v2_template] has index patterns [te*] matching patterns from existing older templates [v1_template] with patterns
(v1_template => [t*, t1*]); this template [v2_template] will take precedence during new index creation"
indices.put_index_template:
name: v2_template
body:
index_patterns: te*
priority: 10
template:
settings:
number_of_shards: 10
number_of_replicas: 2
mappings:
properties:
field:
type: text

- do:
allowed_warnings:
- "index template [winning_v2_template] has index patterns [te*] matching patterns from existing older templates [v1_template] with patterns
(v1_template => [t*, t1*]); this template [winning_v2_template] will take precedence during new index creation"
indices.put_index_template:
name: winning_v2_template
body:
index_patterns: te*
priority: 20
template:
settings:
number_of_shards: 1
number_of_replicas: 0
mappings:
properties:
field:
type: keyword

- do:
indices.simulate_template:
name: winning_v2_template

- match: {template.settings.index.number_of_shards: "1"}
- match: {template.settings.index.number_of_replicas: "0"}
- match: {template.mappings.properties.field.type: "keyword"}
- match: {overlapping.0.name: v1_template}
- match: {overlapping.0.index_patterns: ["t*", "t1*"]}
- match: {overlapping.1.name: v2_template}
- match: {overlapping.1.index_patterns: ["te*"]}
Loading

0 comments on commit d3ccada

Please sign in to comment.