Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add documentation for context and ABC templates #8197

Merged
merged 19 commits into from
Sep 16, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion _api-reference/index-apis/create-index-template.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ Parameter | Type | Description
`priority` | Integer | A number that determines which index templates take precedence during the creation of a new index or data stream. OpenSearch chooses the template with the highest priority. When no priority is given, the template is assigned a `0`, signifying the lowest priority. Optional.
`template` | Object | The template that includes the `aliases`, `mappings`, or `settings` for the index. For more information, see [#template]. Optional.
`version` | Integer | The version number used to manage index templates. Version numbers are not automatically set by OpenSearch. Optional.

`context` | Object | For more information, see [index-context]({{site.url}}{{site.baseurl}}/opensearch/index-context/). Amongst all settings and mappings declared for an index template, the ones coming from context will hold highest priority. As of OpenSearch 2.17, this is an experimental feature. Optional
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

natebower marked this conversation as resolved.
Show resolved Hide resolved
### Template

Expand Down
2 changes: 1 addition & 1 deletion _api-reference/index-apis/create-index.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ timeout | Time | How long to wait for the request to return. Default is `30s`.

## Request body

As part of your request, you can optionally specify [index settings]({{site.url}}{{site.baseurl}}/im-plugin/index-settings/), [mappings]({{site.url}}{{site.baseurl}}/field-types/index/), and [aliases]({{site.url}}{{site.baseurl}}/opensearch/index-alias/) for your newly created index.
As part of your request, you can optionally specify [index settings]({{site.url}}{{site.baseurl}}/im-plugin/index-settings/), [mappings]({{site.url}}{{site.baseurl}}/field-types/index/), [aliases]({{site.url}}{{site.baseurl}}/opensearch/index-alias/), and [context]({{site.url}}{{site.baseurl}}/opensearch/index-context/) (experimental as of OpenSearch 2.17) for your newly created index.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

## Example request

Expand Down
170 changes: 170 additions & 0 deletions _im-plugin/index-context.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,170 @@
---
layout: default
title: Index context
nav_order: 14
redirect_from:
- /opensearch/index-context/
---

# Index context

Context can be used to declare the use case for which you are building your index.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
Using the context information, OpenSearch can apply a pre-determined set of settings, and mappings to do the heavy-lifting around ensuring:
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

1. Available performance optimizations are applied for the index.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
2. Existing settings are tuned according to the use-case.
3. Mappings/Aliases based on [OpenSearch Integrations]({{site.url}}{{site.baseurl}}/integrations/) are applied on the index if applicable.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

These metadata parameters applied are backed by component templates which are automatically loaded when your cluster comes up. These component templates are prefixed with `@abc_template@`, and can only be used based on the usage pattern outlined in this document.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
{: .important}

The `context` parameter is available on indices and index templates as an experimental feature starting OpenSearch 2.17

Check failure on line 21 in _im-plugin/index-context.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.SubstitutionsError] Use 'indexes' instead of 'indices'. Raw Output: {"message": "[OpenSearch.SubstitutionsError] Use 'indexes' instead of 'indices'.", "location": {"path": "_im-plugin/index-context.md", "range": {"start": {"line": 21, "column": 41}}}, "severity": "ERROR"}
{: .warning}

## Usage

### Installation

Check failure on line 26 in _im-plugin/index-context.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.StackedHeadings] Do not stack headings. Insert an introductory sentence between headings. Raw Output: {"message": "[OpenSearch.StackedHeadings] Do not stack headings. Insert an introductory sentence between headings.", "location": {"path": "_im-plugin/index-context.md", "range": {"start": {"line": 26, "column": 1}}}, "severity": "ERROR"}

In order to use the `context` field while creating an index or index-template:

1. Install the `opensearch-system-templates` plugin on all nodes in your cluster using one of the [installation methods]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/#install)
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
2. Set the feature flag `opensearch.experimental.feature.application_templates.enabled` to `true`. For more information about enabling and disabling feature flags, see [Enabling experimental features]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/experimental/).
3. Enable setting `cluster.application_templates.enabled` to `true`. See [configuring settings]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/index/#static-settings) for steps on how to do this.
mgodwan marked this conversation as resolved.
Show resolved Hide resolved


### Create Index

Check failure on line 35 in _im-plugin/index-context.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.HeadingCapitalization] 'Create Index' is a heading and should be in sentence case. Raw Output: {"message": "[OpenSearch.HeadingCapitalization] 'Create Index' is a heading and should be in sentence case.", "location": {"path": "_im-plugin/index-context.md", "range": {"start": {"line": 35, "column": 5}}}, "severity": "ERROR"}

For example, if you are using your indices to store metrics data, you can declare `metrics` as a context on your index:

Check failure on line 37 in _im-plugin/index-context.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.SubstitutionsError] Use 'indexes' instead of 'indices'. Raw Output: {"message": "[OpenSearch.SubstitutionsError] Use 'indexes' instead of 'indices'.", "location": {"path": "_im-plugin/index-context.md", "range": {"start": {"line": 37, "column": 36}}}, "severity": "ERROR"}
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

````
```json
PUT /my-metrics-index
{
"context": {
"name": "metrics"
}
}
```
{% include copy-curl.html %}
````


After creation, the index will have the context added to it, and also few settings applied:
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved


````
```json
GET /my-metrics-index
```
{% include copy-curl.html %}
````

````
```json
{
"my-metrics-index": {
"aliases": {},
"mappings": {},
"settings": {
"index": {
"codec": "zstd_no_dict",
"refresh_interval": "60s",
"number_of_shards": "1",
"provided_name": "my-metrics-index",
"merge": {
"policy": "log_byte_size"
},
"context": {
"created_version": "1",
"current_version": "1"
},
...
}
},
"context": {
"name": "metrics",
"version": "_latest"
}
}
}
```
````

#### Considerations

There are few restrictions which are applied when you decide to use `context` param when creating your indices:

Check failure on line 95 in _im-plugin/index-context.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.SubstitutionsError] Use 'indexes' instead of 'indices'. Raw Output: {"message": "[OpenSearch.SubstitutionsError] Use 'indexes' instead of 'indices'.", "location": {"path": "_im-plugin/index-context.md", "range": {"start": {"line": 95, "column": 104}}}, "severity": "ERROR"}

1. When you create an index using `context` field, any settings which are declared in context cannot be provided as part of index creation, or dynamic settings updates.
2. Once set on an index/index-template, context cannot be removed from it.

The above points ensure that at any point, the recommended settings/mappings are always applied on the indices.

Check warning on line 100 in _im-plugin/index-context.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.DirectionAboveBelow] Use 'previous, preceding, or earlier' instead of 'above' for versions or orientation within a document. Use 'above' and 'below' only for physical space or screen descriptions. Raw Output: {"message": "[OpenSearch.DirectionAboveBelow] Use 'previous, preceding, or earlier' instead of 'above' for versions or orientation within a document. Use 'above' and 'below' only for physical space or screen descriptions.", "location": {"path": "_im-plugin/index-context.md", "range": {"start": {"line": 100, "column": 5}}}, "severity": "WARNING"}

Check failure on line 100 in _im-plugin/index-context.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.SubstitutionsError] Use 'indexes' instead of 'indices'. Raw Output: {"message": "[OpenSearch.SubstitutionsError] Use 'indexes' instead of 'indices'.", "location": {"path": "_im-plugin/index-context.md", "range": {"start": {"line": 100, "column": 104}}}, "severity": "ERROR"}



### Create Index Templates

Check failure on line 104 in _im-plugin/index-context.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.HeadingCapitalization] 'Create Index Templates' is a heading and should be in sentence case. Raw Output: {"message": "[OpenSearch.HeadingCapitalization] 'Create Index Templates' is a heading and should be in sentence case.", "location": {"path": "_im-plugin/index-context.md", "range": {"start": {"line": 104, "column": 5}}}, "severity": "ERROR"}

You can also provide the `context` parameter while creating index templates:

````
```json
PUT _index_template/my-logs
{
"context": {
"name": "logs",
"version": "1"
},
"index_patterns": [
"my-logs-*"
]
}
```
{% include copy-curl.html %}
````

````
```json
GET _index_template/my-logs
```
{% include copy-curl.html %}
````

````
```json
{
"index_templates": [
{
"name": "my-logs2",
"index_template": {
"index_patterns": [
"my-logs1-*"
],
"context": {
"name": "logs",
"version": "1"
}
}
}
]
}
```
````

natebower marked this conversation as resolved.
Show resolved Hide resolved

This will create an index template with the context information as `logs`, and all indices created via this index template will get the metadata provided by the associated component template.

Check failure on line 153 in _im-plugin/index-context.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.SubstitutionsError] Use 'indexes' instead of 'indices'. Raw Output: {"message": "[OpenSearch.SubstitutionsError] Use 'indexes' instead of 'indices'.", "location": {"path": "_im-plugin/index-context.md", "range": {"start": {"line": 153, "column": 84}}}, "severity": "ERROR"}

Check warning on line 153 in _im-plugin/index-context.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.LatinismsSubstitution] Use 'using, through, by accessing, or by choosing' instead of 'via'. Raw Output: {"message": "[OpenSearch.LatinismsSubstitution] Use 'using, through, by accessing, or by choosing' instead of 'via'.", "location": {"path": "_im-plugin/index-context.md", "range": {"start": {"line": 153, "column": 100}}}, "severity": "WARNING"}
mgodwan marked this conversation as resolved.
Show resolved Hide resolved
If there is any overlap between settings/mappings/aliases declared by your template directly and the same coming through the backing component template for the context, the latter gets higher priority during index creation.

## Available Templates for Context

Check failure on line 156 in _im-plugin/index-context.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.HeadingCapitalization] 'Available Templates for Context' is a heading and should be in sentence case. Raw Output: {"message": "[OpenSearch.HeadingCapitalization] 'Available Templates for Context' is a heading and should be in sentence case.", "location": {"path": "_im-plugin/index-context.md", "range": {"start": {"line": 156, "column": 4}}}, "severity": "ERROR"}

Following templates are available to be used through the `context` parameter as of today:
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

1. logs
2. metrics
3. nginx-logs
4. amazon-cloudtrail-logs
5. amazon-elb-logs
6. amazon-s3-logs
7. apache-web-logs
8. k8s-logs

To see more details on what these templates provide, please take a look at the [repository](https://github.com/opensearch-project/opensearch-system-templates/tree/d71e12588e3144ad0fd26745d084ba49e9e08349/src/main/resources/org/opensearch/system/applicationtemplates/v1).

Check warning on line 169 in _im-plugin/index-context.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.Please] Using 'please' is unnecessary. Remove. Raw Output: {"message": "[OpenSearch.Please] Using 'please' is unnecessary. Remove.", "location": {"path": "_im-plugin/index-context.md", "range": {"start": {"line": 169, "column": 54}}}, "severity": "WARNING"}
You can also check the current version of these templates installed on your cluster through `GET /_component_template`
mgodwan marked this conversation as resolved.
Show resolved Hide resolved
Loading