Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add documentation for context and ABC templates #8197

Merged
merged 19 commits into from
Sep 16, 2024
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion _api-reference/index-apis/create-index-template.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ Parameter | Type | Description
`priority` | Integer | A number that determines which index templates take precedence during the creation of a new index or data stream. OpenSearch chooses the template with the highest priority. When no priority is given, the template is assigned a `0`, signifying the lowest priority. Optional.
`template` | Object | The template that includes the `aliases`, `mappings`, or `settings` for the index. For more information, see [#template]. Optional.
`version` | Integer | The version number used to manage index templates. Version numbers are not automatically set by OpenSearch. Optional.

`context` | Object | (Experimental) Context provides use case specific predefined templates which can be applied to an index. Amongst all settings and mappings declared for a template, context templates hold the highest priority. For more information, see [index-context]({{site.url}}{{site.baseurl}}/im-plugin/index-context/).
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

natebower marked this conversation as resolved.
Show resolved Hide resolved
### Template

Expand Down
2 changes: 1 addition & 1 deletion _api-reference/index-apis/create-index.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ timeout | Time | How long to wait for the request to return. Default is `30s`.

## Request body

As part of your request, you can optionally specify [index settings]({{site.url}}{{site.baseurl}}/im-plugin/index-settings/), [mappings]({{site.url}}{{site.baseurl}}/field-types/index/), and [aliases]({{site.url}}{{site.baseurl}}/opensearch/index-alias/) for your newly created index.
As part of your request, you can optionally specify [index settings]({{site.url}}{{site.baseurl}}/im-plugin/index-settings/), [mappings]({{site.url}}{{site.baseurl}}/field-types/index/), [aliases]({{site.url}}{{site.baseurl}}/opensearch/index-alias/), and [context]({{site.url}}{{site.baseurl}}/opensearch/index-context/).
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

## Example request

Expand Down
172 changes: 172 additions & 0 deletions _im-plugin/index-context.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,172 @@
---
layout: default
title: Index context
nav_order: 14
redirect_from:
- /opensearch/index-context/
---

# Index context

This is an experimental feature and is not recommended for use in a production environment. For updates on the progress the feature or if you want to leave feedback, join the discussion in the [OpenSearch forum](https://forum.opensearch.org/).
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
{: .warning}

Index Context declares the use case for an index. Using the context information, OpenSearch applies a pre-determined set of settings and mappings to do the heavy-lifting to ensure. When given the correct context, context provides indexes with:
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

- Optimized performance.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
- Settings tuned to your specific use case.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
- Accurate mappings and aliases based on [OpenSearch Integrations]({{site.url}}{{site.baseurl}}/integrations/).
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

The settings and metadata configuration that are applied using component templates are automatically loaded when your cluster starts. The templates for the components start with `@abc_template@`. To prevent potential configuration issues, only use `@abc_template@` the way it’s described on this page.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
{: .warning}


## Installation

To install the index context feature:

1. Install the `opensearch-system-templates` plugin on all nodes in your cluster using one of the [installation methods]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/#install)
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

2. Set the feature flag `opensearch.experimental.feature.application_templates.enabled` to `true`. For more information about enabling and disabling feature flags, see [Enabling experimental features]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/experimental/).

3. Set the `cluster.application_templates.enabled` setting to `true`. For instructions on how to configure OpenSearch, see [configuring settings]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/index/#static-settings).

## Using the `context` setting.

Check failure on line 34 in _im-plugin/index-context.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.HeadingPunctuation] Don't use punctuation at the end of a heading. Raw Output: {"message": "[OpenSearch.HeadingPunctuation] Don't use punctuation at the end of a heading.", "location": {"path": "_im-plugin/index-context.md", "range": {"start": {"line": 34, "column": 31}}}, "severity": "ERROR"}
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

Use the `context` setting with the Index API to add use-case specific context.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

### Considerations

Consider the following when using the `context` parameter during index creation:

natebower marked this conversation as resolved.
Show resolved Hide resolved
1. If you use the `context` field to create an index, you cannot include any settings declared in context during index creation or dynamic settings updates.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
2. Context becomes permanent when set on an index or index-template.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

When you adhere to these limitations, suggested configurations or mappings are uniformly applied on indexed data within the specified context.

### Examples
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

#### Create Index

Check failure on line 49 in _im-plugin/index-context.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.StackedHeadings] Do not stack headings. Insert an introductory sentence between headings. Raw Output: {"message": "[OpenSearch.StackedHeadings] Do not stack headings. Insert an introductory sentence between headings.", "location": {"path": "_im-plugin/index-context.md", "range": {"start": {"line": 49, "column": 1}}}, "severity": "ERROR"}

Check failure on line 49 in _im-plugin/index-context.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.HeadingCapitalization] 'Create Index' is a heading and should be in sentence case. Raw Output: {"message": "[OpenSearch.HeadingCapitalization] 'Create Index' is a heading and should be in sentence case.", "location": {"path": "_im-plugin/index-context.md", "range": {"start": {"line": 49, "column": 6}}}, "severity": "ERROR"}
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

The following example request creates an index to store metrics data by declaring a `metrics` mapping as the context:
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

```json
PUT /my-metrics-index
{
"context": {
"name": "metrics"
}
}
```
{% include copy-curl.html %}

After creation, the index will have the context added to it, and also have few settings applied:
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved


**GET request**

```json
GET /my-metrics-index
```
{% include copy-curl.html %}


**Response**

```json
{
"my-metrics-index": {
"aliases": {},
"mappings": {},
"settings": {
"index": {
"codec": "zstd_no_dict",
"refresh_interval": "60s",
"number_of_shards": "1",
"provided_name": "my-metrics-index",
"merge": {
"policy": "log_byte_size"
},
"context": {
"created_version": "1",
"current_version": "1"
},
...
}
},
"context": {
"name": "metrics",
"version": "_latest"
}
}
}
```


#### Create index templates
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Create an index template"?

Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

You can also use the `context` parameter when creating an index template. The following example request creates an index template with the context information as `logs`.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

```json
PUT _index_template/my-logs
{
"context": {
"name": "logs",
"version": "1"
},
"index_patterns": [
"my-logs-*"
]
}
```
{% include copy-curl.html %}

All indexes created using this index template will get the metadata provided by the associated component template. The following request and response shows how `context` is added to the template:
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

**Get index template**

```json
GET _index_template/my-logs
```
{% include copy-curl.html %}

**Response**

```json
{
"index_templates": [
{
"name": "my-logs2",
"index_template": {
"index_patterns": [
"my-logs1-*"
],
"context": {
"name": "logs",
"version": "1"
}
}
}
]
}
```

If there is any overlap between any settings, mappings, or aliases declared by your template directly and the backing component template for the context, the latter gets higher priority during index creation.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

natebower marked this conversation as resolved.
Show resolved Hide resolved

## Available templates for context
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

The following templates are available to be used through the `context` parameter as of OpenSearch 2.17:

- `logs`
- `metrics`
- `nginx-logs`
- `amazon-cloudtrail-logs`
- `amazon-elb-logs`
- `amazon-s3-logs`
- `apache-web-logs`
- `k8s-logs`

For more information about these templates, see the [OpenSearch system templates repository](https://github.com/opensearch-project/opensearch-system-templates/tree/main/src/main/resources/org/opensearch/system/applicationtemplates/v1).

To view the current version of these templates on your cluster, use `GET /_component_template`.
Loading