Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document new ingest and search pipeline allowlist settings #7414

Merged
merged 8 commits into from
Jul 1, 2024
4 changes: 4 additions & 0 deletions _ingest-pipelines/processors/index-processors.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,3 +71,7 @@ Processor type | Description
## Batch-enabled processors

Some processors support batch ingestion---they can process multiple documents at the same time as a batch. These batch-enabled processors usually provide better performance when using batch processing. For batch processing, use the [Bulk API]({{site.url}}{{site.baseurl}}/api-reference/document-apis/bulk/) and provide a `batch_size` parameter. All batch-enabled processors have a batch mode and a single-document mode. When you ingest documents using the `PUT` method, the processor functions in single-document mode and processes documents in series. Currently, only the `text_embedding` and `sparse_encoding` processors are batch enabled. All other processors process documents one at a time.

## Selectively enabling processors

Processors defined by the [ingest-common module](https://github.com/opensearch-project/OpenSearch/blob/2.x/modules/ingest-common/src/main/java/org/opensearch/ingest/common/IngestCommonPlugin.java) can be selectively enabled by providing the `ingest-common.processors.allowed` cluster setting. If not provided, all processors are enabled by default. Specifying an empty list disables all processors. If the setting is changed to remove previously enabled processors, any pipeline using a disabled processor will fail after node restart when the new setting takes effect.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
4 changes: 4 additions & 0 deletions _search-plugins/search-pipelines/search-processors.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,3 +121,7 @@ The response contains the `search_pipelines` object that lists the available req

In addition to the processors provided by OpenSearch, additional processors may be provided by plugins.
{: .note}

## Selectively enabling processors

Processors defined by the [search-pipeline-common module](https://github.com/opensearch-project/OpenSearch/blob/2.x/modules/search-pipeline-common/src/main/java/org/opensearch/search/pipeline/common/SearchPipelineCommonModulePlugin.java) can be selectively enabled by providing the following static cluster settings: `search.pipeline.common.request.processors.allowed`, `search.pipeline.common.response.processors.allowed`, or `search.pipeline.common.search.phase.results.processors.allowed`. If not specified, then all processors are enabled by default. If an empty list is specified, then all processors are disabled. If the setting is changed to remove previously-enabled processors, then any pipeline using a disabled processor will fail after the node is restarted and the new setting takes effect.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved