[Search Pipelines] Add default_search_pipeline index setting #7470

msfroh · 2023-05-08T22:31:24Z

Description

Once users have defined and tested a search pipeline, they may want to apply it by default to all queries hitting a given index.

Users should be able to bypass the default pipeline for the index by explicitly specifying search_pipeline=_none in the URL parameter of their search request.

Related Issues

Resolves #6719

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-05-08T23:02:26Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/15068/
CommitID: b2adb8b
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-05-08T23:08:22Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/15069/
CommitID: 60f3718
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-05-09T00:00:58Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/15074/
CommitID: 7d1d96f

codecov-commenter · 2023-05-09T00:03:17Z

Codecov Report

Merging #7470 (0cb9894) into main (27dbcd5) will decrease coverage by 0.10%.
The diff coverage is 91.30%.

@@             Coverage Diff              @@
##               main    #7470      +/-   ##
============================================
- Coverage     70.54%   70.45%   -0.10%     
+ Complexity    59713    59677      -36     
============================================
  Files          4896     4896              
  Lines        286798   286815      +17     
  Branches      41331    41335       +4     
============================================
- Hits         202334   202063     -271     
- Misses        67761    68071     +310     
+ Partials      16703    16681      -22

Impacted Files	Coverage Δ
...pensearch/common/settings/IndexScopedSettings.java	`100.00% <ø> (ø)`
...nsearch/search/pipeline/SearchPipelineService.java	`82.81% <85.71%> (-0.34%)`	⬇️
.../main/java/org/opensearch/index/IndexSettings.java	`84.98% <100.00%> (+0.35%)`	⬆️

... and 501 files with indirect coverage changes

github-actions · 2023-05-09T00:10:41Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/15076/
CommitID: 9a98732

github-actions · 2023-05-12T00:04:29Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/15295/
CommitID: 1170cb5
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

reta · 2023-05-12T00:17:44Z

@msfroh could you please rebase?

kotwanikunal · 2023-05-12T00:56:04Z

Seeing a bunch of

org.opensearch.backwards.IndexingIT > testSeqNoCheckpoints FAILED
    org.opensearch.client.ResponseException: method [GET], host [http://[::1/]:37425], URI [test/_count?preference=_only_nodes%3Av2.8.0-3], status line [HTTP/1.1 500 Internal Server Error]
    {"error":{"root_cause":[{"type":"illegal_state_exception","reason":"unexpected byte [0x3f]"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"test","node":"6sjRAsVtRGa9N2kVkQUBlA","reason":{"type":"illegal_state_exception","reason":"unexpected byte [0x3f]"}}],"caused_by":{"type":"illegal_state_exception","reason":"unexpected byte [0x3f]","caused_by":{"type":"illegal_state_exception","reason":"unexpected byte [0x3f]"}}},"status":500}
        at __randomizedtesting.SeedInfo.seed([FDA6AED85DA67D18:CCCA44139739F844]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:384)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:354)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:329)
        at app//org.opensearch.backwards.IndexingIT.assertCount(IndexingIT.java:358)
        at app//org.opensearch.backwards.IndexingIT.testSeqNoCheckpoints(IndexingIT.java:213)

Ref: https://build.ci.opensearch.org/job/gradle-check/15296/consoleFull

Can it be because of the version mismatch between 2.x and main since #7253 was merged and this isn't?

reta · 2023-05-12T01:24:34Z

Can it be because of the version mismatch between 2.x and main since #7253 was merged and this isn't?

Yes, the fix is bundled there #7470

msfroh · 2023-05-12T18:15:21Z

could you please rebase?

@reta Done

github-actions · 2023-05-12T18:46:48Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/15323/
CommitID: d70a6fe

github-actions · 2023-05-12T23:16:13Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/15353/
CommitID: 42a4f8a
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

qa/mixed-cluster/build.gradle

Once users have defined and tested a search pipeline, they may want to apply it by default to all queries hitting a given index. Users should be able to bypass the default pipeline for the index by explicitly specifying `search_pipeline=_none` in the URL parameter of their search request. Signed-off-by: Michael Froh <[email protected]>

Also change version check for ad hoc pipeline to 2.8 Signed-off-by: Michael Froh <[email protected]>

Signed-off-by: Michael Froh <[email protected]>

github-actions · 2023-05-15T18:43:58Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/15449/
CommitID: 9dc30b4
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Michael Froh <[email protected]>

github-actions · 2023-05-15T21:35:44Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationIT.testScrollWithOngoingSegmentReplication

URL: https://build.ci.opensearch.org/job/gradle-check/15456/
CommitID: 0cb9894
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

msfroh · 2023-05-16T16:36:24Z

Merge conflicts have been fixed. Rebased onto main recently.

Any remaining blockers to merging? @kotwanikunal @andrross @reta

Thanks!

opensearch-trigger-bot · 2023-05-16T16:43:59Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-7470-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 59847356b1c82562498f38896adef66ea1702cd6
# Push it to GitHub
git push --set-upstream origin backport/backport-7470-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-7470-to-2.x.

andrross · 2023-05-16T16:49:39Z

@msfroh Can you pick up the backport since the automated cherry-pick failed?

…rch-project#7470) * [Search Pipelines] Add default_search_pipeline index setting Once users have defined and tested a search pipeline, they may want to apply it by default to all queries hitting a given index. Users should be able to bypass the default pipeline for the index by explicitly specifying `search_pipeline=_none` in the URL parameter of their search request. Signed-off-by: Michael Froh <[email protected]> * Rename default search pipeline setting and add getter/setter Also change version check for ad hoc pipeline to 2.8 Signed-off-by: Michael Froh <[email protected]> * Add tests for new IndexSettings methods Signed-off-by: Michael Froh <[email protected]> * Update test to reflect change to FeatureFlagSetter Signed-off-by: Michael Froh <[email protected]> --------- Signed-off-by: Michael Froh <[email protected]> (cherry picked from commit 5984735)

msfroh · 2023-05-17T01:32:47Z

Backport PR: #7589

@reta

…tting (#7470) (#7589) * [Search Pipelines] Add default_search_pipeline index setting (#7470) * [Search Pipelines] Add default_search_pipeline index setting Once users have defined and tested a search pipeline, they may want to apply it by default to all queries hitting a given index. Users should be able to bypass the default pipeline for the index by explicitly specifying `search_pipeline=_none` in the URL parameter of their search request. Signed-off-by: Michael Froh <[email protected]> * Rename default search pipeline setting and add getter/setter Also change version check for ad hoc pipeline to 2.8 Signed-off-by: Michael Froh <[email protected]> * Add tests for new IndexSettings methods Signed-off-by: Michael Froh <[email protected]> * Update test to reflect change to FeatureFlagSetter Signed-off-by: Michael Froh <[email protected]> --------- Signed-off-by: Michael Froh <[email protected]> (cherry picked from commit 5984735) * Only enable search pipelines for BWC tests after 2.7.0 Signed-off-by: Michael Froh <[email protected]> * Fix version check for search pipelines REST test I previously misunderstood the syntax for skipping tests by version. Thanks @reta for catching this! Signed-off-by: Michael Froh <[email protected]> --------- Signed-off-by: Michael Froh <[email protected]>

…rch-project#7470) * [Search Pipelines] Add default_search_pipeline index setting Once users have defined and tested a search pipeline, they may want to apply it by default to all queries hitting a given index. Users should be able to bypass the default pipeline for the index by explicitly specifying `search_pipeline=_none` in the URL parameter of their search request. Signed-off-by: Michael Froh <[email protected]> * Rename default search pipeline setting and add getter/setter Also change version check for ad hoc pipeline to 2.8 Signed-off-by: Michael Froh <[email protected]> * Add tests for new IndexSettings methods Signed-off-by: Michael Froh <[email protected]> * Update test to reflect change to FeatureFlagSetter Signed-off-by: Michael Froh <[email protected]> --------- Signed-off-by: Michael Froh <[email protected]>

…rch-project#7470) * [Search Pipelines] Add default_search_pipeline index setting Once users have defined and tested a search pipeline, they may want to apply it by default to all queries hitting a given index. Users should be able to bypass the default pipeline for the index by explicitly specifying `search_pipeline=_none` in the URL parameter of their search request. Signed-off-by: Michael Froh <[email protected]> * Rename default search pipeline setting and add getter/setter Also change version check for ad hoc pipeline to 2.8 Signed-off-by: Michael Froh <[email protected]> * Add tests for new IndexSettings methods Signed-off-by: Michael Froh <[email protected]> * Update test to reflect change to FeatureFlagSetter Signed-off-by: Michael Froh <[email protected]> --------- Signed-off-by: Michael Froh <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>

msfroh force-pushed the index_default_pipeline branch from b2adb8b to 60f3718 Compare May 8, 2023 22:32

msfroh force-pushed the index_default_pipeline branch from 60f3718 to 7d1d96f Compare May 8, 2023 23:30

msfroh changed the title ~~[Search Pipelines] Add default_search_pipline index setting~~ [Search Pipelines] Add default_search_pipeline index setting May 8, 2023

msfroh force-pushed the index_default_pipeline branch from 7d1d96f to 9a98732 Compare May 8, 2023 23:36

msfroh marked this pull request as ready for review May 9, 2023 00:16

msfroh requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja, dreamer-89, tlfeng and VachaShah as code owners May 9, 2023 00:16

This was referenced May 12, 2023

Change block check type for index clear cache APIs #7538

Merged

[AUTOCUT] Gradle Check Failure on push to main #7535

Closed

msfroh force-pushed the index_default_pipeline branch from 1170cb5 to d70a6fe Compare May 12, 2023 18:14

noCharger reviewed May 15, 2023

View reviewed changes

qa/mixed-cluster/build.gradle Show resolved Hide resolved

msfroh added 3 commits May 15, 2023 18:23

Rename default search pipeline setting and add getter/setter

f523c8a

Also change version check for ad hoc pipeline to 2.8 Signed-off-by: Michael Froh <[email protected]>

Add tests for new IndexSettings methods

9dc30b4

Signed-off-by: Michael Froh <[email protected]>

msfroh force-pushed the index_default_pipeline branch from 42a4f8a to 9dc30b4 Compare May 15, 2023 18:24

Update test to reflect change to FeatureFlagSetter

0cb9894

Signed-off-by: Michael Froh <[email protected]>

andrross approved these changes May 16, 2023

View reviewed changes

andrross merged commit 5984735 into opensearch-project:main May 16, 2023

msfroh mentioned this pull request Nov 2, 2023

[Search Pipelines] Apply default search pipelines to requests that don't target a single index #11058

Closed

owaiskazi19 mentioned this pull request Apr 18, 2024

[Search Pipeline]Handled default search pipeline for multiple indices #13276

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Search Pipelines] Add default_search_pipeline index setting #7470

[Search Pipelines] Add default_search_pipeline index setting #7470

msfroh commented May 8, 2023 •

edited

Loading

github-actions bot commented May 8, 2023

github-actions bot commented May 8, 2023

github-actions bot commented May 9, 2023

codecov-commenter commented May 9, 2023 •

edited by codecov bot

Loading

github-actions bot commented May 9, 2023

github-actions bot commented May 12, 2023

reta commented May 12, 2023

kotwanikunal commented May 12, 2023

reta commented May 12, 2023

msfroh commented May 12, 2023

github-actions bot commented May 12, 2023

github-actions bot commented May 12, 2023

github-actions bot commented May 15, 2023

github-actions bot commented May 15, 2023

msfroh commented May 16, 2023

opensearch-trigger-bot bot commented May 16, 2023

andrross commented May 16, 2023

msfroh commented May 17, 2023

[Search Pipelines] Add default_search_pipeline index setting #7470

[Search Pipelines] Add default_search_pipeline index setting #7470

Conversation

msfroh commented May 8, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented May 8, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented May 8, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented May 9, 2023

Gradle Check (Jenkins) Run Completed with:

codecov-commenter commented May 9, 2023 • edited by codecov bot Loading

Codecov Report

github-actions bot commented May 9, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented May 12, 2023

Gradle Check (Jenkins) Run Completed with:

reta commented May 12, 2023

kotwanikunal commented May 12, 2023

reta commented May 12, 2023

msfroh commented May 12, 2023

github-actions bot commented May 12, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented May 12, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented May 15, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented May 15, 2023

Gradle Check (Jenkins) Run Completed with:

msfroh commented May 16, 2023

opensearch-trigger-bot bot commented May 16, 2023

andrross commented May 16, 2023

msfroh commented May 17, 2023

msfroh commented May 8, 2023 •

edited

Loading

codecov-commenter commented May 9, 2023 •

edited by codecov bot

Loading