Adds method_parameters in neural search query to support ef_search #787

shatejas · 2024-06-11T21:06:22Z

Description

Adds method_parameters in neural search

Issues Resolved

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed as per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

martin-gaievski

changelog entry is missing

src/main/java/org/opensearch/neuralsearch/common/MinClusterVersionUtil.java

src/main/java/org/opensearch/neuralsearch/query/NeuralQueryBuilder.java

shatejas · 2024-06-12T23:51:10Z

changelog entry is missing

Its being merged in feature branch. Was planning to add it when merging in main. does that work?

navneet1v · 2024-06-13T18:07:08Z

src/main/java/org/opensearch/neuralsearch/query/NeuralQueryBuilder.java

@@ -174,6 +178,9 @@ protected void doXContent(XContentBuilder xContentBuilder, Params params) throws
        if (Objects.nonNull(minScore)) {
            xContentBuilder.field(MIN_SCORE_FIELD.getPreferredName(), minScore);
        }
+        if (Objects.nonNull(methodParameters)) {
+            MethodParametersParser.doXContent(xContentBuilder, methodParameters);


is this class MethodParametersParser coming from k-NN?

This is good. Lets ensure that we add some more docs on the MethodParametersParser function that it is now going beyond the boundaries of k-NN plugin. I would hate to see if we change signature of these functions and then neural plugin starts to break.

https://github.com/opensearch-project/k-NN/pull/1758/files

martin-gaievski

We need to think more on how to commonize logic that's shared between neural search and knn plugins. May be a too heavy lift for this change. At the moment this PR is good, and I see we do have an issue to follow up on later: opensearch-project/k-NN#1756

martin-gaievski · 2024-06-14T00:43:38Z

Please check why CI is failing, probably need to rebase on latest main

shatejas · 2024-06-14T17:17:24Z

Please check why CI is failing, probably need to rebase on latest main

The build is failing because the knn changes aren't in main yet. It is expected to fail as it cannot identify MethodParameterParser. Locally I published to local maven repo for neural to build

navneet1v · 2024-06-17T20:21:51Z

@vibrantvarun can you check

qa/restart-upgrade/src/test/java/org/opensearch/neuralsearch/bwc/HybridSearchIT.java

vibrantvarun · 2024-06-17T20:26:35Z

qa/rolling-upgrade/src/test/java/org/opensearch/neuralsearch/bwc/HybridSearchIT.java

@@ -73,6 +73,7 @@ public void testNormalizationProcessor_whenIndexWithMultipleShards_E2EFlow() thr
                    loadModel(modelId);
                    addDocument(getIndexNameForTest(), "2", TEST_FIELD, TEXT_UPGRADED, null, null);
                    validateTestIndexOnUpgrade(totalDocsCountUpgraded, modelId);
+                    validateTestIndexOnUpgrade(totalDocsCountUpgraded, modelId, Map.of("ef_search", 100));


Same comment as above.

Same reply as above

qa/rolling-upgrade/src/test/resources/processor/IndexMappings.json

vibrantvarun · 2024-06-17T20:43:52Z

There are lot of compilation errors which are not related to KNN. @shatejas Can you check why it is happening?

Signed-off-by: Tejas Shah <[email protected]>

shatejas · 2024-06-17T20:57:46Z

There are lot of compilation errors which are not related to KNN. @shatejas Can you check why it is happening?

See this comment

…pensearch-project#787) Signed-off-by: Tejas Shah <[email protected]>

) (#814) Signed-off-by: Tejas Shah <[email protected]>

) (#814) Signed-off-by: Tejas Shah <[email protected]> (cherry picked from commit dea1fe3)

) (#814) (#818) Signed-off-by: Tejas Shah <[email protected]> (cherry picked from commit dea1fe3) Co-authored-by: Tejas Shah <[email protected]>

…pensearch-project#787) (opensearch-project#814) Signed-off-by: Tejas Shah <[email protected]>

* Adds method_parameters in neural search query to support ef_search (#787) (#814) Signed-off-by: Tejas Shah <[email protected]> * Add BWC for batch ingestion (#769) * Add BWC for batch ingestion Signed-off-by: Liyun Xiu <[email protected]> * Update Changelog Signed-off-by: Liyun Xiu <[email protected]> * Fix spotlessLicenseCheck Signed-off-by: Liyun Xiu <[email protected]> * Fix comments Signed-off-by: Liyun Xiu <[email protected]> * Reuse the same code Signed-off-by: Liyun Xiu <[email protected]> * Rename some functions Signed-off-by: Liyun Xiu <[email protected]> * Rename a function Signed-off-by: Liyun Xiu <[email protected]> * Minor change to trigger rebuild Signed-off-by: Liyun Xiu <[email protected]> --------- Signed-off-by: Liyun Xiu <[email protected]> * Neural sparse query two-phase search processor's bwc test (#777) * Poc of pipeline Signed-off-by: conggguan <[email protected]> * Complete some settings for two phase pipeline. Signed-off-by: conggguan <[email protected]> * Change the implement of two-phase from QueryBuilderVistor to custom process funciton. Signed-off-by: conggguan <[email protected]> * Add It and fix some bug on the state of multy same neuralsparsequerybuilder. Signed-off-by: conggguan <[email protected]> * Simplify some logic, and correct some format. Signed-off-by: conggguan <[email protected]> * Optimize some format. Signed-off-by: conggguan <[email protected]> * Add some test case. Signed-off-by: conggguan <[email protected]> * Optimize some logic for zhichao-aws's comments. Signed-off-by: conggguan <[email protected]> * Optimize a line without application. Signed-off-by: conggguan <[email protected]> * Add some comments, remove some redundant lines, fix some format. Signed-off-by: conggguan <[email protected]> * Remove a redundant null check, fix a if format. Signed-off-by: conggguan <[email protected]> * Fix a typo for a comment, camelcase format for some variable. Signed-off-by: conggguan <[email protected]> * Add some comments to illustrate the influence of the modify on 2-phase search pipeline to neural sparse query builder. Signed-off-by: conggguan <[email protected]> * Add restart and rolling upgrade bwc test for neural sparse two phase processor. Signed-off-by: conggguan <[email protected]> * Spotless on qa. Signed-off-by: conggguan <[email protected]> * Update change log for two-phase BWC test. Signed-off-by: conggguan <[email protected]> * Remove redundant lines of two-phase BWC test. Signed-off-by: conggguan <[email protected]> * Add changelog. Signed-off-by: conggguan <[email protected]> * Add the PR link and number for the CHANGELOG.md. Signed-off-by: conggguan <[email protected]> * [Fix] NeuralSparseTwoPhaseProcessorIT created wrong ingest pipeline, fix it to correct API. Signed-off-by: conggguan <[email protected]> --------- Signed-off-by: conggguan <[email protected]> Signed-off-by: conggguan <[email protected]> * Enable '.' for nested field in text embedding processor (#811) * Added nested structure for text embed processor mapping Signed-off-by: Martin Gaievski <[email protected]> * Fix linux build CI error due to action runner env upgrade node 20 (#821) * Fix linux build CI error due to action runner env upgrade node 20 Signed-off-by: Varun Jain <[email protected]> * Fix linux build on additional integ tests Signed-off-by: Varun Jain <[email protected]> --------- Signed-off-by: Varun Jain <[email protected]> --------- Signed-off-by: Tejas Shah <[email protected]> Signed-off-by: Liyun Xiu <[email protected]> Signed-off-by: conggguan <[email protected]> Signed-off-by: conggguan <[email protected]> Signed-off-by: Martin Gaievski <[email protected]> Signed-off-by: Varun Jain <[email protected]> Co-authored-by: Tejas Shah <[email protected]> Co-authored-by: Liyun Xiu <[email protected]> Co-authored-by: conggguan <[email protected]> Co-authored-by: Martin Gaievski <[email protected]>

…827) * Fix jdk version for CI test secure cluster action (#801) (#806) Signed-off-by: Martin Gaievski <[email protected]> Co-authored-by: Martin Gaievski <[email protected]> * [Part 1] Collector for Sorting Results (#797) * [Part 2] Normalization Phase for Sorting (#802) * Normalization Phase for Sorting Signed-off-by: Varun Jain <[email protected]> * Fixing compile test issue Signed-off-by: Varun Jain <[email protected]> * Optimize code Signed-off-by: Varun Jain <[email protected]> * Add method description Signed-off-by: Varun Jain <[email protected]> * [Part 1] Collector for Sorting Results (#797) * HybridSearchSortUtil class Signed-off-by: Varun Jain <[email protected]> * Add Integ Tests Signed-off-by: Varun Jain <[email protected]> * Add Sorting Integ tests Signed-off-by: Varun Jain <[email protected]> * Add integ test for Sorting Signed-off-by: Varun Jain <[email protected]> * Refactoring normalization processor workflow Signed-off-by: Varun Jain <[email protected]> * Fix Unit Tests Signed-off-by: Varun Jain <[email protected]> * Refactoring Signed-off-by: Varun Jain <[email protected]> * Refactoring Signed-off-by: Varun Jain <[email protected]> * Address Martin Comments Signed-off-by: Varun Jain <[email protected]> * Optimising Normalization Signed-off-by: Varun Jain <[email protected]> * Address Martin Comments Signed-off-by: Varun Jain <[email protected]> * Address Martin Comments Signed-off-by: Varun Jain <[email protected]> * Addressing Martin Comments Signed-off-by: Varun Jain <[email protected]> * Addressing Vijay comments Signed-off-by: Varun Jain <[email protected]> * Address Vijay Comments Signed-off-by: Varun Jain <[email protected]> --------- Signed-off-by: Varun Jain <[email protected]> * Update bwc workflow to include 2.16.0-SNAPSHOT (#809) (#810) * Increment BWC version * Append 2.16.0-SNAPSHOTn in restart upgrade tests --------- Signed-off-by: Varun Jain <[email protected]> * [Part 3] Concurrent segment search bug in Sorting (#808) * Cherry picking Concurrent Segment Search Bug Commit Signed-off-by: Varun Jain <[email protected]> * Fix Concurrent Segment Search Bug in Sorting Signed-off-by: Varun Jain <[email protected]> * Functional Interface Signed-off-by: Varun Jain <[email protected]> * Addressing Martin Comments Signed-off-by: Varun Jain <[email protected]> * Removing comments Signed-off-by: Varun Jain <[email protected]> * Addressing Martin Comments Signed-off-by: Varun Jain <[email protected]> * Addressing Martin Comments Signed-off-by: Varun Jain <[email protected]> * Addressing Martin commnents Signed-off-by: Varun Jain <[email protected]> * Address Martin Comments Signed-off-by: Varun Jain <[email protected]> * Address Martin Comments Signed-off-by: Varun Jain <[email protected]> --------- Signed-off-by: Varun Jain <[email protected]> Co-authored-by: Martin Gaievski <[email protected]> * Rebasing with main (#826) * Adds method_parameters in neural search query to support ef_search (#787) (#814) Signed-off-by: Tejas Shah <[email protected]> * Add BWC for batch ingestion (#769) * Add BWC for batch ingestion Signed-off-by: Liyun Xiu <[email protected]> * Update Changelog Signed-off-by: Liyun Xiu <[email protected]> * Fix spotlessLicenseCheck Signed-off-by: Liyun Xiu <[email protected]> * Fix comments Signed-off-by: Liyun Xiu <[email protected]> * Reuse the same code Signed-off-by: Liyun Xiu <[email protected]> * Rename some functions Signed-off-by: Liyun Xiu <[email protected]> * Rename a function Signed-off-by: Liyun Xiu <[email protected]> * Minor change to trigger rebuild Signed-off-by: Liyun Xiu <[email protected]> --------- Signed-off-by: Liyun Xiu <[email protected]> * Neural sparse query two-phase search processor's bwc test (#777) * Poc of pipeline Signed-off-by: conggguan <[email protected]> * Complete some settings for two phase pipeline. Signed-off-by: conggguan <[email protected]> * Change the implement of two-phase from QueryBuilderVistor to custom process funciton. Signed-off-by: conggguan <[email protected]> * Add It and fix some bug on the state of multy same neuralsparsequerybuilder. Signed-off-by: conggguan <[email protected]> * Simplify some logic, and correct some format. Signed-off-by: conggguan <[email protected]> * Optimize some format. Signed-off-by: conggguan <[email protected]> * Add some test case. Signed-off-by: conggguan <[email protected]> * Optimize some logic for zhichao-aws's comments. Signed-off-by: conggguan <[email protected]> * Optimize a line without application. Signed-off-by: conggguan <[email protected]> * Add some comments, remove some redundant lines, fix some format. Signed-off-by: conggguan <[email protected]> * Remove a redundant null check, fix a if format. Signed-off-by: conggguan <[email protected]> * Fix a typo for a comment, camelcase format for some variable. Signed-off-by: conggguan <[email protected]> * Add some comments to illustrate the influence of the modify on 2-phase search pipeline to neural sparse query builder. Signed-off-by: conggguan <[email protected]> * Add restart and rolling upgrade bwc test for neural sparse two phase processor. Signed-off-by: conggguan <[email protected]> * Spotless on qa. Signed-off-by: conggguan <[email protected]> * Update change log for two-phase BWC test. Signed-off-by: conggguan <[email protected]> * Remove redundant lines of two-phase BWC test. Signed-off-by: conggguan <[email protected]> * Add changelog. Signed-off-by: conggguan <[email protected]> * Add the PR link and number for the CHANGELOG.md. Signed-off-by: conggguan <[email protected]> * [Fix] NeuralSparseTwoPhaseProcessorIT created wrong ingest pipeline, fix it to correct API. Signed-off-by: conggguan <[email protected]> --------- Signed-off-by: conggguan <[email protected]> Signed-off-by: conggguan <[email protected]> * Enable '.' for nested field in text embedding processor (#811) * Added nested structure for text embed processor mapping Signed-off-by: Martin Gaievski <[email protected]> * Fix linux build CI error due to action runner env upgrade node 20 (#821) * Fix linux build CI error due to action runner env upgrade node 20 Signed-off-by: Varun Jain <[email protected]> * Fix linux build on additional integ tests Signed-off-by: Varun Jain <[email protected]> --------- Signed-off-by: Varun Jain <[email protected]> --------- Signed-off-by: Tejas Shah <[email protected]> Signed-off-by: Liyun Xiu <[email protected]> Signed-off-by: conggguan <[email protected]> Signed-off-by: conggguan <[email protected]> Signed-off-by: Martin Gaievski <[email protected]> Signed-off-by: Varun Jain <[email protected]> Co-authored-by: Tejas Shah <[email protected]> Co-authored-by: Liyun Xiu <[email protected]> Co-authored-by: conggguan <[email protected]> Co-authored-by: Martin Gaievski <[email protected]> * Add changelog Signed-off-by: Varun Jain <[email protected]> --------- Signed-off-by: Martin Gaievski <[email protected]> Signed-off-by: Varun Jain <[email protected]> Signed-off-by: Tejas Shah <[email protected]> Signed-off-by: Liyun Xiu <[email protected]> Signed-off-by: conggguan <[email protected]> Signed-off-by: conggguan <[email protected]> Co-authored-by: Martin Gaievski <[email protected]> Co-authored-by: Tejas Shah <[email protected]> Co-authored-by: Liyun Xiu <[email protected]> Co-authored-by: conggguan <[email protected]>

…827) * Fix jdk version for CI test secure cluster action (#801) (#806) Signed-off-by: Martin Gaievski <[email protected]> Co-authored-by: Martin Gaievski <[email protected]> * [Part 1] Collector for Sorting Results (#797) * [Part 2] Normalization Phase for Sorting (#802) * Normalization Phase for Sorting Signed-off-by: Varun Jain <[email protected]> * Fixing compile test issue Signed-off-by: Varun Jain <[email protected]> * Optimize code Signed-off-by: Varun Jain <[email protected]> * Add method description Signed-off-by: Varun Jain <[email protected]> * [Part 1] Collector for Sorting Results (#797) * HybridSearchSortUtil class Signed-off-by: Varun Jain <[email protected]> * Add Integ Tests Signed-off-by: Varun Jain <[email protected]> * Add Sorting Integ tests Signed-off-by: Varun Jain <[email protected]> * Add integ test for Sorting Signed-off-by: Varun Jain <[email protected]> * Refactoring normalization processor workflow Signed-off-by: Varun Jain <[email protected]> * Fix Unit Tests Signed-off-by: Varun Jain <[email protected]> * Refactoring Signed-off-by: Varun Jain <[email protected]> * Refactoring Signed-off-by: Varun Jain <[email protected]> * Address Martin Comments Signed-off-by: Varun Jain <[email protected]> * Optimising Normalization Signed-off-by: Varun Jain <[email protected]> * Address Martin Comments Signed-off-by: Varun Jain <[email protected]> * Address Martin Comments Signed-off-by: Varun Jain <[email protected]> * Addressing Martin Comments Signed-off-by: Varun Jain <[email protected]> * Addressing Vijay comments Signed-off-by: Varun Jain <[email protected]> * Address Vijay Comments Signed-off-by: Varun Jain <[email protected]> --------- Signed-off-by: Varun Jain <[email protected]> * Update bwc workflow to include 2.16.0-SNAPSHOT (#809) (#810) * Increment BWC version * Append 2.16.0-SNAPSHOTn in restart upgrade tests --------- Signed-off-by: Varun Jain <[email protected]> * [Part 3] Concurrent segment search bug in Sorting (#808) * Cherry picking Concurrent Segment Search Bug Commit Signed-off-by: Varun Jain <[email protected]> * Fix Concurrent Segment Search Bug in Sorting Signed-off-by: Varun Jain <[email protected]> * Functional Interface Signed-off-by: Varun Jain <[email protected]> * Addressing Martin Comments Signed-off-by: Varun Jain <[email protected]> * Removing comments Signed-off-by: Varun Jain <[email protected]> * Addressing Martin Comments Signed-off-by: Varun Jain <[email protected]> * Addressing Martin Comments Signed-off-by: Varun Jain <[email protected]> * Addressing Martin commnents Signed-off-by: Varun Jain <[email protected]> * Address Martin Comments Signed-off-by: Varun Jain <[email protected]> * Address Martin Comments Signed-off-by: Varun Jain <[email protected]> --------- Signed-off-by: Varun Jain <[email protected]> Co-authored-by: Martin Gaievski <[email protected]> * Rebasing with main (#826) * Adds method_parameters in neural search query to support ef_search (#787) (#814) Signed-off-by: Tejas Shah <[email protected]> * Add BWC for batch ingestion (#769) * Add BWC for batch ingestion Signed-off-by: Liyun Xiu <[email protected]> * Update Changelog Signed-off-by: Liyun Xiu <[email protected]> * Fix spotlessLicenseCheck Signed-off-by: Liyun Xiu <[email protected]> * Fix comments Signed-off-by: Liyun Xiu <[email protected]> * Reuse the same code Signed-off-by: Liyun Xiu <[email protected]> * Rename some functions Signed-off-by: Liyun Xiu <[email protected]> * Rename a function Signed-off-by: Liyun Xiu <[email protected]> * Minor change to trigger rebuild Signed-off-by: Liyun Xiu <[email protected]> --------- Signed-off-by: Liyun Xiu <[email protected]> * Neural sparse query two-phase search processor's bwc test (#777) * Poc of pipeline Signed-off-by: conggguan <[email protected]> * Complete some settings for two phase pipeline. Signed-off-by: conggguan <[email protected]> * Change the implement of two-phase from QueryBuilderVistor to custom process funciton. Signed-off-by: conggguan <[email protected]> * Add It and fix some bug on the state of multy same neuralsparsequerybuilder. Signed-off-by: conggguan <[email protected]> * Simplify some logic, and correct some format. Signed-off-by: conggguan <[email protected]> * Optimize some format. Signed-off-by: conggguan <[email protected]> * Add some test case. Signed-off-by: conggguan <[email protected]> * Optimize some logic for zhichao-aws's comments. Signed-off-by: conggguan <[email protected]> * Optimize a line without application. Signed-off-by: conggguan <[email protected]> * Add some comments, remove some redundant lines, fix some format. Signed-off-by: conggguan <[email protected]> * Remove a redundant null check, fix a if format. Signed-off-by: conggguan <[email protected]> * Fix a typo for a comment, camelcase format for some variable. Signed-off-by: conggguan <[email protected]> * Add some comments to illustrate the influence of the modify on 2-phase search pipeline to neural sparse query builder. Signed-off-by: conggguan <[email protected]> * Add restart and rolling upgrade bwc test for neural sparse two phase processor. Signed-off-by: conggguan <[email protected]> * Spotless on qa. Signed-off-by: conggguan <[email protected]> * Update change log for two-phase BWC test. Signed-off-by: conggguan <[email protected]> * Remove redundant lines of two-phase BWC test. Signed-off-by: conggguan <[email protected]> * Add changelog. Signed-off-by: conggguan <[email protected]> * Add the PR link and number for the CHANGELOG.md. Signed-off-by: conggguan <[email protected]> * [Fix] NeuralSparseTwoPhaseProcessorIT created wrong ingest pipeline, fix it to correct API. Signed-off-by: conggguan <[email protected]> --------- Signed-off-by: conggguan <[email protected]> Signed-off-by: conggguan <[email protected]> * Enable '.' for nested field in text embedding processor (#811) * Added nested structure for text embed processor mapping Signed-off-by: Martin Gaievski <[email protected]> * Fix linux build CI error due to action runner env upgrade node 20 (#821) * Fix linux build CI error due to action runner env upgrade node 20 Signed-off-by: Varun Jain <[email protected]> * Fix linux build on additional integ tests Signed-off-by: Varun Jain <[email protected]> --------- Signed-off-by: Varun Jain <[email protected]> --------- Signed-off-by: Tejas Shah <[email protected]> Signed-off-by: Liyun Xiu <[email protected]> Signed-off-by: conggguan <[email protected]> Signed-off-by: conggguan <[email protected]> Signed-off-by: Martin Gaievski <[email protected]> Signed-off-by: Varun Jain <[email protected]> Co-authored-by: Tejas Shah <[email protected]> Co-authored-by: Liyun Xiu <[email protected]> Co-authored-by: conggguan <[email protected]> Co-authored-by: Martin Gaievski <[email protected]> * Add changelog Signed-off-by: Varun Jain <[email protected]> --------- Signed-off-by: Martin Gaievski <[email protected]> Signed-off-by: Varun Jain <[email protected]> Signed-off-by: Tejas Shah <[email protected]> Signed-off-by: Liyun Xiu <[email protected]> Signed-off-by: conggguan <[email protected]> Signed-off-by: conggguan <[email protected]> Co-authored-by: Martin Gaievski <[email protected]> Co-authored-by: Tejas Shah <[email protected]> Co-authored-by: Liyun Xiu <[email protected]> Co-authored-by: conggguan <[email protected]> (cherry picked from commit d22e1b8)

…827) (#829) * Fix jdk version for CI test secure cluster action (#801) (#806) Signed-off-by: Martin Gaievski <[email protected]> Co-authored-by: Martin Gaievski <[email protected]> * [Part 1] Collector for Sorting Results (#797) * [Part 2] Normalization Phase for Sorting (#802) * Normalization Phase for Sorting Signed-off-by: Varun Jain <[email protected]> * Fixing compile test issue Signed-off-by: Varun Jain <[email protected]> * Optimize code Signed-off-by: Varun Jain <[email protected]> * Add method description Signed-off-by: Varun Jain <[email protected]> * [Part 1] Collector for Sorting Results (#797) * HybridSearchSortUtil class Signed-off-by: Varun Jain <[email protected]> * Add Integ Tests Signed-off-by: Varun Jain <[email protected]> * Add Sorting Integ tests Signed-off-by: Varun Jain <[email protected]> * Add integ test for Sorting Signed-off-by: Varun Jain <[email protected]> * Refactoring normalization processor workflow Signed-off-by: Varun Jain <[email protected]> * Fix Unit Tests Signed-off-by: Varun Jain <[email protected]> * Refactoring Signed-off-by: Varun Jain <[email protected]> * Refactoring Signed-off-by: Varun Jain <[email protected]> * Address Martin Comments Signed-off-by: Varun Jain <[email protected]> * Optimising Normalization Signed-off-by: Varun Jain <[email protected]> * Address Martin Comments Signed-off-by: Varun Jain <[email protected]> * Address Martin Comments Signed-off-by: Varun Jain <[email protected]> * Addressing Martin Comments Signed-off-by: Varun Jain <[email protected]> * Addressing Vijay comments Signed-off-by: Varun Jain <[email protected]> * Address Vijay Comments Signed-off-by: Varun Jain <[email protected]> --------- Signed-off-by: Varun Jain <[email protected]> * Update bwc workflow to include 2.16.0-SNAPSHOT (#809) (#810) * Increment BWC version * Append 2.16.0-SNAPSHOTn in restart upgrade tests --------- Signed-off-by: Varun Jain <[email protected]> * [Part 3] Concurrent segment search bug in Sorting (#808) * Cherry picking Concurrent Segment Search Bug Commit Signed-off-by: Varun Jain <[email protected]> * Fix Concurrent Segment Search Bug in Sorting Signed-off-by: Varun Jain <[email protected]> * Functional Interface Signed-off-by: Varun Jain <[email protected]> * Addressing Martin Comments Signed-off-by: Varun Jain <[email protected]> * Removing comments Signed-off-by: Varun Jain <[email protected]> * Addressing Martin Comments Signed-off-by: Varun Jain <[email protected]> * Addressing Martin Comments Signed-off-by: Varun Jain <[email protected]> * Addressing Martin commnents Signed-off-by: Varun Jain <[email protected]> * Address Martin Comments Signed-off-by: Varun Jain <[email protected]> * Address Martin Comments Signed-off-by: Varun Jain <[email protected]> --------- Signed-off-by: Varun Jain <[email protected]> Co-authored-by: Martin Gaievski <[email protected]> * Rebasing with main (#826) * Adds method_parameters in neural search query to support ef_search (#787) (#814) Signed-off-by: Tejas Shah <[email protected]> * Add BWC for batch ingestion (#769) * Add BWC for batch ingestion Signed-off-by: Liyun Xiu <[email protected]> * Update Changelog Signed-off-by: Liyun Xiu <[email protected]> * Fix spotlessLicenseCheck Signed-off-by: Liyun Xiu <[email protected]> * Fix comments Signed-off-by: Liyun Xiu <[email protected]> * Reuse the same code Signed-off-by: Liyun Xiu <[email protected]> * Rename some functions Signed-off-by: Liyun Xiu <[email protected]> * Rename a function Signed-off-by: Liyun Xiu <[email protected]> * Minor change to trigger rebuild Signed-off-by: Liyun Xiu <[email protected]> --------- Signed-off-by: Liyun Xiu <[email protected]> * Neural sparse query two-phase search processor's bwc test (#777) * Poc of pipeline Signed-off-by: conggguan <[email protected]> * Complete some settings for two phase pipeline. Signed-off-by: conggguan <[email protected]> * Change the implement of two-phase from QueryBuilderVistor to custom process funciton. Signed-off-by: conggguan <[email protected]> * Add It and fix some bug on the state of multy same neuralsparsequerybuilder. Signed-off-by: conggguan <[email protected]> * Simplify some logic, and correct some format. Signed-off-by: conggguan <[email protected]> * Optimize some format. Signed-off-by: conggguan <[email protected]> * Add some test case. Signed-off-by: conggguan <[email protected]> * Optimize some logic for zhichao-aws's comments. Signed-off-by: conggguan <[email protected]> * Optimize a line without application. Signed-off-by: conggguan <[email protected]> * Add some comments, remove some redundant lines, fix some format. Signed-off-by: conggguan <[email protected]> * Remove a redundant null check, fix a if format. Signed-off-by: conggguan <[email protected]> * Fix a typo for a comment, camelcase format for some variable. Signed-off-by: conggguan <[email protected]> * Add some comments to illustrate the influence of the modify on 2-phase search pipeline to neural sparse query builder. Signed-off-by: conggguan <[email protected]> * Add restart and rolling upgrade bwc test for neural sparse two phase processor. Signed-off-by: conggguan <[email protected]> * Spotless on qa. Signed-off-by: conggguan <[email protected]> * Update change log for two-phase BWC test. Signed-off-by: conggguan <[email protected]> * Remove redundant lines of two-phase BWC test. Signed-off-by: conggguan <[email protected]> * Add changelog. Signed-off-by: conggguan <[email protected]> * Add the PR link and number for the CHANGELOG.md. Signed-off-by: conggguan <[email protected]> * [Fix] NeuralSparseTwoPhaseProcessorIT created wrong ingest pipeline, fix it to correct API. Signed-off-by: conggguan <[email protected]> --------- Signed-off-by: conggguan <[email protected]> Signed-off-by: conggguan <[email protected]> * Enable '.' for nested field in text embedding processor (#811) * Added nested structure for text embed processor mapping Signed-off-by: Martin Gaievski <[email protected]> * Fix linux build CI error due to action runner env upgrade node 20 (#821) * Fix linux build CI error due to action runner env upgrade node 20 Signed-off-by: Varun Jain <[email protected]> * Fix linux build on additional integ tests Signed-off-by: Varun Jain <[email protected]> --------- Signed-off-by: Varun Jain <[email protected]> --------- Signed-off-by: Tejas Shah <[email protected]> Signed-off-by: Liyun Xiu <[email protected]> Signed-off-by: conggguan <[email protected]> Signed-off-by: conggguan <[email protected]> Signed-off-by: Martin Gaievski <[email protected]> Signed-off-by: Varun Jain <[email protected]> Co-authored-by: Tejas Shah <[email protected]> Co-authored-by: Liyun Xiu <[email protected]> Co-authored-by: conggguan <[email protected]> Co-authored-by: Martin Gaievski <[email protected]> * Add changelog Signed-off-by: Varun Jain <[email protected]> --------- Signed-off-by: Martin Gaievski <[email protected]> Signed-off-by: Varun Jain <[email protected]> Signed-off-by: Tejas Shah <[email protected]> Signed-off-by: Liyun Xiu <[email protected]> Signed-off-by: conggguan <[email protected]> Signed-off-by: conggguan <[email protected]> Co-authored-by: Martin Gaievski <[email protected]> Co-authored-by: Tejas Shah <[email protected]> Co-authored-by: Liyun Xiu <[email protected]> Co-authored-by: conggguan <[email protected]> (cherry picked from commit d22e1b8) Co-authored-by: Varun Jain <[email protected]>

shatejas requested review from heemin32, navneet1v, VijayanB, vamshin, jmazanec15, naveentatikonda, junqiu-lei, martin-gaievski, sean-zheng-amazon, model-collapse, zane-neo, ylwu-amzn, jngz-es, vibrantvarun and zhichao-aws as code owners June 11, 2024 21:06

martin-gaievski reviewed Jun 12, 2024

View reviewed changes

shatejas force-pushed the feature/ef-search branch from cda168e to 0d9a6b2 Compare June 13, 2024 00:09

navneet1v reviewed Jun 13, 2024

View reviewed changes

martin-gaievski approved these changes Jun 14, 2024

View reviewed changes

martin-gaievski added the skip-changelog label Jun 14, 2024

shatejas requested a review from navneet1v June 14, 2024 17:24

vibrantvarun reviewed Jun 17, 2024

View reviewed changes

Adds method_parameters in neural search query to support ef_search

0983ba2

Signed-off-by: Tejas Shah <[email protected]>

shatejas force-pushed the feature/ef-search branch from 0d9a6b2 to 0983ba2 Compare June 17, 2024 20:55

shatejas requested a review from vibrantvarun June 17, 2024 20:58

vibrantvarun approved these changes Jun 17, 2024

View reviewed changes

vibrantvarun merged commit 726a482 into opensearch-project:feature/ef-search Jun 17, 2024
9 of 72 checks passed

shatejas added a commit to shatejas/neural-search that referenced this pull request Jul 3, 2024

Adds method_parameters in neural search query to support ef_search (o…

e770cb9

…pensearch-project#787) Signed-off-by: Tejas Shah <[email protected]>

shatejas added a commit to shatejas/neural-search that referenced this pull request Jul 3, 2024

Adds method_parameters in neural search query to support ef_search (o…

bcb2566

…pensearch-project#787) Signed-off-by: Tejas Shah <[email protected]>

shatejas mentioned this pull request Jul 4, 2024

[Enhancement] Adds method_parameters in neural search query to support ef_search #814

Merged

5 tasks

navneet1v pushed a commit that referenced this pull request Jul 4, 2024

Adds method_parameters in neural search query to support ef_search (#787

dea1fe3

) (#814) Signed-off-by: Tejas Shah <[email protected]>

opensearch-trigger-bot bot pushed a commit that referenced this pull request Jul 4, 2024

Adds method_parameters in neural search query to support ef_search (#787

6e760ea

) (#814) Signed-off-by: Tejas Shah <[email protected]> (cherry picked from commit dea1fe3)

vibrantvarun pushed a commit to vibrantvarun/neural-search that referenced this pull request Jul 9, 2024

Adds method_parameters in neural search query to support ef_search (o…

935829a

…pensearch-project#787) (opensearch-project#814) Signed-off-by: Tejas Shah <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds method_parameters in neural search query to support ef_search #787

Adds method_parameters in neural search query to support ef_search #787

shatejas commented Jun 11, 2024

martin-gaievski left a comment

shatejas commented Jun 12, 2024

navneet1v Jun 13, 2024

shatejas Jun 13, 2024

navneet1v Jun 13, 2024

shatejas Jun 14, 2024

martin-gaievski left a comment

martin-gaievski commented Jun 14, 2024

shatejas commented Jun 14, 2024

navneet1v commented Jun 17, 2024

vibrantvarun Jun 17, 2024

shatejas Jun 17, 2024

vibrantvarun commented Jun 17, 2024

shatejas commented Jun 17, 2024

Adds method_parameters in neural search query to support ef_search #787

Adds method_parameters in neural search query to support ef_search #787

Conversation

shatejas commented Jun 11, 2024

Description

Issues Resolved

Check List

martin-gaievski left a comment

Choose a reason for hiding this comment

shatejas commented Jun 12, 2024

navneet1v Jun 13, 2024

Choose a reason for hiding this comment

shatejas Jun 13, 2024

Choose a reason for hiding this comment

navneet1v Jun 13, 2024

Choose a reason for hiding this comment

shatejas Jun 14, 2024

Choose a reason for hiding this comment

martin-gaievski left a comment

Choose a reason for hiding this comment

martin-gaievski commented Jun 14, 2024

shatejas commented Jun 14, 2024

navneet1v commented Jun 17, 2024

vibrantvarun Jun 17, 2024

Choose a reason for hiding this comment

shatejas Jun 17, 2024

Choose a reason for hiding this comment

vibrantvarun commented Jun 17, 2024

shatejas commented Jun 17, 2024