Allow doc value search on unindexed boolean and date fields #11650

harshavamsi · 2023-12-19T18:22:06Z

Description

Fixes remaining field mappers that support doc_values based searching to make use of IndexOrDocValuesQuery where applicable.

Related Issues

Resolves #11205

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-12-19T18:33:55Z

❌ Gradle check result for ebeea24: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2023-12-19T18:42:44Z

Compatibility status:

Checks if related components are compatible with change 1914641

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer.git]

github-actions · 2023-12-19T19:11:23Z

❌ Gradle check result for fd22221: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2023-12-19T22:11:03Z

❌ Gradle check result for a843bf4: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-01-19T05:36:39Z

❌ Gradle check result for 52c5247: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-01-19T17:53:40Z

❌ Gradle check result for a71bc21: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

harshavamsi · 2024-04-30T15:09:05Z

@dblock @msfroh merge?

dblock · 2024-04-30T15:15:15Z

❕ Gradle check result for 7444538: UNSTABLE

TEST FAILURES:
      1 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testWriteBlobWithRetries
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Test Result (1 failure / -353)
org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testWriteBlobWithRetries

#7934

dblock · 2024-04-30T15:16:15Z

❌ Gradle check result for fc94f6c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Test Result (2 failures / +1)
org.opensearch.index.query.TermsQueryBuilderTests.testToQuery
org.opensearch.index.remote.RemoteSegmentTransferTrackerTests.testComputeTimeLagOnUpdate

@harshavamsi at least testToQuery looks new, care to review and open a bug please?

github-actions · 2024-04-30T15:48:20Z

❌ Gradle check result for fc94f6c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-04-30T15:56:58Z

❌ Gradle check result for 52747c7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-04-30T16:18:06Z

❌ Gradle check result for 65c84c2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-04-30T16:45:39Z

❌ Gradle check result for f690458: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-04-30T16:59:58Z

❌ Gradle check result for 65c84c2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

harshavamsi · 2024-04-30T22:55:22Z

@dblock looks like all these are flaky tests

dblock · 2024-05-01T12:08:12Z

@dblock looks like all these are flaky tests

Can you please look at the logs and link the failed tests?

github-actions · 2024-05-01T12:59:25Z

❌ Gradle check result for 65c84c2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

harshavamsi · 2024-05-01T17:34:49Z

@dblock looks like all these are flaky tests

Can you please look at the logs and link the failed tests?

Opened #13487, #13480, and #13479

Signed-off-by: Harsha Vamsi Kalluri <[email protected]>

VachaShah

LGTM! @harshavamsi Does this need any documentation updates?

harshavamsi · 2024-05-02T19:12:00Z

LGTM! @harshavamsi Does this need any documentation updates?

@VachaShah I fixed the documentation in an earlier effort here -- opensearch-project/documentation-website#5619

github-actions · 2024-05-02T20:00:51Z

✅ Gradle check result for 921dee5: SUCCESS

* Making fields searchable Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Adding tests for boolean fields Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Added tests for date fields Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Updating termsQuery logic Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Spotless Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Updating terms test Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Spotless Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Ensure that the points are intersecting for doc_values Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Spotless Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Adding missing javadocs Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Adding more tests for indexedValueForSearch Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Adding MatchAllDocsQuery to asserts Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Fix changelog Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Simplify loop criteria + remove IndexOrDocValuesQuery Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Add some comments Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Fix indendation Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Fixing boolean field tests Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Revert to correct logic for termsQuery Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Update terms logic to be more succinct Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Fix terms test Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Adding tests for boolean range query + fix range query to use term query inside Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Checking if upper and lower terms are valid Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Checking if upper and lower terms are valid Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Fixing changelog Signed-off-by: Harsha Vamsi Kalluri <[email protected]> --------- Signed-off-by: Harsha Vamsi Kalluri <[email protected]> (cherry picked from commit dc77c46) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

msfroh · 2024-05-02T21:43:09Z

@harshavamsi, #13480 is almost certainly a result of this change.

The TermsQueryBuilder expects to get one of a few query types, but because of this change, it's able to get a TermQuery instead.

…13514) * Making fields searchable * Adding tests for boolean fields * Added tests for date fields * Updating termsQuery logic * Spotless * Updating terms test * Spotless * Ensure that the points are intersecting for doc_values * Spotless * Adding missing javadocs * Adding more tests for indexedValueForSearch * Adding MatchAllDocsQuery to asserts * Fix changelog * Simplify loop criteria + remove IndexOrDocValuesQuery * Add some comments * Fix indendation * Fixing boolean field tests * Revert to correct logic for termsQuery * Update terms logic to be more succinct * Fix terms test * Adding tests for boolean range query + fix range query to use term query inside * Checking if upper and lower terms are valid * Checking if upper and lower terms are valid * Fixing changelog --------- (cherry picked from commit dc77c46) Signed-off-by: Harsha Vamsi Kalluri <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ch-project#11650) * Making fields searchable Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Adding tests for boolean fields Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Added tests for date fields Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Updating termsQuery logic Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Spotless Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Updating terms test Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Spotless Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Ensure that the points are intersecting for doc_values Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Spotless Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Adding missing javadocs Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Adding more tests for indexedValueForSearch Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Adding MatchAllDocsQuery to asserts Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Fix changelog Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Simplify loop criteria + remove IndexOrDocValuesQuery Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Add some comments Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Fix indendation Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Fixing boolean field tests Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Revert to correct logic for termsQuery Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Update terms logic to be more succinct Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Fix terms test Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Adding tests for boolean range query + fix range query to use term query inside Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Checking if upper and lower terms are valid Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Checking if upper and lower terms are valid Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Fixing changelog Signed-off-by: Harsha Vamsi Kalluri <[email protected]> --------- Signed-off-by: Harsha Vamsi Kalluri <[email protected]>

github-actions bot added enhancement Enhancement or improvement to existing feature or request Search:Performance v2.12.0 Issues and PRs related to version 2.12.0 labels Dec 19, 2023

harshavamsi force-pushed the doc_values_searching branch from ebeea24 to fd22221 Compare December 19, 2023 18:42

harshavamsi force-pushed the doc_values_searching branch from a843bf4 to 52c5247 Compare January 19, 2024 05:21

harshavamsi force-pushed the doc_values_searching branch from 52c5247 to a71bc21 Compare January 19, 2024 17:37

harshavamsi marked this pull request as ready for review January 19, 2024 18:40

harshavamsi requested review from abbashus, adnapibar, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, dbwiddis, dreamer-89, gbbafna, kartg, kotwanikunal, mch2, msfroh, nknize, owaiskazi19 and reta as code owners January 19, 2024 18:40

Merge branch 'main' into doc_values_searching

65c84c2

harshavamsi force-pushed the doc_values_searching branch 2 times, most recently from 52747c7 to f690458 Compare April 30, 2024 15:55

harshavamsi force-pushed the doc_values_searching branch from f690458 to 65c84c2 Compare April 30, 2024 15:59

Merge branch 'main' into doc_values_searching

921dee5

Signed-off-by: Harsha Vamsi Kalluri <[email protected]>

VachaShah approved these changes May 2, 2024

View reviewed changes

VachaShah merged commit dc77c46 into opensearch-project:main May 2, 2024
27 of 28 checks passed

opensearch-trigger-bot bot mentioned this pull request May 2, 2024

[Backport 2.x] Allow doc value search on unindexed boolean and date fields #13514

Merged

msfroh mentioned this pull request May 2, 2024

[BUG] org.opensearch.index.query.TermsQueryBuilderTests.testToQuery is flaky #13480

Closed

BrewTestBot mentioned this pull request Jun 26, 2024

opensearch 2.15.0 Homebrew/homebrew-core#175637

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow doc value search on unindexed boolean and date fields #11650

Allow doc value search on unindexed boolean and date fields #11650

harshavamsi commented Dec 19, 2023 •

edited

Loading

github-actions bot commented Dec 19, 2023

github-actions bot commented Dec 19, 2023 •

edited

Loading

github-actions bot commented Dec 19, 2023

github-actions bot commented Dec 19, 2023

github-actions bot commented Jan 19, 2024

github-actions bot commented Jan 19, 2024

harshavamsi commented Apr 30, 2024

dblock commented Apr 30, 2024

dblock commented Apr 30, 2024

github-actions bot commented Apr 30, 2024

github-actions bot commented Apr 30, 2024

github-actions bot commented Apr 30, 2024

github-actions bot commented Apr 30, 2024

github-actions bot commented Apr 30, 2024

harshavamsi commented Apr 30, 2024

dblock commented May 1, 2024

github-actions bot commented May 1, 2024

harshavamsi commented May 1, 2024

VachaShah left a comment

harshavamsi commented May 2, 2024

github-actions bot commented May 2, 2024

msfroh commented May 2, 2024

Allow doc value search on unindexed boolean and date fields #11650

Allow doc value search on unindexed boolean and date fields #11650

Conversation

harshavamsi commented Dec 19, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Dec 19, 2023

github-actions bot commented Dec 19, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Dec 19, 2023

github-actions bot commented Dec 19, 2023

github-actions bot commented Jan 19, 2024

github-actions bot commented Jan 19, 2024

harshavamsi commented Apr 30, 2024

dblock commented Apr 30, 2024

dblock commented Apr 30, 2024

github-actions bot commented Apr 30, 2024

github-actions bot commented Apr 30, 2024

github-actions bot commented Apr 30, 2024

github-actions bot commented Apr 30, 2024

github-actions bot commented Apr 30, 2024

harshavamsi commented Apr 30, 2024

dblock commented May 1, 2024

github-actions bot commented May 1, 2024

harshavamsi commented May 1, 2024

VachaShah left a comment

Choose a reason for hiding this comment

harshavamsi commented May 2, 2024

github-actions bot commented May 2, 2024

msfroh commented May 2, 2024

harshavamsi commented Dec 19, 2023 •

edited

Loading

github-actions bot commented Dec 19, 2023 •

edited

Loading