Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DerivedField: PR-2 Implementation for all supported types and DerivedField… #83

Closed
wants to merge 9 commits into from

Conversation

rishabhmaurya
Copy link
Owner

@rishabhmaurya rishabhmaurya commented Mar 20, 2024

Description

Derived Field PR2 (against PR1)-
PR1 - opensearch-project#12746
Implementation for all supported types and DerivedFieldType. We support following types -

boolean
date
geo_point
ip
keyword
long
double

Related Issues

Resolves opensearch-project#12508

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)
  • Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

peternied and others added 9 commits March 22, 2024 21:19
* Make GetTermVersionAction internal

GetTermVersion action has been added to check if cluster state needs to
be updated as a performance improvement on all cluster state checks.
The Authorization systems in the Security Plugin check all actions that
do not start with `internal:` for permissions causing users without
`cluster:monitor/*` permissions to start getting 403 exceptions.

This change signals that this action does not require an authorization
check by any security systems since it cannot be called via REST APIs.

Signed-off-by: Peter Nied <[email protected]>

* Move TermVerson action outside of admin/cluster namespace

Signed-off-by: Peter Nied <[email protected]>

---------

Signed-off-by: Peter Nied <[email protected]>
…project#12876)

* Change from configureEach to all to resolve incompatibility

Signed-off-by: Zelin Hao <[email protected]>

* Update Changelog

Signed-off-by: Zelin Hao <[email protected]>

---------

Signed-off-by: Zelin Hao <[email protected]>
…oject#12877)

With this plugin enabled we see significant increase in memory usage.
This is due to a static map that collects Span entries per write/replication request.
Tests with higher counts of independent writes are at risk of running oom or experiencing
slow down / warnings related to this.

Disabled the plugin by default until this is resolved given it is still under FeatureFlag.

Signed-off-by: Marc Handalian <[email protected]>
…rch-project#12746)

First in a series of commits to support derived fields, a form of schema-on-read.

This commit adds:
1. DerivedFieldScript factory: This script factory will be used to execute scripts defined against 
   derived fields of any type.
2. DerivedFieldValueFetcher: The value fetcher contains logic to execute script and fetch the value 
   in form of List<Object>. It expects DerivedFieldScript.LeafFactory as an input and sets the contract 
   with consumer to call setNextReader() whenever a segment is switched.
3. DerivedFieldQuery: This query will be used by any of the derived fields. It expects an input query and
   DerivedFieldValueFetcher. It uses 2-phase iterator approach with approximation iterator set to match 
   all docs. On a match, it creates a lucene MemoryIndex for a given doc, fetches the value of the derived
   field from _source using DerivedFieldValueFetcher and executes the input query against.

---------

Signed-off-by: Rishabh Maurya <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants