[Tiered Caching] Caching tier policies

[peteralfonsi]

Description

This PR adds caching tier policies. In a tiered caching setup, we might not want to allow all values into all tiers, so we want to define policies to allow or reject values. For example, we might not want to add values that took a very short time to compute to a disk tier, since the time to recompute them could be shorter than the time to get them back from the disk.

This PR adds an interface for policies and integrates them to restrict access to the disk tier in the TieredSpilloverCache. Currently the only implementation is the took time policy, but it should be easy to add more as they're needed.

When QuerySearchResult values are written into the IndicesRequestCache, we first write a CachePolicyInfoWrapper object, which has any values we might want to provide to policies. This way policies don't have to read the entire QSR. This wrapper is discarded when values come out of the cache.

There is also a new cluster setting to set the threshold for this policy, but right now it has no effect since the TieredSpilloverCache isn't integrated into the IndicesRequestCache yet.

Related Issues

This is part of the tiered caching feature (#10024).

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Compatibility status:

Checks if related components are compatible with change 24448fc

Incompatible components

Skipped components

Compatible components

[github-actions]

❕ Gradle check result for 4c78763: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing
      1 org.opensearch.indices.replication.SegmentReplicationIT.classMethod

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Attention: Patch coverage is 91.83673% with 4 lines in your changes are missing coverage. Please review.

Project coverage is 71.36%. Comparing base (5c82ab8) to head (f907e1b).
Report is 161 commits behind head on main.

❗ Current head f907e1b differs from pull request most recent head 24448fc. Consider uploading reports for the commit 24448fc to get more accurate results

Files	Patch %	Lines
...arch/common/cache/tier/DiskTierTookTimePolicy.java	90.00%	2 Missing ⚠️
...search/common/cache/tier/TieredSpilloverCache.java	87.50%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #11863      +/-   ##
============================================
- Coverage     71.43%   71.36%   -0.08%     
+ Complexity    59407    59296     -111     
============================================
  Files          4921     4921              
  Lines        278989   278956      -33     
  Branches      40543    40548       +5     
============================================
- Hits         199287   199063     -224     
- Misses        63086    63275     +189     
- Partials      16616    16618       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ansjcy]

Do we need to do some benchmarking tests to better understand what should be the default value?

[peteralfonsi]

Yes - this is one of the planned benchmarking scenarios for tiered caching. We plan to have a range of queries that take different lengths of time, and see how short that time can get before sending queries to the disk tier stops providing any benefit. The current value is a placeholder. The benchmarking scenarios are on hold right now because of changes being made to some of Sagar's tiered caching PRs.

[github-actions]

❌ Gradle check result for 24448fc: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?