Star Tree File Formats

[sarthakaggarwal97]

Description

Introducing the file formats for composite index. There could be multiple implementations of composite index, star-tree being one of them.

With this change, we introduce four new files.

1. cid - Composite Index Data
2. cim - Composite Index Meta
3. cidvd - Composite Doc Values Data
4. cidvm - Composite Doc Values Meta

Related Issues

Resolves #14837

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for b93a0ce: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 94d1928: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 26337a8: SUCCESS

[github-actions]

✅ Gradle check result for 1b5009e: SUCCESS

[github-actions]

❌ Gradle check result for 1fc9add:

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 72ef06f: SUCCESS

[github-actions]

❌ Gradle check result for 9250579: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for cb32164: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[mgodwan]

Can this PR be broken down?
One way I see is to use separate PRs for format write, format read, and integration with build flow. Otherwise it may take time to review this in a single go.

[github-actions]

✅ Gradle check result for 7ded5ca: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 89.72746% with 49 lines in your changes missing coverage. Please review.

Project coverage is 72.07%. Comparing base (758c2aa) to head (e13e400).
Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
...posite/composite99/Composite99DocValuesReader.java	77.10%	11 Missing and 8 partials ⚠️
...eindex/datacube/startree/index/StarTreeValues.java	81.53%	7 Missing and 5 partials ⚠️
...datacube/startree/builder/BaseStarTreeBuilder.java	96.12%	3 Missing and 3 partials ⚠️
...index/datacube/startree/node/InMemoryTreeNode.java	88.46%	0 Missing and 6 partials ⚠️
...posite/composite99/Composite99DocValuesWriter.java	93.47%	1 Missing and 2 partials ⚠️
...tacube/startree/builder/OnHeapStarTreeBuilder.java	92.30%	1 Missing ⚠️
.../startree/fileformats/data/StarTreeDataWriter.java	93.33%	0 Missing and 1 partial ⚠️
...rtree/fileformats/meta/StarTreeMetadataWriter.java	83.33%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #14809      +/-   ##
============================================
+ Coverage     72.02%   72.07%   +0.04%     
- Complexity    63769    63904     +135     
============================================
  Files          5249     5249              
  Lines        297795   298087     +292     
  Branches      43034    43071      +37     
============================================
+ Hits         214480   214838     +358     
+ Misses        65735    65633     -102     
- Partials      17580    17616      +36     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[bharath-techie]

Thanks for the changes @sarthakaggarwal97 , this is a really important and tricky PR , thanks for addressing all the comments, adding extensive tests and refactoring the code.

LGTM.

[mgodwan]

Overall changes LGTM. Thanks for working through the changes.
Left some minor comments to be addressed

[github-actions]

❕ Gradle check result for e13e400: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.