LUCENE-10216: Use MergeScheduler and MergePolicy to run addIndexes(CodecReader[]) merges. #1051
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…decReader[]) merges. (apache#633) * Use merge policy and merge scheduler to run addIndexes merges * wrapped reader does not see deletes - debug * Partially fixed tests in TestAddIndexes * Use writer object to invoke addIndexes merge * Use merge object info * Add javadocs for new methods * TestAddIndexes passing * verify field info schemas upfront from incoming readers * rename flag to track pooled readers * Keep addIndexes API transactional * Maintain transactionality - register segments with iw after all merges complete * fix checkstyle * PR comments * Fix pendingDocs - numDocs mismatch bug * Tests with 1-1 merges and partial merge failures * variable renaming and better comments * add test for partial merge failures. change tests to use 1-1 findmerges * abort pending merges gracefully * test null and empty merge specs * test interim files are deleted * test with empty readers * test cascading merges triggered * remove nocommits * gradle check errors * remove unused line * remove printf * spotless apply * update TestIndexWriterOnDiskFull to accept mergeException from failing addIndexes calls * return singleton reader mergespec in NoMergePolicy * rethrow exceptions seen in merge threads on failure * spotless apply * update test to new exception type thrown * spotlessApply * test for maxDoc limit in IndexWriter * spotlessApply * Use DocValuesIterator instead of DocValuesFieldExistsQuery for counting soft deletes * spotless apply * change exception message for closed IW * remove non-essential comments * update api doc string * doc string update * spotless * Changes file entry * simplify findMerges API, add 1-1 merges to MockRandomMergePolicy * update merge policies to new api * remove unused imports * spotless apply * move changes entry to end of list * fix testAddIndicesWithSoftDeletes * test with 1-1 merge policy always enabled * please spotcheck * tidy * test - never use 1-1 merge policy * use 1-1 merge policy randomly * Remove concurrent addIndexes findMerges from MockRandomMergePolicy * Bug Fix: RuntimeException in addIndexes Aborted pending merges were slipping through the merge exception check in API, and getting caught later in the RuntimeException check. * tidy * Rebase on main. Move changes to 10.0 * Synchronize IW.AddIndexesMergeSource on outer class IW object * tidy
…tions (apache#1012) * Fix failures in TestAssertingPointsFormat.testWithExceptions * remove redundant finally block * tidy * remove TODO as it is done now
mikemccand
approved these changes
Jul 28, 2022
|
Wooot! |
This was referenced Aug 24, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport the following changes to branch 9x -
Cherry picked commits:
698f40adandca791747.