LUCENE-10216: Use MergeScheduler and MergePolicy to run addIndexes(Co…

…decReader[]) merges. (#633) * Use merge policy and merge scheduler to run addIndexes merges * wrapped reader does not see deletes - debug * Partially fixed tests in TestAddIndexes * Use writer object to invoke addIndexes merge * Use merge object info * Add javadocs for new methods * TestAddIndexes passing * verify field info schemas upfront from incoming readers * rename flag to track pooled readers * Keep addIndexes API transactional * Maintain transactionality - register segments with iw after all merges complete * fix checkstyle * PR comments * Fix pendingDocs - numDocs mismatch bug * Tests with 1-1 merges and partial merge failures * variable renaming and better comments * add test for partial merge failures. change tests to use 1-1 findmerges * abort pending merges gracefully * test null and empty merge specs * test interim files are deleted * test with empty readers * test cascading merges triggered * remove nocommits * gradle check errors * remove unused line * remove printf * spotless apply * update TestIndexWriterOnDiskFull to accept mergeException from failing addIndexes calls * return singleton reader mergespec in NoMergePolicy * rethrow exceptions seen in merge threads on failure * spotless apply * update test to new exception type thrown * spotlessApply * test for maxDoc limit in IndexWriter * spotlessApply * Use DocValuesIterator instead of DocValuesFieldExistsQuery for counting soft deletes * spotless apply * change exception message for closed IW * remove non-essential comments * update api doc string * doc string update * spotless * Changes file entry * simplify findMerges API, add 1-1 merges to MockRandomMergePolicy * update merge policies to new api * remove unused imports * spotless apply * move changes entry to end of list * fix testAddIndicesWithSoftDeletes * test with 1-1 merge policy always enabled * please spotcheck * tidy * test - never use 1-1 merge policy * use 1-1 merge policy randomly * Remove concurrent addIndexes findMerges from MockRandomMergePolicy * Bug Fix: RuntimeException in addIndexes Aborted pending merges were slipping through the merge exception check in API, and getting caught later in the RuntimeException check. * tidy * Rebase on main. Move changes to 10.0 * Synchronize IW.AddIndexesMergeSource on outer class IW object * tidy
apache · Jul 6, 2022 · 698f40a · 698f40a
1 parent d537013
commit 698f40a
Show file tree

Hide file tree

Showing 9 changed files with 706 additions and 136 deletions.
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
@@ -45,6 +45,9 @@ Improvements
 * LUCENE-10416: Update Korean Dictionary to mecab-ko-dic-2.1.1-20180720 for Nori.
   (Uihyun Kim)
 
+* LUCENE-10216: Use MergePolicy to define and MergeScheduler to trigger the reader merges
+  required by addIndexes(CodecReader[]) API. (Vigya Sharma, Michael McCandless)
+
 Optimizations
 ---------------------
 (No changes)

diff --git a/lucene/core/src/java/org/apache/lucene/index/FieldInfos.java b/lucene/core/src/java/org/apache/lucene/index/FieldInfos.java
@@ -352,6 +352,14 @@ static final class FieldNumbers {
       this.softDeletesFieldName = softDeletesFieldName;
     }
 
+    void verifyFieldInfo(FieldInfo fi) {
+      String fieldName = fi.getName();
+      verifySoftDeletedFieldName(fieldName, fi.isSoftDeletesField());
+      if (nameToNumber.containsKey(fieldName)) {
+        verifySameSchema(fi);
+      }
+    }
+
     /**
      * Returns the global field number for the given field name. If the name does not exist yet it
      * tries to add it with the given preferred field number assigned if possible otherwise the

diff --git a/lucene/core/src/java/org/apache/lucene/index/FilterMergePolicy.java b/lucene/core/src/java/org/apache/lucene/index/FilterMergePolicy.java
@@ -47,6 +47,11 @@ public MergeSpecification findMerges(
     return in.findMerges(mergeTrigger, segmentInfos, mergeContext);
   }
 
+  @Override
+  public MergeSpecification findMerges(CodecReader... readers) throws IOException {
+    return in.findMerges(readers);
+  }
+
   @Override
   public MergeSpecification findForcedMerges(
       SegmentInfos segmentInfos,