update rollover to leverage write-alias semantics #32216

talevy · 2018-07-19T22:14:44Z

Updates Rollover to leverage the ability to preserve aliases and swap which is the write index.

Historically, Rollover would swap which index had the designated alias for writing documents against. This required users to keep a separate read-alias that enabled reading against both rolled over and newly created indices, whiles the write-alias was being re-assigned at every rollover.

With the ability for aliases to designate a write index, Rollover can be a bit more flexible with its use of aliases.

Updates include:

Rollover validates that the target alias has a write index (the index that is being rolled over). This means that the restriction that aliases only point to one index is no longer necessary.
Rollover explicitly (and atomically) swaps which index is the write-index by explicitly assigning the existing index to have is_write_index: false and have the newly created index have its rollover alias as is_write_index: true. This is only done when is_write_index: true on the write index. Default behavior of removing the alias from the rolled over index stays when is_write_index is not explicitly set

Relevant things that are staying the same:

Rollover is rejected if there exist any templates that match the newly-created index and configure the rollover-alias
- I think this existed to prevent the situation where an alias pointed to two indices for a short while. Although this can technically be relaxed, the specific cases that are safe are really particular and difficult to reason, so leaving the broad restriction sounds good

elasticmachine · 2018-07-19T22:14:45Z

Pinging @elastic/es-core-infra

Rollover should not swap aliases when `is_write_index` is set to `true`. Instead, both the new and old indices should have the rollover alias, with the newly created index as the new write index

dakrone

This LGTM, I left one comment that should be resolved but no need for another review

dakrone · 2018-07-27T17:28:25Z

server/src/test/java/org/elasticsearch/action/admin/indices/rollover/RolloverIT.java

+        Alias testAlias = new Alias("test_alias");
+        boolean explicitWriteIndex = randomBoolean();
+        if (explicitWriteIndex) {
+            testAlias.writeIndex(true);


can you change these to be randomBoolean() rather than always being set to true (if set at all) so we make sure that explicit false settings have the correct behavior?

Instead, I will add another test method that verifies behavior when there is no write index, similar to testValidation in TransportRolloverActionTests

to clarify, I don't thinkrandomBoolean() fits here because making it false would result in no rollover done. I've added a dedicated test to verify the integration between validation and the transport action. For when it is a dry-run, and for real.

Rollover should not swap aliases when `is_write_index` is set to `true`. Instead, both the new and old indices should have the rollover alias, with the newly created index as the new write index Updates Rollover to leverage the ability to preserve aliases and swap which is the write index. Historically, Rollover would swap which index had the designated alias for writing documents against. This required users to keep a separate read-alias that enabled reading against both rolled over and newly created indices, whiles the write-alias was being re-assigned at every rollover. With the ability for aliases to designate a write index, Rollover can be a bit more flexible with its use of aliases. Updates include: - Rollover validates that the target alias has a write index (the index that is being rolled over). This means that the restriction that aliases only point to one index is no longer necessary. - Rollover explicitly (and atomically) swaps which index is the write-index by explicitly assigning the existing index to have `is_write_index: false` and have the newly created index have its rollover alias as `is_write_index: true`. This is only done when `is_write_index: true` on the write index. Default behavior of removing the alias from the rolled over index stays when `is_write_index` is not explicitly set Relevant things that are staying the same: - Rollover is rejected if there exist any templates that match the newly-created index and configure the rollover-alias - I think this existed to prevent the situation where an alias pointed to two indices for a short while. Although this can technically be relaxed, the specific cases that are safe are really particular and difficult to reason, so leaving the broad restriction sounds good

* master: Logging: Make node name consistent in logger (#31588) Mute SSLTrustRestrictionsTests on JDK 11 Increase max chunk size to 256Mb for repo-azure (#32101) Docs: Fix README upgrade mention (#32313) Changed ReindexRequest to use Writeable.Reader (#32401) Mute KerberosAuthenticationIT Fix AutoIntervalDateHistogram.testReduce random failures (#32301) fix no=>not typo (#32463) Mute QueryProfilerIT#testProfileMatchesRegular() HLRC: Add delete watch action (#32337) High-level client: fix clusterAlias parsing in SearchHit (#32465) Fix calculation of orientation of polygons (#27967) [Kerberos] Add missing javadocs (#32469) [Kerberos] Remove Kerberos bootstrap checks (#32451) Make get all app privs requires "*" permission (#32460) Switch security to new style Requests (#32290) Switch security spi example to new style Requests (#32341) Painless: Add PainlessConstructor (#32447) update rollover to leverage write-alias semantics (#32216) Update Fuzzy Query docs to clarify default behavior re max_expansions (#30819) INGEST: Clean up Java8 Stream Usage (#32059) Ensure KeyStoreWrapper decryption exceptions are handled (#32464)

@throws

* 6.x: Fix scriptdocvalues tests with dates Correct minor typo in explain.asciidoc for HLRC Fix painless whitelist and warnings from backporting #31441 Build: Add elastic maven to repos used by BuildPlugin (#32549) Scripting: Conditionally use java time api in scripting (#31441) [ML] Improve error when no available field exists for rule scope (#32550) [ML] Improve error for functions with limited rule condition support (#32548) [ML] Remove multiple_bucket_spans [ML] Fix thread leak when waiting for job flush (#32196) (#32541) Painless: Clean Up PainlessField (#32525) Add @AwaitsFix for #32554 Remove broken @link in Javadoc Add AwaitsFix to failing test - see #32546 SQL: Added support for string manipulating functions with more than one parameter (#32356) [DOCS] Reloadable Secure Settings (#31713) Fix compilation error introduced by #32339 [Rollup] Remove builders from TermsGroupConfig (#32507) Use hostname instead of IP with SPNEGO test (#32514) Switch x-pack rolling restart to new style Requests (#32339) [DOCS] Small fixes in rule configuration page (#32516) Painless: Clean up PainlessMethod (#32476) SQL: Add test for handling of partial results (#32474) Docs: Add missing migration doc for logging change Build: Remove shadowing from benchmarks (#32475) Docs: Add all JDKs to CONTRIBUTING.md Logging: Make node name consistent in logger (#31588) High-level client: fix clusterAlias parsing in SearchHit (#32465) REST high-level client: parse back _ignored meta field (#32362) backport fix of reduceRandom fix (#32508) Add licensing enforcement for FIPS mode (#32437) INGEST: Clean up Java8 Stream Usage (#32059) (#32485) Improve the error message when an index is incompatible with field aliases. (#32482) Mute testFilterCacheStats Scripting: Fix painless compiler loader to know about context classes (#32385) [ML][DOCS] Fix typo applied_to => applies_to Mute SSLTrustRestrictionsTests on JDK 11 Changed ReindexRequest to use Writeable.Reader (#32401) Increase max chunk size to 256Mb for repo-azure (#32101) Mute KerberosAuthenticationIT fix no=>not typo (#32463) HLRC: Add delete watch action (#32337) Fix calculation of orientation of polygons (#27967) [Kerberos] Add missing javadocs (#32469) Fix missing JavaDoc for @throws in several places in KerberosTicketValidator. Make get all app privs requires "*" permission (#32460) Ensure KeyStoreWrapper decryption exceptions are handled (#32472) update rollover to leverage write-alias semantics (#32216) [Kerberos] Remove Kerberos bootstrap checks (#32451) Switch security to new style Requests (#32290) Switch security spi example to new style Requests (#32341) Painless: Add PainlessConstructor (#32447) Update Fuzzy Query docs to clarify default behavior re max_expansions (#30819) Remove > from Javadoc (fatal with Java 11) Tests: Fix convert error tests to use fixed value (#32415) IndicesClusterStateService should replace an init. replica with an init. primary with the same aId (#32374) auto-interval date histogram - 6.x backport (#32107) [CI] Mute DocumentSubsetReaderTests testSearch [TEST] Mute failing InternalEngineTests#testSeqNoAndCheckpoints TEST: testDocStats should always use forceMerge (#32450) TEST: Avoid deletion in FlushIT AwaitsFix IndexShardTests#testDocStats Painless: Add method type to method. (#32441) Remove reference to non-existent store type (#32418) [TEST] Mute failing FlushIT test Fix ordering of bootstrap checks in docs (#32417) Wrong discovery.type for azure in breaking changes (#32432) Mute ConvertProcessorTests failing tests TESTS: Move netty leak detection to paranoid level (#32354) (#32425) Upgrade to Lucene-7.5.0-snapshot-608f0277b0 (#32390) [Kerberos] Avoid vagrant update on precommit (#32416) TEST: Avoid triggering merges in FlushIT [DOCS] Fixes formatting of scope object in job resource Switch x-pack/plugin to new style Requests (#32327) Release requests in cors handle (#32410) Remove BouncyCastle dependency from runtime (#32402) Copy missing segment attributes in getSegmentInfo (#32396) Rest HL client: Add put license action (#32214) Docs: Correcting a typo in tophits (#32359) Build: Stop double generating buildSrc pom (#32408) Switch x-pack full restart to new style Requests (#32294) Painless: Clean Up PainlessClass Variables (#32380) [ML] Consistent pattern for strict/lenient parser names (#32399) Add Restore Snapshot High Level REST API Update update-settings.asciidoc (#31378) Introduce index store plugins (#32375) Rank-Eval: Reduce scope of an unchecked supression Make sure _forcemerge respects `max_num_segments`. (#32291)

talevy added WIP :Data Management/Indices APIs APIs to create and manage indices and templates labels Jul 19, 2018

talevy mentioned this pull request Jul 19, 2018

[meta] Alias is_write_index feature tracking #31959

Closed

5 tasks

talevy force-pushed the write-index-alias-rollover branch from d247db0 to 53fcdb7 Compare July 19, 2018 22:49

talevy added WIP and removed WIP labels Jul 19, 2018

talevy force-pushed the write-index-alias-rollover branch 3 times, most recently from 53af7f1 to 080cb91 Compare July 26, 2018 23:27

talevy force-pushed the write-index-alias-rollover branch from 080cb91 to b3cfe24 Compare July 26, 2018 23:42

talevy removed the WIP label Jul 26, 2018

talevy force-pushed the write-index-alias-rollover branch from b3cfe24 to b03af7d Compare July 26, 2018 23:49

update rollover to use write indices when set

fa078f8

Rollover should not swap aliases when `is_write_index` is set to `true`. Instead, both the new and old indices should have the rollover alias, with the newly created index as the new write index

talevy force-pushed the write-index-alias-rollover branch from b03af7d to fa078f8 Compare July 27, 2018 00:33

talevy requested a review from jasontedor July 27, 2018 00:33

talevy added the review label Jul 27, 2018

talevy requested review from dakrone and removed request for jasontedor July 27, 2018 15:06

talevy added the >enhancement label Jul 27, 2018

dakrone approved these changes Jul 27, 2018

View reviewed changes

talevy added 3 commits July 27, 2018 14:03

add IT test for rolling over with no write index

e964487

Merge branch 'master' into write-index-alias-rollover

e5cf20d

Merge branch 'master' into write-index-alias-rollover

cf9d509

talevy removed the review label Jul 30, 2018

talevy merged commit 1e0fceb into elastic:master Jul 30, 2018

talevy deleted the write-index-alias-rollover branch July 30, 2018 21:32

talevy added the v7.0.0 label Jul 30, 2018

talevy added the v6.5.0 label Jul 31, 2018

colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update rollover to leverage write-alias semantics #32216

update rollover to leverage write-alias semantics #32216

talevy commented Jul 19, 2018 •

edited

Loading

elasticmachine commented Jul 19, 2018

dakrone left a comment

dakrone Jul 27, 2018

talevy Jul 27, 2018

talevy Jul 27, 2018 •

edited

Loading

update rollover to leverage write-alias semantics #32216

update rollover to leverage write-alias semantics #32216

Conversation

talevy commented Jul 19, 2018 • edited Loading

elasticmachine commented Jul 19, 2018

dakrone left a comment

Choose a reason for hiding this comment

dakrone Jul 27, 2018

Choose a reason for hiding this comment

talevy Jul 27, 2018

Choose a reason for hiding this comment

talevy Jul 27, 2018 • edited Loading

Choose a reason for hiding this comment

talevy commented Jul 19, 2018 •

edited

Loading

talevy Jul 27, 2018 •

edited

Loading