[Rollup][Test] Force a stop to save rollup state before continuing #32787
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We only upgrade the ID when the state is saved in one of four scenarios: - when we reach a checkpoint (every 50 pages) - when we run out of data - when explicitly stopped - on failure The test was relying on the pre-upgrade to finish, save state and then the post-upgrade to start, hit the end of data and upgrade ID. THEN get the new doc and apply the new ID. But I think this is vulnerable to timing issues. If the pre-upgrade portion shutdown before it saved the state, when restarting we would run through all the data from the beginning with the old ID, meaning both docs would still have the old scheme. This change makes the pre-upgrade wait for the job to go back to STARTED so that we know it persisted the end point. Post-upgrade, it stops and restarts the job to ensure the state was persisted and the ID upgraded. That _should_ rule out the above timing issue. Closes elastic#32773
polyfractal
added
>test
jimczi
approved these changes
Aug 13, 2018
polyfractal
added a commit
that referenced
this pull request
Aug 13, 2018
We only upgrade the ID when the state is saved in one of four scenarios: - when we reach a checkpoint (every 50 pages) - when we run out of data - when explicitly stopped - on failure The test was relying on the pre-upgrade to finish, save state and then the post-upgrade to start, hit the end of data and upgrade ID. THEN get the new doc and apply the new ID. But I think this is vulnerable to timing issues. If the pre-upgrade portion shutdown before it saved the state, when restarting we would run through all the data from the beginning with the old ID, meaning both docs would still have the old scheme. This change makes the pre-upgrade wait for the job to go back to STARTED so that we know it persisted the end point. Post-upgrade, it stops and restarts the job to ensure the state was persisted and the ID upgraded. That _should_ rule out the above timing issue. Closes #32773
polyfractal
added a commit
that referenced
this pull request
Aug 13, 2018
We only upgrade the ID when the state is saved in one of four scenarios: - when we reach a checkpoint (every 50 pages) - when we run out of data - when explicitly stopped - on failure The test was relying on the pre-upgrade to finish, save state and then the post-upgrade to start, hit the end of data and upgrade ID. THEN get the new doc and apply the new ID. But I think this is vulnerable to timing issues. If the pre-upgrade portion shutdown before it saved the state, when restarting we would run through all the data from the beginning with the old ID, meaning both docs would still have the old scheme. This change makes the pre-upgrade wait for the job to go back to STARTED so that we know it persisted the end point. Post-upgrade, it stops and restarts the job to ensure the state was persisted and the ID upgraded. That _should_ rule out the above timing issue. Closes #32773
jasontedor
added a commit
to jasontedor/elasticsearch
that referenced
this pull request
Aug 14, 2018
…e-types * elastic/master: (199 commits) Watcher: Remove unused hipchat render method (elastic#32211) Watcher: Remove extraneous auth classes (elastic#32300) Watcher: migrate PagerDuty v1 events API to v2 API (elastic#32285) [TEST] Select free port for Minio (elastic#32837) MINOR: Remove `IndexTemplateFilter` (elastic#32841) Core: Add java time version of rounding classes (elastic#32641) Aggregations/HL Rest client fix: missing scores (elastic#32774) HLRC: Add Delete License API (elastic#32586) INGEST: Create Index Before Pipeline Execute (elastic#32786) Fix NOOP bulk updates (elastic#32819) Remove client connections from TcpTransport (elastic#31886) Increase logging testRetentionPolicyChangeDuringRecovery AwaitsFix case-functions.sql-spec Mute security-cli tests in FIPS JVM (elastic#32812) SCRIPTING: Support BucketAggScript return null (elastic#32811) Unmute WildFly tests in FIPS JVM (elastic#32814) [TEST] Force a stop to save rollup state before continuing (elastic#32787) [test] disable packaging tests for suse boxes Mute IndicesRequestIT#testBulk [ML][DOCS] Refer to rules feature as custom rules (elastic#32785) ...
jasontedor
added a commit
to jasontedor/elasticsearch
that referenced
this pull request
Aug 14, 2018
…listeners * elastic/master: Watcher: Remove unused hipchat render method (elastic#32211) Watcher: Remove extraneous auth classes (elastic#32300) Watcher: migrate PagerDuty v1 events API to v2 API (elastic#32285) [TEST] Select free port for Minio (elastic#32837) MINOR: Remove `IndexTemplateFilter` (elastic#32841) Core: Add java time version of rounding classes (elastic#32641) Aggregations/HL Rest client fix: missing scores (elastic#32774) HLRC: Add Delete License API (elastic#32586) INGEST: Create Index Before Pipeline Execute (elastic#32786) Fix NOOP bulk updates (elastic#32819) Remove client connections from TcpTransport (elastic#31886) Increase logging testRetentionPolicyChangeDuringRecovery AwaitsFix case-functions.sql-spec Mute security-cli tests in FIPS JVM (elastic#32812) SCRIPTING: Support BucketAggScript return null (elastic#32811) Unmute WildFly tests in FIPS JVM (elastic#32814) [TEST] Force a stop to save rollup state before continuing (elastic#32787) [test] disable packaging tests for suse boxes Mute IndicesRequestIT#testBulk [ML][DOCS] Refer to rules feature as custom rules (elastic#32785)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We only upgrade the ID when the state is saved in one of four scenarios:
The test was relying on the pre-upgrade to finish, save state and then the post-upgrade to start, hit the end of data and upgrade ID. THEN get the new doc and apply the new ID.
But I think this is vulnerable to timing issues. If the pre-upgrade portion shutdown before it saved the state, when restarting we would run through all the data from the beginning with the old ID, meaning both docs would still have the old scheme.
This change makes the pre-upgrade wait for the job to go back to STARTED so that we know it persisted the end point. Post-upgrade, it stops and restarts the job to ensure the state was persisted and the ID upgraded.
That should rule out the above timing issue. I think. :) @jimczi does this all look reasonable?
Closes #32773