SHS-NG M4.7: Remove JobProgressListener. #13
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
To avoid having the writes to the status store stall the listener bus thread, this change adds a caching layer that also does writes asynchronously. The write layer avoids high memory usage by de-duplicating writes; this is done by "overwriting" previous write operations to the same key that haven't yet been written to the underlying store. The read path is kept up to date with these operations, so that the listener that is doing the writes gets a consistent view of the store. Note the caching only works for single reads (i.e. not iteration). Because of that, the status listener needs to keep track of some extra information in memory so that it doesn't need to rely on data being written to the underlying store for handling certain events. On top of those, there's a lot of work in this change to keep track of certain metrics that can help us measure how well the cache is working and whether it needs some tuning. Tested with pretty large log files with lots of tasks and verified that the write queue doesn't grow too much.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The only remaining use of this class was the SparkStatusTracker, which
was modified to use the new state store. The test code to wait for
executors was moved to TestUtils and now uses the SparkStatusTracker API.
As part of this change I also modified the streaming UI to read the needed
data from the store, which was missed in the previous patch that made
JobProgressListener redundant.