fix: retry watcher failure causes infinite loop #8786
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for determined-ui canceled.
|
eecsliu
added
the
to-cherry-pick
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #8786 +/- ##
==========================================
- Coverage 47.69% 47.69% -0.01%
==========================================
Files 1049 1049
Lines 167230 167233 +3
Branches 2239 2241 +2
==========================================
- Hits 79766 79765 -1
- Misses 87306 87310 +4
Partials 158 158
Flags with carried forward coverage won't be shown. Click here to find out more.
|
stoksc
approved these changes
Feb 1, 2024
dai-release bot
pushed a commit
that referenced
this pull request
Feb 1, 2024
(cherry picked from commit e873381)
4 tasks
maxrussell
pushed a commit
that referenced
this pull request
Mar 21, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
When k8s retrywatcher fails, it causes us to receive infinite nil events and pins the CPU to 100%. Since a retrywatcher failure likely means something bad happened, we'll panic when the issue comes up.
Test Plan
Manually tested - as long as the cluster is able to work normally without crashing, this fix works. The bug that this fixes happens infrequently and we'll land a test PR after this one.
Commentary (optional)
Checklist
docs/release-notes/.See Release Note for details.
Ticket