-
Notifications
You must be signed in to change notification settings - Fork 866
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: check isScalingEvent only on stable and newRS #3883
Conversation
Signed-off-by: Zach Aller <[email protected]>
Published E2E Test Results 4 files 4 suites 3h 18m 19s ⏱️ For more details on these failures, see this check. Results for commit 0e92e32. ♻️ This comment has been updated with latest results. |
Published Unit Test Results2 276 tests 2 276 ✅ 2m 59s ⏱️ Results for commit 0e92e32. ♻️ This comment has been updated with latest results. |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #3883 +/- ##
==========================================
- Coverage 83.88% 83.85% -0.04%
==========================================
Files 163 163
Lines 18564 18564
==========================================
- Hits 15573 15567 -6
- Misses 2119 2122 +3
- Partials 872 875 +3 ☔ View full report in Codecov by Sentry. |
Signed-off-by: Zach Aller <[email protected]>
Signed-off-by: Zach Aller <[email protected]>
Signed-off-by: Zach Aller <[email protected]>
Signed-off-by: Zach Aller <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please check my comment
Signed-off-by: Zach Aller <[email protected]>
Signed-off-by: Zach Aller <[email protected]>
Signed-off-by: Zach Aller <[email protected]>
Quality Gate passedIssues Measures |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
This bug can cause rollouts to get stuck in a tight loop of thinking it's constantly in a scaling event. Argo rollouts does not ever update the
/desired-replicas
annotation on ReplicaSets that are not stable or desired (newRs) there can be races where a ReplicaSet will get marked as old with a value in/desired-replicas
that dose not matchspec.replicas
on the rollout this could either be due to a scaling event right after that rs gets moved to old or a code path that does not update the rs before marking it as old.This change makes this loop not possible because we will not check all replicasets but instead only check stable and newRs which are the only ones we scale on scaling events.
Example state of issue below: We have an old replicaset that does not match spec.replicas of the rollout this causes isScalingEvent to get into a loop.
Rollout:
Desired/Canary
Old
Stable