Manually scale Pod Autoscaler of a revision #15480
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: saileshd1402 The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
The committers listed above are authorized under a signed CLA. |
|
Welcome @saileshd1402! It looks like this is your first PR to knative/serving 🎉 |
knative-prow
bot
added
needs-ok-to-test
|
Hi @saileshd1402. Thanks for your PR. I'm waiting for a knative member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
| if !equality.Semantic.DeepEqual(tmpl.Spec, pa.Spec) { | ||
| diff, _ := kmp.SafeDiff(tmpl.Spec, pa.Spec) // Can't realistically fail on PASpec. | ||
| logger.Infof("PA %s needs reconciliation, diff(-want,+got):\n%s", pa.Name, diff) | ||
| if !equality.Semantic.DeepEqual(tmpl.Spec, pa.Spec) || !equality.Semantic.DeepEqual(tmpl.Annotations, pa.Annotations) { |
There was a problem hiding this comment.
Hi @saileshd1402 I suppose you want to avoid creating a new revision, what is the use case you have? Could you describe the problem that you have?
There was a problem hiding this comment.
Hi @skonto, essentially we are facing issues with manually scaling/updating replicaset. By default when we change scale, knative-serving brings up the pods associated with the new revision, wait for them to be active then scales down the previous revision. This doesn't work for on prem scenarios since there are fixed limited resources especially when dealing with GPUs so the new revision never gets ready.
There was a problem hiding this comment.
Hi @saileshd1402 this is something https://github.com/knative-extensions/serving-progressive-rollout tries to address. You can ask @houshengbo on this one, they are facing the same in their org. Vincent maintains the extension and has some ideas on the topic.
There was a problem hiding this comment.
We did try using progressive rollout extension but it had it's own issues we faced:
- Can there be an approach to terminate the last pod of the previous revision when most pods of new revision is up knative-extensions/serving-progressive-rollout#200
- With the resourceUtil strategy, graceful traffic transfer does not occur during consecutive update requests. knative-extensions/serving-progressive-rollout#202
- With the resourceUtil strategy, traffic transfer does not occur during an update when deployment hits quota limit knative-extensions/serving-progressive-rollout#203
In a brief summary, we have noticed that the "resourceUtil" strategy fails to do graceful traffic transfer during consecutive updates, particularly when resource limits are hit. This leads to stuck states and failed requests, as traffic may still continue to direct to terminated revisions while new pods remain in pending state.
Proposed Changes
Update revision resource reconciler such that change in Revision annotations are reflected in it's Pod Autoscaler. With this change, we will be able to manually set the min-scale/max-scale of a PA by updating the annotations of a revision.
This will give us more control to manually scale up and down pods of a revision if needed.
Example:
Edit the annotations
autoscaling.knative.dev/max-scaleandautoscaling.knative.dev/min-scaleof revision to update the Pod Autoscaler as well