Skip to content

Commit

Permalink
Address 503s when the autoscaler is being rolled (#12621)
Browse files Browse the repository at this point in the history
The activator's readiness depends on the status of web socket connection
to the autoscaler. When the connection is down the activator will report
ready=false. This can occur when the autoscaler deployment is updating.

PR #12614 made the activator's readiness probe fail aggressively after
a single failure. This didn't seem to impact istio but with contour it
started returning 503s since the activator started to report ready=false
immediately.

This PR does two things to mitigate 503s:
- bump the readiness threshold to give the autoscaler more time to
  rollout/startup. This still remains lower than the drain duration
- Update the autoscaler rollout strategy so we spin up a new instance
  prior to bring down the older one. This is done using maxUnavailable=0

Co-authored-by: dprotaso <[email protected]>
  • Loading branch information
knative-prow-robot and dprotaso authored Feb 13, 2022
1 parent bf906e8 commit af7f31e
Show file tree
Hide file tree
Showing 2 changed files with 5 additions and 1 deletion.
2 changes: 1 addition & 1 deletion config/core/deployments/activator.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ spec:
- name: k-kubelet-probe
value: "activator"
periodSeconds: 5
failureThreshold: 1
failureThreshold: 5
livenessProbe:
httpGet:
port: 8012
Expand Down
4 changes: 4 additions & 0 deletions config/core/deployments/autoscaler.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,10 @@ spec:
selector:
matchLabels:
app: autoscaler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
template:
metadata:
annotations:
Expand Down

0 comments on commit af7f31e

Please sign in to comment.